Timeout for manual jobs in gitlab

There is a timeout functionality in gitlab pipelines. The running jobs are terminated after reaching configured period of time. However if the job has a manual step - it may hang in that state forever, as it is not actively running. This functionality is missing. See also: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/29574.

But we do not need to wait for gitlab implementation of that feature.

Our solution will consist of an assisting watchdog job, run in parrallel in a separate stage. As this will require new `.gitlab-ci.yml` file, we will use an extra `util` repository, to handle it. Our util pipeline It will invoke a shell script that will just check whether the time passed is already above the limit, we have declared. We will need to pass some arguments like - the parent project id - so we can check the state and eventually terminate it, and the timeout value itself - so our solution is flexible and we avoid to hardcode it in watchdog.

In the parent pipeline we need to add a `watchdog` stage. We need to invoke the utility watchdog pipeline by calling build.

main pipeline

stages: [watchdog, build, deploy]
  
watch_pipeline:
  stage: watchdog
  trigger:
    project: your-group/util
    branch: main
  variables:
    PARENT_PIPELINE_ID: $CI_PIPELINE_ID
    PARENT_PROJECT_URL: $CI_PROJECT_URL
    MAX_PIPELINE_TIMEOUT: 3600          # 1 hour
 
 # later the build goes and whatever

Here is the `.gitlab-ci.yml` in util project:

stages:
  	- watchdog
  
watch_pipeline:
  	stage: watchdog
  	image: find_some_image_that_has_bash_curl_jq
  	script:
  		- bash ./watchdog.sh
  	allow_failure: true
    when: always
    parallel: 1

And the last piece is the script itself:

#!/bin/bash
set -euo pipefail  
  
printf "PARENT_PROJECT_URL=%s\nPARENT_PIPELINE_ID=%s\nMAX_PIPELINE_TIMEOUT=%s\n" \
  "$PARENT_PROJECT_URL" "$PARENT_PIPELINE_ID" "$MAX_PIPELINE_TIMEOUT"

echo "Watching pipeline $PARENT_PIPELINE_ID in $PARENT_PROJECT_URL"
echo "Max time: $MAX_PIPELINE_TIMEOUT sekund"


readonly START_TIMESTAMP=$(date +%s)

CURRENT_TIMESTAMP=$(date +%s)
ELAPSED_TIME=$((CURRENT_TIMESTAMP - START_TIMESTAMP))

if [ "$ELAPSED_TIME" -gt "$MAX_PIPELINE_TIMEOUT" ]
then
    echo "Cancelling pipeline: ${PARENT_PIPELINE_ID}!"

    curl --request POST \
         --header "PRIVATE-TOKEN: $TOKEN" \
         --header "Accept: application/json" \
         "$CI_API_V4_URL/projects/$PARENT_PROJECT_ID/pipelines/$PARENT_PIPELINE_ID/cancel"
fi
  

Last thing that is required is to add our Personal Access Token as TOKEN variable in gitlab CICD settings/variables. Make it protected and masked, so nobody can read it and access it. Unfortunatelly CI_JOB_TOKEN would not work in this case, due to missing permissions.

Using .env with maven

Dotfiles are quite popular in ruby and js ecosystem.
They are not popular however in java world as we can use maven (still not 0xDEAD ;) and pass configuration directly from the command line or create script for that. But why not take some standards from the other universes?
Having other ways to configure app provided by eg. spring boot it may be still useful for java developers.

You can use .env file to inject variables into your maven build, then it can get picked by spring boot and passed to configuration eg. while running tests.
Use case is to avoid putting the secrets into version control, while letting local integration tests to run. Without any extra script files and using tool that is known by other developers.

There is of course a maven plugin for that:

<groupid>io.github.mjourard</groupid>
<artifactid>env-file-maven-plugin</artifactid>

I have prepared a small example of usage with groovy and spring boot.
You can find it here: https://github.com/konopski/using-dotenv-with-maven.

It is a simplest spring boot application containing one bean (TheBean) which is using to configuration values. The very standard stuff to obtain a DB connection.

@Component
class TheBean {

    @Value('${spring.datasource.username}')
    String username

    @Value('${spring.datasource.password}')
    String pass
//...

Now spring lets us deliver the actual values using externalized config mechanism . Typically we use properties files and spring profiles. In our example there is a 'dev' profile defined, that provides a clear text password. It is good enough if you remeber to keep the dedicated profile properties file out of your version control.
The file (application-dev.properties) can look like this:

spring.datasource.password=secret

In the main application.properties file we do not define this entry - so in case it is missing (eg. ommiting the profile) the app will not start.

You can also have another mechanism controlling your password. Spring will take a corresponding definition from system environment.
In our case it is delivered by our .env file.

SPRING_DATASOURCE_PASSWORD=t3st_s3cr3t

That is verified in ApplicationTest class:

@SpringBootTest
class ApplicationTests {

    @Autowired TheBean bean

//...

    @Test
    void shouldReadFromDotEnv() {
            bean.pass == "t3st_s3cr3t"
    }
}

We can also use different values by using spring's properties overriding, adding to our .env file:

OVERRIDE_DB_PASS=password_override

And test for that:

@SpringBootTest(
    properties = ['spring.datasource.password=${OVERRIDE_DB_PASS}'] )
class PropsOverrideTest {

    @Autowired TheBean bean

//...

    @Test
    void shouldOverride() {
        bean.pass == "password_override1"
    }

}

For me this approach is quite clean, allows easy sharing among the team and composes quite well with existing environment.
And remember to keep your passwords away from git.

Remove one file from git commit to a remote branch

Ooops, I did it again. Commited and pushed one file too much.

? git push --set-upstream origin bugfix/JRASERVER-65811

What now? How to remove the file from a public commit?

Let's first go back to the base branch.

? git checkout -

Switched to branch 'develop'
Your branch is up to date with 'origin/develop'.

I assume I may have some changes in my local copy so first need to clean it. I am going to delete my local branch.

? git branch -D bugfix/JRASERVER-65811
Deleted branch bugfix/JRASERVER-65811 (was 5e17f72f).

Next step is to synchronize with remote repository and fetch the branch again.

? git pull
remote: Enumerating objects: 119, done.
remote: Counting objects: 100% (119/119), done.
remote: Compressing objects: 100% (119/119), done.
remote: Total 119 (delta 41), reused 0 (delta 0)
Receiving objects: 100% (119/119), 43.63 KiB | 1.82 MiB/s, done.
Resolving deltas: 100% (41/41), done.
From gitlab:project/frontend
   736ac14a..ea0ba57e  bugfix/JRASERVER-65811-allopenissues -> origin/bugfix/JRASERVER-65811-allopenissues
Already up to date.

I can now switch to the branch back.

? git checkout bugfix/JRASERVER-65811
Switched to a new branch 'bugfix/JRASERVER-65811'
Branch 'bugfix/JRASERVER-65811' set up to track remote branch 'bugfix/JRASERVER-65811' from 'origin'.

Time to bring the last commit into staging.

? git reset --soft HEAD^

I can see this was done by showing status.

? git status
On branch bugfix/JRASERVER-65811
Your branch is behind 'origin/bugfix/JRASERVER-65811' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Changes to be committed:
  (use "git reset HEAD ..." to unstage)

        new file:   src/app/lktree/lktree.component.spec.ts
        modified:   src/app/modules/spaces/data/space-datasource.ts
        modified:   src/app/modules/spaces/document-details/document-details.component.html
        modified:   src/app/modules/spaces/spaces-tree/spaces-tree.component.ts
        modified:   src/app/modules/user-context/data/user-context-datasource.ts

The first of files is the one I want to extract and unstage from to working copy. This simply undo git add.

? git reset src/app/lktree/lktree.component.spec.ts

Yeah! My changes are now in state I wanted!

? git status
On branch bugfix/JRASERVER-65811
Your branch is behind 'origin/bugfix/JRASERVER-65811' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)

Changes to be committed:
  (use "git reset HEAD ..." to unstage)

        modified:   src/app/modules/spaces/data/space-datasource.ts
        modified:   src/app/modules/spaces/document-details/document-details.component.html
        modified:   src/app/modules/spaces/spaces-tree/spaces-tree.component.ts
        modified:   src/app/modules/user-context/data/user-context-datasource.ts

Untracked files:
  (use "git add ..." to include in what will be committed)

        src/app/lktree/

I can commit again - this time the right files.

? git commit src/app/modules/spaces/ src/app/modules/user-context/data/user-context-datasource.ts -m "JRASERVER-65811: allopenissues"
[bugfix/JRASERVER-65811 80002b42] JRASERVER-65811: allopenissues
 4 files changed, 60 insertions(+), 25 deletions(-)

Last commit is to overwrite the remote branch. Do not do it at home ;)

? git push --force

Idea: Ctrl+w considered harmful

After some years of using Eclipse as my IDE I have found myself being quite efficient in code production using that environment. My typical flow was like:
- code
- select
- extract variable/method
For selection of expressions I have been using Shift+Alt+UpArrow. It selects expression under a caret position, and each repetition will cause expanding selection to next lexical unit. Very useful.  For some time I could not find it in any other editor or IDE.
I used to be a bit affraid to loose my typing efficiency when some colleagues suggested me trying IntelliJ Idea. So I refused. Learning keymap of new IDE actually is a time consuming process and you may not feel instant gratification. And Idea's kyes seemed so weird to Eclipse guys. Also IDEA's supporters arguing that F6 changes name of file in Total Commander were... not really helpful.
I remember my first ride with IDEA during a code randori session. I switched to 'Eclipse keymap' just to edit code. It should just work like in Eclipse, right? So let's start with running some tests. Ctrl+x t. So easy. Wait! It deleted something. Damn! Code no longer compiles and public is getting bored. Not a nice first impression.
Well, today really lot of companies are using Idea, so if you want to smoothly fit into team's coding manner, probably you will end up using IntelliJ. It makes also sense to learn basic commands and shortcuts available out of the box, as non-default keymaps are not that well documented (refcards etc) and as I mentioned you will not feel like in Eclipse anymore.
And here comes the surprise: Idea also has my favourite word selection feature built in. It's named 'expand selection'. A nice surprise. The bad part is how this shortcut is located. Namely Ctrl+w. Sounds like a crime, doesn't it? This very shortcut is something that was closing current editor tab at your previous IDE is now giving you a selection. Try using both IDEs (which I think initially everyone does when switching). What you have is getting some words selected when trying to close, and (while getting back to Eclipse) situation is even worse: you close your current tab when trying to select.
 You can think about the shortcut policy in Idea as a 'vendor lock in'. You just cannot switch to any other IDE after getting used to IDEA's native keymap. I believe at IntelliJ they call it feature.
But situation is even worse: this is not only about other IDE.
Every other program you use, and it supports tab view, probably uses Ctrl+w to close current tab. Last week when trying to select few words in Chrome browser I accidentally closed window I was working with. Such things are really disturbing and cause programmer to leave their 'zone' immediatelly. Too much for me.
But nice thing about IDEA is that you are able to turn the shortcut off. Today I visited Settings | Keymap, found 'Expand selection' and remapped to old, good shortcut from Eclipse. I should just have done it when starting with IDEA two years ago.

using ed

Having a moment of spare time I decided to go back to my approach to 99 scala problems. When I started I have used pure scala scripts, so I needed eg. to call them like this:

scala -i p05.scala -i p04.scala -i p03.scala p06.scala

Now I've learnt that it may be easier to go on using

sbt console

However my code was not sbt-friendly, as it caused complaints regarding missing object or class definition. You cannot just define methods like in REPL. So I ended up with urgent neccessity to include my functions into some nice objects. As I am currently playing with rather small machine full IDE like IntelliJ is not an option. So basically I needed to use my command line to:

add object definition line before content

add closing brace after content

where the latter is trivial append to file, usually

echo "}" >> P01.scala

How to append a line of text at beggining of file? Enter ed.

Ed is an editor with tradition, used to scarry out younger programmers in team. I have read a fantastic intro to ed not so long ago. This is the ed script for making the changes with line by line explanation what it is doing (note ed will not get this comment):

  
1                                    //goto first line 
i                                    //start inserting
package konopski.skala.ninety.nine { //actual text
  object PP00 {                      //more text
.                                    //single dote
                                     //ends writing
$                                    //goto last line
a                                    //append 
  }                                  //some  
}                                    //text
.                                    //ok, that's enough

w                                    //wq seems familiar
q                                    //have you left any program this way?

After that it took one line in shell I use to complete the task:

  
for p in p*.scala; cat edme | sed -e s/PP00/$p -e s/\.scala// | ed $p ; end

Note I have used ed's famous cousin - sed to replace placeholder value PP00 to actual file name without extension. That's it, the massive work is done and I have not used copy/paste or even any vim macro. I must say it was amusing equally to playing with scala, however seems to be even more practical than next way to reverse a list ;>

jq one more time - see only what really matters

Sometimes you can find lot of information noise. Say I want to see only document id and it's content. Check this:

  
 curl -XPOST "http://es1.grey:9200/settings/d/_search?limit=50" -d'
                         {"query": { "match_all": {}}}' | jq ".hits.hits[] | [ ._id,  ._source ]"

new favourite command

A must have for console addicted git users:

git checkout -

This jumps back and forward between two last branches. Really awesome.

Using jq to talk with elasticsearch

I've recently found a new tool, quite useful, if you often talk to json services. It's named jq, please do not confuse it with jQuery. Eg. let's take Elasticsearch. I want to know which JVM it runs within. So I need to ask like this:
curl -XGET es1.grey:9200/_cluster/stats
so here's the output:

{"timestamp":1418913590962,"cluster_name":"gc-pb-grey","status":"green","indices":{"count":31,"shards":{"total":252,"primaries":126,"replication":1.0,"index":{"shards":{"min":4,"max":10,"avg":8.129032258064516},"primaries":{"min":2,"max":5,"avg":4.064516129032258},"replication":{"min":1.0,"max":1.0,"avg":1.0}}},"docs":{"count":162713289,"deleted":188051},"store":{"size":"44.2gb","size_in_bytes":47543046975,"throttle_time":"1.9d","throttle_time_in_millis":172301216},"fielddata":{"memory_size":"2.6gb","memory_size_in_bytes":2858909795,"evictions":0},"filter_cache":{"memory_size":"4.8gb","memory_size_in_bytes":5161692177,"evictions":504563},"id_cache":{"memory_size":"0b","memory_size_in_bytes":0},"completion":{"size":"0b","size_in_bytes":0},"segments":{"count":2000,"memory":"2.9gb","memory_in_bytes":3169080575}},"nodes":{"count":{"total":8,"master_only":0,"data_only":0,"master_data":8,"client":0},"versions":["0.90.12"],"os":{"available_processors":32,"mem":{"total":"188.7gb","total_in_bytes":202616012800},"cpu":[{"vendor":"AMD","model":"Opteron","mhz":2599,"total_cores":4,"total_sockets":4,"cores_per_socket":1,"cache_size":"512b","cache_size_in_bytes":512,"count":8}]},"process":{"cpu":{"percent":115},"open_file_descriptors":{"min":1356,"max":1537,"avg":1429}},"jvm":{"max_uptime":"8d","max_uptime_in_millis":693602733,"versions":[{"version":"1.6.0_30","vm_name":"OpenJDK 64-Bit Server VM","vm_version":"23.25-b01","vm_vendor":"Sun Microsystems Inc.","count":8}],"mem":{"heap_used":"59.5gb","heap_used_in_bytes":63949850024,"heap_max":"127.7gb","heap_max_in_bytes":137160032256},"threads":621},"fs":{"total":"94.4gb","total_in_bytes":101461655552,"free":"48.9gb","free_in_bytes":52530081792,"available":"44.1gb","available_in_bytes":47376134144,"disk_reads":570224,"disk_writes":164405664,"disk_io_op":164975888,"disk_read_size":"16.1gb","disk_read_size_in_bytes":17347735552,"disk_write_size":"612.6gb","disk_write_size_in_bytes":657792794624,"disk_io_size":"628.7gb","disk_io_size_in_bytes":675140530176,"disk_queue":"0","disk_service_time":"0.9"},"plugins":[{"name":"bigdesk","description":"No description found for bigdesk.","url":"/_plugin/bigdesk/","jvm":false,"site":true},{"name":"hq","description":"No description found for hq.","url":"/_plugin/hq/","jvm":false,"site":true},{"name":"head","description":"No description found for head.","url":"/_plugin/head/","jvm":false,"site":true}]}}

Quite human unreadable. Let's try now with jq:
curl -XGET es1.grey:9200/_cluster/stats | jq "."
Seems more elegant now:

{
  "nodes": {
    "plugins": [
      {
        "site": true,
        "jvm": false,
        "url": "/_plugin/bigdesk/",
        "description": "No description found for bigdesk.",
        "name": "bigdesk"
      },
      {
        "site": true,
        "jvm": false,
        "url": "/_plugin/hq/",
        "description": "No description found for hq.",
        "name": "hq"
      },
      {
        "site": true,
        "jvm": false,
        "url": "/_plugin/head/",
        "description": "No description found for head.",
        "name": "head"
      }
    ],
    "fs": {
      "disk_service_time": "0.3",
      "disk_writes": 164465520,
      "disk_reads": 570266,
      "available_in_bytes": 47369240576,
      "available": "44.1gb",
      "free_in_bytes": 52523188224,
      "free": "48.9gb",
      "total_in_bytes": 101461655552,
      "total": "94.4gb",
      "disk_io_op": 165035786,
      "disk_read_size": "16.1gb",
      "disk_read_size_in_bytes": 17348456448,
      "disk_write_size": "612.8gb",
      "disk_write_size_in_bytes": 658024804352,
      "disk_io_size": "628.9gb",
      "disk_io_size_in_bytes": 675373260800,
      "disk_queue": "8.3E-5"
    },
    "jvm": {
      "threads": 621,
      "mem": {
        "heap_max_in_bytes": 137160032256,
        "heap_max": "127.7gb",
        "heap_used_in_bytes": 53713885664,
        "heap_used": "50gb"
      },
      "versions": [
        {
          "count": 8,
          "vm_vendor": "Sun Microsystems Inc.",
          "vm_version": "23.25-b01",
          "vm_name": "OpenJDK 64-Bit Server VM",
          "version": "1.6.0_30"
        }
      ],
      "max_uptime_in_millis": 694176092,
      "max_uptime": "8d"
    },
    "process": {
      "open_file_descriptors": {
        "avg": 1432,
        "max": 1546,
        "min": 1355
      },
      "cpu": {
        "percent": 134
      }
    },
    "os": {
      "cpu": [
        {
          "count": 8,
          "vendor": "AMD",
          "model": "Opteron",
          "mhz": 2599,
          "total_cores": 4,
          "total_sockets": 4,
          "cores_per_socket": 1,
          "cache_size": "512b",
          "cache_size_in_bytes": 512
        }
      ],
      "mem": {
        "total_in_bytes": 202616012800,
        "total": "188.7gb"
      },
      "available_processors": 32
    },
    "versions": [
      "0.90.12"
    ],
    "count": {
      "client": 0,
      "master_data": 8,
      "data_only": 0,
      "master_only": 0,
      "total": 8
    }
  },
  "indices": {
    "segments": {
      "memory_in_bytes": 3169089995,
      "memory": "2.9gb",
      "count": 2023
    },
    "count": 31,
    "shards": {
      "index": {
        "replication": {
          "avg": 1,
          "max": 1,
          "min": 1
        },
        "primaries": {
          "avg": 4.064516129032258,
          "max": 5,
          "min": 2
        },
        "shards": {
          "avg": 8.129032258064516,
          "max": 10,
          "min": 4
        }
      },
      "replication": 1,
      "primaries": 126,
      "total": 252
    },
    "docs": {
      "deleted": 188053,
      "count": 162713334
    },
    "store": {
      "throttle_time_in_millis": 172306131,
      "throttle_time": "1.9d",
      "size_in_bytes": 47552519067,
      "size": "44.2gb"
    },
    "fielddata": {
      "evictions": 0,
      "memory_size_in_bytes": 2858939869,
      "memory_size": "2.6gb"
    },
    "filter_cache": {
      "evictions": 504563,
      "memory_size_in_bytes": 5181806342,
      "memory_size": "4.8gb"
    },
    "id_cache": {
      "memory_size_in_bytes": 0,
      "memory_size": "0b"
    },
    "completion": {
      "size_in_bytes": 0,
      "size": "0b"
    }
  },
  "status": "green",
  "cluster_name": "gc-pb-grey",
  "timestamp": 1418914164326
}

Ok, this does not changes much however, it's easy to get json formatted. But try this: curl -XGET es1.grey:9200/_cluster/stats | jq ".nodes.jvm" and we get:

{
  "threads": 619,
  "mem": {
    "heap_max_in_bytes": 137160032256,
    "heap_max": "127.7gb",
    "heap_used_in_bytes": 54125025648,
    "heap_used": "50.4gb"
  },
  "versions": [
    {
      "count": 8,
      "vm_vendor": "Sun Microsystems Inc.",
      "vm_version": "23.25-b01",
      "vm_name": "OpenJDK 64-Bit Server VM",
      "version": "1.6.0_30"
    }
  ],
  "max_uptime_in_millis": 694408309,
  "max_uptime": "8d"
}

I took the outer object by first dot, 'nodes' property, and from inside it 'jvm' property. It is now trivial to get actual jvm version: curl -XGET es1.grey:9200/_cluster/stats | jq ".nodes.jvm.versions[0].version" Tadam:

"1.6.0_30"

user friendly search on windows

Windows with its cmd.exe seems not to be user friendly environment for developer. But this may be changed easily, you need only to install few programs we miss from normal systems. See http://gnuwin32.sourceforge.net and download coreutils, findutils, sed, awk, grep etc. and put them to PATH. As find.exe confilicts with windows file.exe I always rename it to gfind.exe . You have to be careful with quoting - only double quote is welcomed. Also paths may be tricki, remember to replace / with \. Eg. xargs needs extra backslash to interpret one properly.

Let's find all inserts in sql files.

find . -name '*.sql' | xargs grep insert 

It is the win32 version:

gfind . -name "*.sql" | sed s!/!\\\\!g | xargs grep insert

Summary: having in mind few simple rules you may make your system much easier to talk to.