Enabling a new adhoc gitlab-runner

How to put an adhoc gitlab-runner into service

Problem

One or more of the native gitlab-runners is down, and pipelines get stuck in the prepare or build-tools stages because no native builders of a certain architecture are available.

Steps

  1. Escalate the unavailable runners via all appropriate channels. Make sure it is understood that this breaks all kernel development and delivery, especially for CVE fixes.

  2. Checkout deployment-all and export the encryption password:

    export ENVPASSWORD=....
    
  3. Setup one or more new adhoc Beaker machines for the required architecture via

    podman run --rm -it \
        -e ENVPASSWORD -v .:/data -w /data \
        registry.gitlab.com/cki-project/cki-tools/cki-tools:production \
        ./beaker_provision.sh adhoc-N-ARCH
    

    Replace N by a running number (1, 2) and ARCH by the required architecture (s390x, ppc64le).

    Wait until the machines are provisioned. The machines should be automatically added to ansible/inventory/beaker.yml. Create a new merge request with this change.

  4. On the MR pipeline, switch to the jobs page. In order, execute the following jobs:

    1. beaker-playbooks: [gitlab-runner-config]
    2. gitlab-runner: [configurations]

    When everything completed successfully you can proceed to merge the merge request.

  5. When the adhoc gitlab-runners are not needed anymore, run the following to return the machines and remove them from ansible/inventory/beaker.yml:

    podman run --rm -it \
        -e ENVPASSWORD -v .:/data -w /data \
        registry.gitlab.com/cki-project/cki-tools/cki-tools:production \
        ./beaker_cancel.sh adhoc-N-ARCH
    

    As above, create and merge a merge request with this change.

Last modified February 1, 2022: Document adding and removing an adhoc runner (b4a6513)