Enabling a new adhoc gitlab-runner

Problem

One or more of the native gitlab-runners is down, and pipelines get stuck in the prepare or build-tools stages because no native builders of a certain architecture are available.

Steps

1. Escalate the unavailable runners via all appropriate channels. Make sure it is understood that this breaks all kernel development and delivery, especially for CVE fixes.

2. Checkout deployment-all and export the encryption password:

   export ENVPASSWORD=....
   

3. Setup one or more new adhoc Beaker machines for the required architecture via

   podman run \
       --env ENVPASSWORD \
       --interactive \
       --rm \
       --tty \
       --volume ~/.ssh:/cki/.ssh \
       --volume .:/data \
       --workdir /data \
       quay.io/cki/cki-tools:production \
       ./beaker_provision.sh adhoc-N-ARCH
   

   Replace N by a running number (1, 2) and ARCH by the required architecture (s390x, ppc64le).

   Wait until the machines are provisioned. The machines should be automatically added to ansible/inventory/beaker.yml. Create a new merge request with this change.

4. On the MR pipeline, switch to the jobs page. In order, execute the following jobs:

   1. beaker-playbooks: [gitlab-runner-config]
   2. gitlab-runner: [configurations]

   When everything completed successfully you can proceed to merge the merge request.

5. When the adhoc gitlab-runners are not needed anymore, run the following to return the machines and remove them from ansible/inventory/beaker.yml:

   podman run \
       --env ENVPASSWORD \
       --interactive \
       --rm \
       --tty \
       --volume ~/.ssh:/cki/.ssh \
       --volume .:/data \
       --workdir /data \
       quay.io/cki/cki-tools:production \
       ./beaker_cancel.sh adhoc-N-ARCH
   

   As above, create and merge a merge request with this change.