Enabling a new adhoc gitlab-runner

How to put an adhoc gitlab-runner into service

This page has an internal companion page which might contain additional information.

Problem

One or more of the native gitlab-runners is down, and pipelines get stuck in the prepare or build-tools stages because no native GitLab runners of a certain architecture are available.

Steps

Escalate the unavailable runners via all appropriate channels. Make sure it is understood that this breaks all kernel development and delivery, especially for CVE fixes.
Checkout and cd to deployment-all, export the encryption password:
```
export ENVPASSWORD=....
```
Setup one or more new adhoc Beaker machines for the required architecture via
```
podman run \
    --network host \
    --env ENVPASSWORD \
    --interactive \
    --rm \
    --tty \
    --volume .:/data:z \
    --workdir /data \
    quay.io/cki/cki-tools:production \
    ./beaker_provision.sh adhoc-N-ARCH
```
Replace N by a running number (1, 2) and ARCH by the required architecture (s390x, ppc64le).

Wait until the machines are provisioned. The machines should be automatically added to ansible/inventory/beaker.yml. Create a new merge request with this change.
On the MR pipeline, switch to the jobs page. In order, execute the following jobs:
1. various: [beaker-gitlab-runner-instance]
2. scripts: [gitlab-runner-config, configurations apply]
When playbooks have completed successfully for the new hosts (jobs will fail for offline hosts), you can proceed to merge the change.
Verify that gitlab-runners are online in runners section of both cki-internal-contributors and cki-trusted-contributors, and are not paused.
On Runners page, verify that jobs are succeeding for each added runner. If there are no jobs running, retrigger the pipeline for the target architecture as in the example of mentioning cki-ci-bot.

When the adhoc gitlab-runners are not needed anymore, run the following to return the machines and remove them from ansible/inventory/beaker.yml:

podman run \
    --network host \
    --env ENVPASSWORD \
    --interactive \
    --rm \
    --tty \
    --volume .:/data:z \
    --workdir /data \
    quay.io/cki/cki-tools:production \
    ./beaker_cancel.sh adhoc-N-ARCH

As above, create and merge a merge request with this change.