Enabling a new adhoc gitlab-runner

Problem

One or more of the native gitlab-runners is down, and pipelines get stuck in the prepare or build-tools stages because no native GitLab runners of a certain architecture are available.

Steps

1. Escalate the unavailable runners via all appropriate channels. Make sure it is understood that this breaks all kernel development and delivery, especially for CVE fixes.

2. Checkout and cd to deployment-all, login into HashiCorp Vault:

   cki_secrets_login --oidc
   

3. The cki-tools container image does not include the Red Hat CA bundle, which is required for Vault and Beaker access over TLS. Copy it from the host first:

   cp /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /tmp/ca-bundle.pem
   

   The host CA bundle cannot be bind-mounted directly with the :z SELinux relabel flag because it is a system-owned file.

4. Setup one or more new adhoc Beaker machines for the required architecture via

   podman run \
       --env VAULT_ADDR \
       --env REQUESTS_CA_BUNDLE=/tmp/ca-bundle.pem \
       --env SSL_CERT_FILE=/tmp/ca-bundle.pem \
       --interactive \
       --network host \
       --rm \
       --tty \
       --volume ~/.vault-token:/cki/.vault-token:z \
       --volume ~/.ssh:/cki/.ssh:z \
       --volume /tmp/ca-bundle.pem:/tmp/ca-bundle.pem:z \
       --volume .:/data:z \
       --workdir /data \
       quay.io/cki/cki-tools:production \
       ./beaker_provision.sh adhoc-N-ARCH
   

   Replace N by a running number (1, 2) and ARCH by the required architecture (s390x, ppc64le).

   The extra environment variables and flags are:

   • REQUESTS_CA_BUNDLE: tells Python requests (used by cki_secret/Vault client) to trust the Red Hat CA.
   • SSL_CERT_FILE: tells Python’s ssl module (used by the bkr CLI for XML-RPC over HTTPS) to trust the Red Hat CA.
   • --network host: uses the host’s network stack so that internal DNS names resolve correctly.

   For debugging, add --env CKI_LOGGING_LEVEL=DEBUG.

   Wait until the machines are provisioned. The machines should be automatically added to ansible/inventory/beaker.yml. Create a new merge request with this change.

5. On the MR pipeline, switch to the jobs page. In order, execute the following jobs:

   1. various: [beaker-gitlab-runner-instance]
   2. scripts: [gitlab-runner-config, configurations apply]

   When playbooks have completed successfully for the new hosts (jobs will fail for offline hosts), you can proceed to merge the change.

6. Verify that gitlab-runners are online in runners section of both cki-internal-contributors and cki-trusted-contributors, and are not paused.

7. On Runners page, verify that jobs are succeeding for each added runner. If there are no jobs running, retrigger the pipeline for the target architecture as in the example of mentioning cki-ci-bot.

8. When the adhoc gitlab-runners are not needed anymore, run the following to return the machines and remove them from ansible/inventory/beaker.yml:

   podman run \
       --env VAULT_ADDR \
       --env REQUESTS_CA_BUNDLE=/tmp/ca-bundle.pem \
       --env SSL_CERT_FILE=/tmp/ca-bundle.pem \
       --interactive \
       --network host \
       --rm \
       --tty \
       --volume ~/.vault-token:/cki/.vault-token:z \
       --volume ~/.ssh:/cki/.ssh:z \
       --volume /tmp/ca-bundle.pem:/tmp/ca-bundle.pem:z \
       --volume .:/data:z \
       --workdir /data \
       quay.io/cki/cki-tools:production \
       ./beaker_cancel.sh adhoc-N-ARCH
   

   As above, create and merge a merge request with this change.