Upgrading machines to a newer Fedora version

How to reprovision machines with a newer Fedora version without causing an outage

This page has an internal companion page which contains additional information.

According to the official Fedora Linux Release Life Cycle, Fedora Linux has releases every six months, with releases being supported for about 13 months. The Fedora Project schedules contain the details for the individual releases.

Upgrading the container images

Container image Fedora upgrades are automated via Renovate: per-image _BASE_IMAGE_TAG digest bumps and RPM lockfile regeneration happen automatically within the allowed version range.

The Renovate preset caps auto-upgrades with allowedVersions: "<= 44" (configured in cki-lib/.gitlab/renovate/default.json). To enable a new Fedora version:

Validate the new Fedora release across the CKI image fleet.
Bump the cap in cki-lib/.gitlab/renovate/default.json.
Renovate will then file MRs upgrading images to the new version.

Upgrading the machines

In general, machines can be moved to a newer Fedora release either by reprovisioning or in-place upgrading. While reprovisioning is preferred, in-place upgrades are documented here as well for cases where reprovisioning is temporarily broken.

Independent of the way of upgrading, the individual machines comprising a service should be upgraded one by one, with the corresponding service staying available in a degraded state at all times.

The Machine kernel and OS versions dashboard in Grafana contains an overview of the kernel and OS versions currently running on CKI machines.

Preparations

Create a tracking ticket similar to CKI-7575.
Familiarize yourself with the machines in the various deployment environments and how to access them.
Follow the steps in the README file in the deployment-all repository to get access to the production environments and verify you can access all machines via ./ansible_ssh.sh.
In the deployment-all repository in vars.yml, change FEDORA_CORE_VERSIONS to the target version.
From the Fedora Cloud Base Images, determine the AMI IDs for the needed architectures in US East (N. Virginia) (us-east-1) for Amazon Public Cloud. Update the fedora_ami_ids variable in vars.yml with the new IDs.
Before shutting down an individual machine, gracefully stop all running services on it by logging into the machine via ./ansible_ssh.sh in the deployment-all repository checkout.

For GitLab runners, stop the gitlab-runner service via
```
sudo systemctl stop gitlab-runner
```
Note that it can take hours or even days for some systems to finish all their running jobs before gitlab-runner finally shuts down. You can see all the running jobs via a simple podman command, wait until it returns no active containers to continue the upgrade process:
```
sudo podman ps
```
If necessary, GitLab runners can be disabled on the GitLab side as well. If needed, determine the corresponding runners for a machine in the deployment-all repository checkout via the output of
```
gitlab-runner-config/deploy.sh activations generate
```
and disable them via
```
gitlab-runner-config/deploy.sh activations apply --deactivate REGEX
```
File a merge request with the changes, but do not merge it!

Changing the AMI IDs of dynamically spawned machines on AWS

Machines spawned dynamically by gitlab-runner use the changed AMI IDs via launch templates. Deploy new staging versions via

CKI_DEPLOYMENT_ENVIRONMENT=staging \
    PLAYBOOK_NAME=ansible/playbooks/aws-arr-launch-templates.yml \
    ./ansible_deploy.sh

In a dummy MR in any CKI repository with the bot hooked up, retrigger a pipeline via something like @cki-ci-bot please test [centos/c9s]. Verify that the machines spawned for the pipeline jobs via the staging launch templates are working correctly.
Submit a dummy MR in the containers repository. Verify that the machines spawned for the buildah container image integration tests via the staging launch templates are working correctly.

Reprovisioning machines

In a first step, the currently running machines need to be removed, and, in the case of machines controlled by Beaker, reprovisioned with a clean operating system.

For a Beaker-based machine, reprovision the machine in the deployment-all repository checkout via

podman run \
    --env VAULT_ADDR \
    --interactive \
    --rm \
    --tty \
    --volume ~/.vault-token:/cki/.vault-token:z \
    --volume ~/.ssh:/cki/.ssh:z \
    --volume .:/data \
    --workdir /data \
    --pull always \
    quay.io/cki/cki-tools:production \
    ./beaker_provision.sh <FQDN>

For an OpenStack-based machine, navigate to Project -> Compute -> Instances and select Delete Instance from the context menu for the existing machine.
For an AWS-based machine, navigate to EC2 -> Instances, and disable the termination protection for the instance via Actions -> Instance settings -> Change termination protection. Terminate the instance via Instance state -> Terminate instance. Click on the small edit icon next to the name of the instance and replace it by terminated. On the Tags tab, modify the CkiAnsibleGroup value to terminated as well.
For RabbitMQ machines, also remove the node from the RabbitMQ cluster. In the deployment-all repository checkout, log into any of the remaining RabbitMQ cluster nodes via ./ansible_ssh.sh, and get the list of cluster nodes via
```
sudo rabbitmqctl cluster_status
```
Compare the Disk Nodes and Running Nodes lists to find the name of the terminated node, and remove it from the cluster via
```
sudo rabbitmqctl forget_cluster_node <NODENAME>
```

After that, new machines can be configured in the deployment-all repository checkout via the playbook given in the table in the deployment environments documentation via

PLAYBOOK_NAME=<INSTANCE-PLAYBOOK> ./ansible_deploy.sh

Replace <INSTANCE-PLAYBOOK> by the appropriate playbook name.

Finally, newly configured GitLab runner machines need to get the correct gitlab-runner configuration in the deployment-all repository checkout via

gitlab-runner-config/deploy.sh configurations apply

In-place upgrades

Log into the machines via ./ansible_ssh.sh in the deployment-all repository. Make sure to use the root user with sudo -i to avoid permission issues. If ssh fails, connect via conserver instead.

For a Beaker-based machine, manually update the Beaker repository files in /etc/yum.repos.d/beaker-*.repo on the machine itself to the target version via

source /etc/os-release
TARGET_VERSION=44
TARGET_STREAM=Gold
for repo in /etc/yum.repos.d/beaker-*.repo; do
   out="/tmp/$(basename "$repo")-$TARGET_VERSION"
   sed -E "s|F-[^/]+/[^/]+/|F-$TARGET_VERSION/$TARGET_STREAM/|" "$repo" > "$out"
   diff "$repo" "$out" && echo "$repo is already up to date" || mv "$out" "$repo"
done

Download updates via

TARGET_VERSION=44
dnf -y system-upgrade download --releasever=$TARGET_VERSION

Trigger the upgrade process via
```
dnf -y system-upgrade reboot
```
Wait for the machine to be ready. If you’re using conserver you should be able to see it. Otherwise, just wait 30 minutes and try to configure the GitLab runners on the new machines via:
```
PLAYBOOK_NAME=beaker-gitlab-runner-instance ./ansible_deploy.sh -l <HOSTNAME>
gitlab-runner-config/deploy.sh registrations apply
gitlab-runner-config/deploy.sh configurations apply
```

Cleanup

If GitLab runners were disabled on the GitLab side, reactivate them again in the deployment-all repository checkout via
```
gitlab-runner-config/deploy.sh activations apply
```