Operations handbook

Procedures for running and fixing the CKI setup

Adding a new gitlab-runner configuration

How to create a new gitlab-runner configuration and attach it to the correct projects

Adding a new Kubernetes deployment context

How to enable Kubernetes deployments to a new cluster and/or namespace in deployment-all

Administration Handbook

Instructions for sane DataWarehouse maintenance

Celebrating success

How to celebrate a success from IRC

Changing the Fedora release for the tested Koji builds

How to change the Fedora release that has its Koji kernel builds tested

Creating diagrams

How to add or modify diagrams

Creating DW service accounts

How to create service account for DataWarehouse

DataWarehouse LDAP Groups

Synchronizing LDAP groups with DataWarehouse permissions

Debugging a failing GitLab pipeline job

How to allow a kernel developer to investigate from the inside why a GitLab pipeline job failed

Debugging DataWarehouse triager and issue regexes

How to investigate why a certain test was not triaged in DataWarehouse

Disabling zstream testing

Steps to do when a variant or a complete kernel stream drops out of support

Enabling a new adhoc gitlab-runner

How to put an adhoc gitlab-runner into service

Enabling scratch build testing

How to enable Koji/Brew scratch build testing for a new RHEL major version

Fixing GitLab job system failures

How to investigate GitLab CI/CD job runner system failures

Fixing missing Koji/Brew builds

How to investigate a report of a missing Koji/Brew build

Fixing missing OSCI results

How to investigate a report of a missing OSCI results

Freezing deployments

How to stop continuous deployment for a while

Investigating UMB problems

How to investigate problems with interfacing to the Red Hat Universal Message Bus

Mirroring a centos-stream/tests/kernel into the VPN

How to set up mirorring of a repository in the centos-stream/tests/kernel group into the VPN

Profiling DataWarehouse API performance (time, query and memory)

How to profile and optimize slow and resource-intensive endpoints

Changing the configuration of the RabbitMQ nodes

How to mess with the RabbitMQ cluster without causing an outage

Refreshing recipe Beaker machines

How to solve lack of machines after an outage

Refreshing tokens

How to renew/rotate time-limited secret tokens

Removing a kernel zstream

Steps to do when support for a kernel stream has to be removed.

Purging a merge request from GitLab

How to remove all data associated with a merge request

Renewing AWS savings plans and instance reservations

How to safely navigate AWS for the scary task of committing for future spent

Syncing the data in the staging instance of DataWarehouse

How to restore a production backup into the staging instance of DataWarehouse

Adding an expression to the pipeline herder

How to get the pipeline-herder to retry GitLab jobs with certain characteristics

Rotating secrets

How to systematically rotate all secrets

Shutting down CKI

How to shut down CKI kernel testing, and communicate a planned or unplanned shutdown

Triggering test pipelines using the CKI bot

Features and details of CKI bot configuration

Updating cross-compilers

How to update the cross-compiler packages

Upgrading machines to a newer Fedora version

How to reprovision machines with a newer Fedora version without causing an outage

Updating pipeline images

How to update the container images used in the pipeline

Updating pipeline repositories

How to update the pipeline repositories

Updating y-stream composes

How to update y-zstream composes

Working with Kubernetes

How to access and work with production environments based on Kuberenetes

Zstreaming a kernel release

Steps to do when a kernel stream is to be released.