Shutting down CKI

How to shut down CKI kernel testing, and communicate a planned or unplanned shutdown

Dealing with Beaker shutdowns

When Beaker shuts down, it might make sense to also shutdown CKI testing to avoid ambiguous errors because of Beaker job scheduling timing out.

Jobs in Beaker are scheduled by the test stage and handled by specific gitlab-runner configurations. To disable job scheduling,

  1. Go to the settings page of one of the pipeline repositories
  2. Disable the wf-native-xci30-docker-low and wf-native-xci32-docker-low runners by clicking on the pause button ⏸.

To reenable job scheduling after the shutdown,

  1. In cee/deployment-all, reduce the number of concurrent test jobs in gitlab-runner-config/runners.yml to 50 per machine by modifying the concurrent key in the xci30 and xci32 runner deployments. File a merge request and wait for it to be deployed completely before continuing.
  2. Go to the settings page of one of the pipeline repositories
  3. Enable the wf-native-xci30-docker-low and wf-native-xci32-docker-low runners by clicking on the play button ▶.
  4. While keeping an eye on the Beaker load, repeat the first step for increments of 50 jobs per machine until the initial value of the concurrent key has been reached.

Announcing shutdowns

Emails are used to announce planned and unplanned shutdowns. The same templates can be used for internal and external clients, but verify no confidential information are included in the external communication. Do send a separate announcement to external clients to also avoid leaking the internal mailing lists!

Template

To: <EMAILS>
Cc: cki-project@redhat.com
Subject: CKI {Outage,Maintenance} <DATE_TIME>

Hello,

The CKI Team is shutting down the kernel testing pipelines due to <REASON>.

Shutdown timeline estimation:

    2019-12-23 16:00 CET: CKI kernel testing pipelines are disabled.
    2019-12-30 12:00 CET: CKI kernel testing pipelines back online and testing.

We'll update you if there are any new information or changes to the schedule.


FAQ:

Q: What if a test is running for one of my commits when the pipelines
   are disabled?
A: All of the tests that are running when the pipelines are disabled will
   be allowed to finish.

Q: What if I commit patches / submit an MR / submit a build after the
   pipelines are disabled?
A: The tip of the kernel tree, all MRs and builds will be tested when the
   pipelines are reenabled.

Q: I have more questions about how this shutdown will affect me.
A: Reply to this email and ask!

Thank you!
<NAME> and the CKI Team 🤖

Recipients

  • Always keep the CKI list in CC, as is in the template! This allows the team members to easily see what’s going on and follow up on questions if needed.
  • Usually, an announcement to external clients is not needed. In the rare cases when it is, you can find the appropriate recipients in the reporting rules of the pipeline data configs
Last modified July 27, 2022: Slowly ramp up Beaker testing after a shutdown (fea862d)