release

Errata advisory creation, compose testing, and shipping to customers

Release testing is split by who owns the risk:

  • Y-stream (major releases like RHEL 9.8, 10.2): the entire OS is changing, so the functional kernel QE teams (networking, storage, realtime, etc.) and RTT own testing. They validate how the kernel integrates with everything else across multiple compose stages. Kernel maintainers have limited testing scope here.
  • Z-stream (errata updates like 9.6.z): targeted kernel-only changes with well-understood scope, so kernel maintainers (kmaint) own the verification. Functional kernel QE teams don’t need to re-validate because the change surface is small.

Y-stream

Y-stream compose testing workflow

Major releases go through multiple compose stages, each with its own testing and investigation cycle:

  1. Integration compose — built ~4 times daily as content lands from merged dist-git MRs.
  2. RTT testing — the Release Test Team (external to qe-pipeline-definition) runs tests against integration composes. The first integration compose each day that passes RTT testing is promoted as the nightly integration compose.
  3. Scheduled regression testing — run by kernel-qe against nightly composes with KID (via the qe-pipeline-definition weekly schedule). RTT investigates failures and escalates to the responsible teams.
  4. Milestone compose — an existing integration compose tagged for broader validation. Functional kernel QE teams investigate failures and file JIRA issues.
  5. CTC regression testing — Comprehensive Test Cycle testing with KID via kernel-qe (manually triggered by a Kernel QE Lead against a milestone compose, via CTC schedules).
  6. Candidate compose — an existing compose tagged as the final pre-release compose. Functional kernel QE teams investigate and file JIRA for any remaining issues.

Z-stream (current)

Current z-stream errata testing workflow

When an errata advisory transitions from NEW_FILES to QE, it triggers the kernel-errata schedule:

  1. Regression testing with KID runs via kernel-qe.
  2. kmaint QE investigates any unknown failures.
  3. kmaint QE waives failures and files JIRA issues for tracking.
  4. Errata moves to REL_PREP.

Testing is done synchronously — kmaint waits for results before moving the errata forward — but it never truly blocks delivery. Failures are investigated and waived; they don’t prevent the errata from shipping.

Z-stream (future)

Future z-stream errata testing workflow

The key change: testing runs in parallel with the errata transition. The advisory proceeds to REL_PREP without waiting for test results.

  • kmaint QE still investigates failures and files JIRA, but after the fact.
  • No waiving step is needed since testing no longer gates the transition.