src-git

Regression and preliminary testing of kernel merge requests before they can be merged

When a developer pushes a merge request to the kernel source-git repository, it goes through automated build and test pipelines before it can be merged. The goal is to catch regressions early while keeping developers unblocked.

Current src-git testing workflow

Build

A GitLab CI pipeline builds the kernel from the MR source. The resulting build artifacts are used for all subsequent testing stages.

Regression testing (kt0)

CKI runs kernel tier-0 (kt0) regression tests against the MR build (via pipeline-definition). Failing tests are selectively waived by Known Issue Detection (KID), which compares failures against a database of known issues (maintained in DataWarehouse) to distinguish new regressions from pre-existing problems.

Functional kernel QE teams also run their tests on MR builds via the kernel-cki schedule in qe-pipeline-definition, triggered automatically when CKI finishes building.

If all tests pass or all failures match known issues, the MR can proceed automatically.

Developer investigation

When regression testing finds an unknown failure, the developer must investigate:

  • Caused by the MR: the developer fixes the code and pushes an update, which restarts the pipeline.
  • Pre-existing / infrastructure issue: the developer requests a waiver. A test maintainer reviews the failure and, if appropriate, files a JIRA issue and new known issue in DataWarehouse. This will automatically waive the test failure so the MR can proceed.

Preliminary testing

Once regression testing passes, the MR reaches the readyForQA state. QE is notified via the readyForQA label and the JIRA Preliminary Testing field being set to Requested. QE tests the MR build to verify the fix works as intended.

If preliminary testing finds issues, QE and the developer collaborate to resolve them. When tests pass, QE sets Preliminary Testing to Pass in JIRA, which unblocks the final merge gate.

Merge gate

The readyForMerge label is applied automatically when all required conditions are met, including:

  • CKI::OK — all required pipelines succeeded or were waived
  • Acks::OK — code review approvals from relevant subsystem teams
  • JIRA::OK — JIRA policy checks pass (including PT Pass)
  • CommitRules::OK — DCO sign-off and commit message checks
  • Merge::OK — MR merges cleanly with the target branch

The release pipeline picks up readyForMerge MRs and creates a dist-git MR from them (see dist-git testing). After the dist-git MR is merged and the build is confirmed, the src-git MRs get merged automatically.

Future workflow

Future src-git testing workflow

The src-git testing workflow is evolving with two key changes:

Build system: The kernel build will eventually move to Konflux, aligning with the broader RHEL build infrastructure modernization.

Test pipelines: Regression test running moves from CKI’s pipeline-definition repository to the general qe-pipeline-definition repository.

Waiver process: In the future workflow, developers will be able to waive unrelated test failures directly instead of going through a test maintainer. This unblocks kernel developers faster when a test failure is not caused by their MR — no other human is needed in the hot path.