stable URL: /l/cki-testing

CKI kernel testing

Guidelines on what CKI tests should be enabled across a kernel’s life cycle

The Linux kernel as used in Fedora-based distributions can be tested in various places throughout its stages of creation:

  • across kernel trees:
    • upstream kernel trees such as mainline or kernel-ark used for Rawhide/ELN and Fedora releases
    • downstream CentOS Stream and RHEL kernel trees
  • for a given kernel tree:
    • when a merge request is submitted or a patch set is posted
    • when a kernel is integrated with a collection of merge requests or patches
    • when a kernel is released and combined with other packages to form a compose

The following figure shows the matrix resulting from these two dimensions. CKI testing is available for the non-dashed boxes.

Development and testing stages

Guidelines whether to enable testing

Value of enabling testing

Value of testing is mostly derived from notifications and gating:

  • early notification of potential kernel issues, the earlier the further “left” the testing happens
  • preventing breaking changes from being merged into a kernel tree or integrated into a release

On a higher level, testing can be considered valuable if stakeholders act meaningfully based on the test results.

Costs of enabling testing

Depending on the exact type of testing, the following stakeholders are significantly affected by the decision to enable a test:

  • mail recipients, as their inboxes get fuller
  • kernel developers, if their merge requests are blocked
  • kernel maintainers, if their Koji/Brew builds are blocked
  • test maintainers, if they are asked to investigate a test error/failure (which normally has to happen within 24 hours)

TL;DR

CKI testing should only be enabled if the value it provides exceeds the associated costs, as determined by all involved stakeholders.

Examples

A non-exhaustive list of examples of what the above might mean follows.

Upstream (subsystem) trees

Enabling testing: For an upstream subsystem tree, kernel developers request CKI runs for specific branch names and test sets. Notifications are sent via email. Notifications to test maintainers are enabled as well if all affected parties agree that the additional upstream testing provides actual value.

Not enabling testing: Testing/notifications should not be enabled if the notification emails to either kernel developers or test maintainers are ignored.

Merge requests

Enabling testing: For merge requests, testing should be enabled for tests where any true test failure will block the code under test from getting merged. Some examples for such testing are targeted tests based on affected subsystems and global ABI checks. Additionally, stakeholders might agree on enabling further (non-blocking) tests.

Not enabling testing: Tests should not be enabled for merge requests if test failures will only result in the filing of tracking tickets, to be prioritized later. In that case, testing further to the right is preferred. This will be able to detect the same issues, but without interfering with the kernel development workflow.

Koji/Brew builds

Enabling testing: For Koji/Brew builds, testing should be enabled for tests where any true test failure will block the build from landing in the compose. Some examples for such testing are instability checks and boot tests. Additionally, stakeholders might agree on enabling further (non-blocking) tests.

Not enabling testing: Similar as for merge requests, tests should not be enabled if test failures will only be recorded in tracking tickets and waived. In that case, testing further to the right is preferred. This will be able to detect the same issues, but without interfering with the kernel maintainer workflow.