Hello from the CKI Project team!

By Major Hayden |

The mission of the Continuous Kernel Integration (CKI) Project is to find bugs in kernel patches before they are committed to an upstram kernel tree. We are team of kernel developers, kernel testers, and automation engineers.

We want to make each kernel maintainer’s job a little bit easier by providing automated feedback on patchsets proposed for the Linux kernel. In addition, we want to make it easier for new kernel developers to contribute and get testing feedback for their patches as well.

Testing categories

Our testing falls into two categories:

  1. Baseline testing - We clone the repository, check out the latest commit in a branch, and perform tests on that kernel. Feedback is sent in a new thread to the kernel tree’s mailing list.

  2. Patchset testing - We monitor patches that are proposed for a kernel tree (often using Patchwork instances) and apply proposed patches to known working commits. Our feedback goes directly into the email thread where the patchset was proposed.

How we test

There are five main parts to our testing process.

Finding known good baselines

Before we apply patches to a kernel tree, we test the HEAD of the appropriate branch from the kernel tree to ensure it boots and passes basic tests. We then use that commit as our known good commit when we do testing on patchsets.

Merging proposed patchsets

As developers propose patchsets to a kernel tree via a mailing list, we attempt to merge those patchsets against the known good kernel commit. If a merge conflict appears, we stop testing and report the failure with an excerpt of the git am log.

Patchsets that merge successfully will move on to the next step.

Compiling the kernel

Now that we have a known good commit with patches successfully applied on top, it’s time to compile the kernel. We compile kernels for multiple architectures and our current list includes:

  • aarch64
  • ppc64le
  • s390x
  • x86_64

All of our kernels are compiled on x86_64 servers and we cross-compile the kernels for the other architectures. Any failures during this step are sent to the appropriate kernel mailing list.

If the kernel compile is successful, we move on to testing.

Kernel testing

Each successfully compiled kernel is booted on a server at Red Hat using Beaker. Red Hat has an extensive fleet of enterprise hardware across multiple architectures.

If the kernel boots successfully, we run multiple tests against the kernel on its native architecture. Our current focus is on testing with LTP but we are working to add more tests from Red Hat’s extensive library of kernel tests.

Reporting feedback

Each patchset receives feedback on success or failure along with relevant snippets of log files. That feedback is sent back to the kernel mailing list thread where the patchset originally appeared. Our goal is to deliver this feedback within four hours.

Getting involved

Feedback on our testing process is welcomed! We are currently testing and sending feedback for the stable Linux kernel for the 4.18 and 4.19 branches and we hope to expand that to more repositories soon.

Our code is in GitLab and you can email us at cki-project@redhat.com with any questions.