The mission of the Continuous Kernel Integration (CKI) Project is to find bugs in kernel patches before they are commited to an upstram kernel tree. We are team of kernel developers, kernel testers, and automation engineers.
We want to make each kernel maintainer’s job a little bit easier by providing automated feedback on patchsets proposed for the Linux kernel. In addition, we want to make it easier for new kernel developers to contribute and get testing feedback for their patches as well.
Our testing falls into two categories:
Baseline testing - We clone the repository, check out the latest commit in a branch, and perform tests on that kernel. Feedback is sent in a new thread to the kernel tree’s mailing list.
Patchset testing - We monitor patches that are proposed for a kernel tree (often using Patchwork instances) and apply proposed patches to known working commits. Our feedback goes directly into the email thread where the patchset was proposed.
How we test
There are five main parts to our testing process.
Finding known good baselines
Before we apply patches to a kernel tree, we test the
HEAD of the
appropriate branch from the kernel tree to ensure it boots and passes basic
tests. We then use that commit as our known good commit when we do testing on
Merging proposed patchsets
As developers propose patchsets to a kernel tree via a mailing list, we
attempt to merge those patchsets against the known good kernel commit. If a
merge conflict appears, we stop testing and report the failure with an
excerpt of the
git am log.
Patchsets that merge successfully will move on to the next step.
Compiling the kernel
Now that we have a known good commit with patches successfully applied on top, it’s time to compile the kernel. We compile kernels for multiple architectures and our current list includes:
All of our kernels are compiled on
x86_64 servers and we cross-compile the
kernels for the other architectures. Any failures during this step are sent
to the appropriate kernel mailing list.
If the kernel compile is successful, we move on to testing.
Each successfully compiled kernel is booted on a server at Red Hat using Beaker. Red Has has an extensive fleet of enterprise hardware across multiple architectures.
If the kernel boots successfully, we run multiple tests against the kernel on its native architecture. Our current focus is on testing with LTP but we are working to add more tests from Red Hat’s extensive library of kernel tests.
Each patchset receives feedback on success or failure along with relevant snippets of log files. That feedback is sent back to the kernel mailing list thread where the patchset originally appeared. Our goal is to deliver this feedback within four hours.
Feedback on our testing process is welcomed! We are currently testing and
sending feedback for the stable Linux
4.19 branches and we hope to expand that to more