Old blog posts 2018-2021 on CKI Project

Celebrating our successes

Thu, 07 Oct 2021 00:00:00 +0000

Last Friday we accepted another facet of our new, mostly-remote reality, and held an online party to celebrate and acknowledge the successes we had in the past year. Normally, we would’ve met several times a year at conferences and meetings in Brno, Brussels and elsewhere, and while another online meeting didn’t really have the same flavor, it was still nice to get together for something else but work proper.

We had plenty of causes:

Advancing CI system cooperation with KernelCI

Wed, 07 Jul 2021 00:00:00 +0000

In 2019 CKI Project held a hackfest inviting all maintainers of CI systems for the Linux Kernel for discussion and cooperation.

The hackfest went well, and among other things we agreed that parsing all the different report e-mails, navigating the various dashboards, and correlating results manually, takes too much of developer and maintainer time, and lowers effectiveness of CI and testing. Building and maintaining all these systems separately is also a lot of work, which we can minimize if we join our forces and produce something everyone could use instead.

Welcoming clang built kernels into CKI family

Thu, 21 Jan 2021 00:00:00 +0000

Up till now, CKI was only building kernels using gcc toolchains. This was the standard, especially in the Fedora and RHEL land we’re interacting with. However, we are happy to announce that this is no longer the case, and CKI now allows building upstream kernels using clang/llvm! 🎉🥳

We have already enabled mainline building and testing for x86_64, aarch64 and ppc64le. Both aarch64 and x86_64 are using the full LLVM toolchain (with LLVM=1), while ppc64le is compiled with CC=clang.

Building multi-arch images

Tue, 08 Sep 2020 00:00:00 +0000

In the last post, we succeeded in building foreign arch images via qemu. As each architecture gets its own tag, it becomes a bit cumbersome to use them across architectures.

Luckily, buildah supports multi-arch manifests which allow the client to choose the suitable image for the current architecture by itself. Again, the DevConf.CZ 2020 presentation on building multi-arch container images with buildah by Nalin Dahyabhai is the best introduction that I could find.

Building foreign-arch images

Mon, 07 Sep 2020 00:00:00 +0000

The CKI pipeline is powered by the internal PSI OpenShift cluster running on x86_64 machines. For compiling kernels on x86_64, the native compiler toolchain is used. For all other architectures, the kernel is cross-compiled. While this setup is supported by the RHEL cross-compiler team for the kernel itself, this is not the case for the kernel tools.

Up to now, this was not a problem as the CKI pipeline did not build those RPMs. As that is going to change in the near future, we would like to build native versions of our container images to bootstrap the effort.

Plumbers and CKI hackfest highlights

Tue, 01 Oct 2019 00:00:00 +0000

This September, the CKI team got together for Linux Plumbers. There were a lot of great discussions and we couldn’t take part of every single one, but here are some highlights from the conference mainly related to CI, workflows and testing!

Plumbers discussions

Laura Abbott led the distribution kernel microconference. The main goal was to discuss common problems arising from maintaining a non-mainline kernel, whether it comes to keeping in sync with bugfixes, packaging or testing. A lot of the topics circled around leveraging the similar work everyone is doing and figuring out common tooling and some discussions even lead to the distribution testing BoF.

CKI hackfest agenda for Plumbers

Wed, 10 Jul 2019 00:00:00 +0000

Following the hackfest invite from the previous post, we’re here with more detailed information!

Schedule

September 12:

Time slot	Topic
09:30-10:00	Breakfast (provided)
10:00-10:30	Introductions
10:30-11:00	Test data standardization
11:10-12:00	Common place for upstream results
12:10-13:30	How to avoid effort duplication
13:30-14:30	Lunch break (not provided)
14:30-16:00	Open testing philosophy
16:10-17:00	Common hardware pools

September 13:

Time slot	Topic
09:30-10:00	Breakfast (provided)
10:00-11:20	Getting results to developers/maintainers
11:30-12:30	Onboarding new trees and tests
12:40-13:30	CI bug tracking
13:30-14:30	Lunch break (not provided)
14:30-16:00	Bugs and result interpretation
16:10-17:30	Security when testing untrusted patches

The time slots aren’t set in stone. Sessions might run longer than planned (based on how much there is to discuss and plan) and thus the whole hackfest might end later as well. Discussions and implementations can (and should) continue on mailing lists and code repositories based on created agreements.

CKI pipeline under the hood part 2: Don't reinvent the wheel

Mon, 27 May 2019 00:00:00 +0000

Now that we know what to test, we need to figure out how to do it. To reiterate, the base stages we need are:

Apply the patch(es) to the git tree
Build the kernel
Test the kernel
Profit (send the report)

As we discussed in the previous post, not all steps are relevant to all types of testing.

When the project started we used Jenkins to execute the stages. We’ve since moved away as we didn’t find it flexible enough for our needs. We often ran into errors with plugins and general maintenance, deployments took longer than we wanted and let’s be honest, writing and debugging Groovy is no fun either. We’ve spent way too much time maintaining the setup and didn’t have time to actually work on new features.

CKI pipeline under the hood part 1: Figuring out what to test

Mon, 08 Apr 2019 00:00:00 +0000

So we want to test the kernel, great idea! But what does it mean? Manually watch a random git tree, build it and submit a test job? Git is a great tool but it’s not exactly known for sending you notifications on updates. And what about testing completed builds from build systems like koji or COPR? Or if the developers come and want you to test their patches from Patchwork? It’s easy to get lost, and we haven’t even started talking about the actual test pipeline. So, let’s take a look at pipeline triggers, our solution to taming the chaos.

CKI updates and news

Mon, 01 Apr 2019 00:00:00 +0000

It’s been a while since the initial introduction of how and what we do and things have since changed. Let’s cover the high-level changes first, and then get into full technical details of our current setup in another post(s)!

First, let’s get the most important change out of the way: we are slowly migrating to GitLab! There’s still some code living in the old GitHub location (update 2020-06-03: not anymore!) which will stay there for a while. Welcome to the chaos of multiple repositories! More details about why we decided for this change will be provided in one the next posts.

Join us at Devconf.cz in Brno!

Fri, 04 Jan 2019 00:00:00 +0000

The CKI team will be together in Brno later this week for Devconf.cz and team meetings. We have a talk on Friday afternoon called “Cookies for Kernel Developers” and we hope to see you there!

Our talk covers the origin of the project, where we are today, and where we want to be in the future. We want more kernel developers and testers to be involved as well. There will be plenty of information about all of that in the talk.

Hello from the CKI Project team!

Thu, 08 Nov 2018 14:53:14 -0600

The mission of the Continuous Kernel Integration (CKI) Project is to find bugs in kernel patches before they are committed to an upstram kernel tree. We are team of kernel developers, kernel testers, and automation engineers.

We want to make each kernel maintainer’s job a little bit easier by providing automated feedback on patchsets proposed for the Linux kernel. In addition, we want to make it easier for new kernel developers to contribute and get testing feedback for their patches as well.