Building foreign-arch images

The CKI pipeline is powered by the internal PSI OpenShift cluster running on x86_64 machines. For compiling kernels on x86_64, the native compiler toolchain is used. For all other architectures, the kernel is cross-compiled. While this setup is supported by the RHEL cross-compiler team for the kernel itself, this is not the case for the kernel tools.

Up to now, this was not a problem as the CKI pipeline did not build those RPMs. As that is going to change in the near future, we would like to build native versions of our container images to bootstrap the effort.

Installing the necessary pieces in the builder image

We use a Fedora-based builder container with cpp and buildah preinstalled to build other images. To make that work for non-native architectures, surprisingly few steps are needed.

One bit of the puzzle is qemu-user-static. It allows to transparently use qemu for foreign architectures. For a Fedora container, the only package that needs to be added to the container image is qemu-user-static via

RUN dnf install qemu-user-static

The other bit is the host setup. The documentation for qemu-user-static recommends running the multiarch/qemu-user-static setup container to prepare the host. Internally, the container uses the upstream qemu-binfmt-conf.sh to register the binfmt interpreter. As we don’t want to run a container during our builds, we add the script directly to our builder container image via

ADD https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-binfmt-conf.sh \
    /usr/local/bin/qemu-binfmt-conf.sh
RUN chmod +x /usr/local/bin/qemu-binfmt-conf.sh

Building via buildah

Building foreign-arch container images with buildah is actually pretty easy once you know how to do it. The DevConf.CZ 2020 presentation on building multi-arch container images with buildah by Nalin Dahyabhai is the best introduction that I could find.

As we want to build our image in a priviledged container managed by gitlab-runner, the qemu-binfmt-conf.sh script needs to be invoked as first part of the build process before starting the actual build via

qemu-binfmt-conf.sh --qemu-suffix -static --qemu-path /usr/bin --persistent yes

Then, passing --arch arm64 (for selecting the architecture of the resulting image) and --override-arch arm64 (for selecting the architecture of the image for FROM) to buildah is enough to build arm64 images instead of native ones.

The code in this post can be found in the container images repository.