Building multi-arch images

This page has an internal companion page which might contain additional information.

In the last post, we succeeded in building foreign arch images via qemu. As each architecture gets its own tag, it becomes a bit cumbersome to use them across architectures.

Luckily, buildah supports multi-arch manifests which allow the client to choose the suitable image for the current architecture by itself. Again, the DevConf.CZ 2020 presentation on building multi-arch container images with buildah by Nalin Dahyabhai is the best introduction that I could find.

In theory, it is a mere two lines to generate a manifest and upload it. For images that were just built via buildah and tagged as ‘image:tag-amd64’ and ‘image:tag-arm64’, just running the following is good enough:

buildah manifest create image:tag image:tag-amd64 image:tag-arm64
buildah manifest push image:tag docker://quay.io/repo/image:tag --all

Buildah will determine the architectures of the individual from the image manifests automatically.

Details

The devil is as always in the details, especially when it comes to the registries that should accept the resulting images.

Buildah will default to OCI-formatted images, which nearly no registry is handling properly. The web interface of registry.gitlab.com will crash internally, docker.io will pretend that the images are not multi-arch and quay.io will not accept them at all. This can be changed via --format docker or

export BUILDAH_FORMAT=docker

The registry at quay.io is quite strict in what it accepts. For example, image architectures need to be correctly tagged for the manifest to be accepted. Using aarch64 instead of arm64 will get the manifest rejected with a less-than-helpful manifest invalid error message. It is also a bit sensitive to the specific combination of manifest versions for images and image lists, as we will see in the next section.

Parallel builds

With the above setup, it is easy to build multi-arch images sequentially, i.e. build the single-arch images one after the other followed by the multi-arch manifest. The biggest disadvantage of that is that sequentially building all single-arch images is painfully slow. In contrast to a single native-arch image which might take around 5 minutes to build, building another 3 foreign-arch-on-qemu images and wrapping it up in a manifest can easily take multiple hours.

The whole process can be sped up quite a bit if the single-arch image builds and uploads are done in parallel, followed by a second stage for the multi-arch manifest. The only change that needs to be done is modifying the image references from localhost to the remote image registry like

buildah manifest create image:tag docker://quay.io/repo/image:tag-amd64 docker://quay.io/repo/image:tag-arm64

This works for registry.gitlab.com and docker.io, but not for quay.io. After a lot of debugging, the issue seems to be related to manifest versions (buildah issue, Red Hat Bugzilla). Basically, for quay.io, pulling down the single-arch images seems to downgrade the image manifest version. These downgraded image manifests are then not accepted as part of a multi-arch manifest. To work around that, it is possible to download the images manually beforehand and upgrade the manifest versions again via skopeo’s --format v2s2 parameter with

IMAGE=quay.io/repo/image:tag-amd64
skopeo copy docker://$IMAGE containers-storage:$IMAGE --format v2s2

With that in place, parallel multi-arch image building and uploading works flawlessly on the runners on gitlab.com for all registries.

The code in this post can be found in the container images repository.