Building multi-arch images
In the last post, we succeeded in building foreign arch images via qemu. As each architecture gets its own tag, it becomes a bit cumbersome to use them across architectures.
Luckily, buildah supports multi-arch manifests which allow the client to choose the suitable image for the current architecture by itself. Again, the DevConf.CZ 2020 presentation on building multi-arch container images with buildah by Nalin Dahyabhai is the best introduction that I could find.
In theory, it is a mere two lines to generate a manifest and upload it. For images that were just built via buildah and tagged as ‘image:tag-amd64’ and ‘image:tag-arm64’, just running the following is good enough:
buildah manifest create image:tag image:tag-amd64 image:tag-arm64
buildah manifest push image:tag docker://quay.io/repo/image:tag --all
Buildah will determine the architectures of the individual from the image manifests automatically.
The devil is as always in the details, especially when it comes to the registries that should accept the resulting images.
Buildah will default to OCI-formatted images, which nearly no registry is
handling properly. The web interface of registry.gitlab.com will crash
internally, docker.io will pretend that the images are not multi-arch and
quay.io will not accept them at all. This can be changed via
--format docker or
The registry at quay.io is quite strict in what it accepts. For example, image
architectures need to be correctly tagged for the manifest to be accepted.
aarch64 instead of
arm64 will get the manifest rejected with a
manifest invalid error message. It is also a bit sensitive
to the specific combination of manifest versions for images and image lists, as
we will see in the next section.
With the above setup, it is easy to build multi-arch images sequentially, i.e. build the single-arch images one after the other followed by the multi-arch manifest. The biggest disadvantage of that is that sequentially building all single-arch images is painfully slow. In contrast to a single native-arch image which might take around 5 minutes to build, building another 3 foreign-arch-on-qemu images and wrapping it up in a manifest can easily take multiple hours.
The whole process can be sped up quite a bit if the single-arch image builds and uploads are done in parallel, followed by a second stage for the multi-arch manifest. The only change that needs to be done is modifying the image references from localhost to the remote image registry like
buildah manifest create image:tag docker://quay.io/repo/image:tag-amd64 docker://quay.io/repo/image:tag-arm64
This works for registry.gitlab.com and docker.io, but not for quay.io. After a
lot of debugging, the issue seems to be related to manifest versions (buildah
issue, Red Hat
Bugzilla). Basically, for
quay.io, pulling down the single-arch images seems to downgrade the image
manifest version. These downgraded image manifests are then not accepted as
part of a multi-arch manifest. To work around that, it is possible to download
the images manually beforehand and upgrade the manifest versions again via
--format v2s2 parameter with
skopeo copy docker://$IMAGE containers-storage:$IMAGE --format v2s2
With that in place, parallel multi-arch image building and uploading works flawlessly on the runners on gitlab.com for all registries.
The code in this post can be found in the container images repository.