Container images

How the CKI team creates, builds, tests and deploys container images used for services, cron jobs and the pipeline

The CKI Project uses container images for most of it services and cron jobs as well as in the kernel build and test pipeline.

Repositories

Container images are built in two places:

The container image repository contains the container image build files (i.e. Dockerfiles) for

  • some basic container images that are used all across the CKI project (base, buildah)
  • the (builder) images used in the pipeline (python, builder-*)

For the micro services and cron jobs, the container image build files are distributed alongside the source code.

General guidelines

  • All containers other than the builder images should be directly or indirectly based on the latest Fedora release
  • Outside of the container image repository, all container images for services and cron jobs should derive from the base container image built in the container image repository
  • For nearly all GitLab CI purposes, the cki-tools container image built in the cki-tools repository should be used
  • Container images should never contain any sensitive information

Creating container images

Container image build files (i.e. Dockerfiles) should be placed in builds/image-name.in and will be preprocessed with the C preprocessor cpp by buildah. This allows to include the following building blocks with #include:

  • setup-base: FROM command for a generic Python application based on the base container image with support for customizing image name and tag
  • python-requirements: commands for a generic Python application that contains setup.cfg or requirements.txt for dependencies and a top-level run.sh as the entry point
  • cleanup: remove caches from dnf and pip, should always be at the end of your image build file

As an example, an image build file for a Python application in builds/image-name.in could look like

#include "setup-base"
/* any steps here that should happen before pip install */
#include "python-requirements"
/* any steps here that should happen after pip install */
#include "cleanup"

As the preprocessor will consider anything starting with a # as a preprocessor command, comments need to be put between C-style delimiters like /* comment */.

Building container images

A helper script cki_build_image.sh that simplifies building and pushing images is provided in the container image repository and included in the buildah image.

The script will use buildah with an image build file from builds/ via the IMAGE_NAME environment variable to build a container image. IMAGE_NAME should contain the base file name of the image build file located under the builds/, without extension, i.e. IMAGE_NAME=image-name for builds/image-name.in.

When run inside a GitLab pipeline, the script will push the container images to the project container image registry with a tag of p-<pipeline-id>.

Building foreign architectures

When setting the IMAGE_ARCH environment variable to a non-empty value, images can be built for non-native architectures via qemu. The resulting images have tags with an appended -arch. Currently, amd64, arm64, ppc64le and s390x are verified to be compatible with quay.io.

When setting the IMAGE_ARCHES environment variable to a stringified list of architectures, a multi-arch manifest can be built and uploaded. The source images are downloaded from the registry before creating the manifest to work around some peculiarities of quay.io.

Building locally

To build a container image locally, run the helper script in the buildah container with the correct IMAGE_NAME environment variable via

podman run \
    --rm \
    -e IMAGE_NAME=image-name \
    --privileged \
    -w /code \
    -v .:/code \
    -v ~/.local/share/containers:/var/lib/containers \
    registry.gitlab.com/cki-project/containers/buildah:latest \
    cki_build_image.sh

This will mount the current directory as /code inside the container, and call cki_build_image.sh with the correct IMAGE_NAME environment variable. Additionally, it will share the container storage of your local user in ~/.local/share/containers with the container. In this way, the resulting container image will be available on the host as well.

If you cannot see the container images generated inside the container on your host system, check store.GraphDriverName returned from buildah info inside and outside the buildah container. If your host system uses a different graph driver (e.g. vfs), you can force buildah inside the container to use the same driver via additionally passing -e "STORAGE_DRIVER=vfs" to the podman command line above.

Building via GitLab

The cki-lib repository contains common GitLab CI building blocks for building container images via the cki-common.yml include file.

It can be included in .gitlab-ci.yml with something like

include:
  - project: cki-project/cki-lib
    file: .gitlab/ci_templates/cki-common.yml

It provides the following job templates for container image building:

  • .publish: publish a container image named after the project (CI_PROJECT_NAME)
  • .publish_job: publish a container image named after the job (CI_JOB_NAME)
  • .tag: tag a container image with git-<12345678>, mr-<123>, <tag> and latest as appropriate
  • .deploy_production: manual job to tag a container image with production and deploy into the production environment
  • .deploy_production_image: as above, but for custom image names
  • .deploy_production_tag: manual job to tag the current git commit as production, this needs a GITLAB_JOB_DEPLOY_PRODUCTION_ACCESS_TOKEN deployment token with repo write access
  • .deploy_mr: manual job to tag a container image with mr-<123> and deploy into a testing environment
  • .deploy_mr_image: as above, but for custom image names
  • .stop_mr: stop job for the testing environments

Tags and environments

Depending on the kind of GitLab pipeline where container images are built, the images are tagged with various tags:

tag/pipeline default branch tag merge request
p-123456 always always always
git-1234abcd on success on success on success
latest on success
tag on success
mr-123 on success

For all pipelines, the p-123456 tag will always be available independent of whether the testing passes. All other tags will only be pushed after testing passed sucessfully.

Additionally, pipeline jobs that deploy into an environment can be added to a project.

For deployments to the production environment, container images are tagged with production. This job will run automatically on the default branch, but is also be available for manual execution on other pipelines.

For deployments into dynamic review environments per merge request, container images are tagged with mr-123. This job is only available for manual execution in merge request pipelines.

All deployment jobs only tag the container images. The actual deployment of these images needs to happen elsewhere.

Single container image per repository

To publish and tag a container image named after the project, and (optionally) create production and review environments, add the following jobs to .gitlab-ci.yml:

publish:
  extends: .publish

tag:
  extends: .tag

deploy-production:
  extends: .deploy_production

deploy-mr:
  extends: .deploy_mr

stop-mr:
  extends: [deploy-mr, .stop_mr]

Multiple container images per repository

If multiple container images should be built, add multiple container image build files in builds/. As more jobs are needed, the .gitlab-ci.yml jobs get slightly more complicated 🙈.

As an example, to publish and tag two images backend and frontend and (optionally) create production and review environments, the following jobs have to be added:

.images:
  parallel:
    matrix:
      - IMAGE_NAME: backend
        CHANGES: src/{core,backend}  # "/*" is automatically appended
      - IMAGE_NAME: frontend
        CHANGES: src/{core,frontend}  # "/*" is automatically appended

publish:
  extends: [.publish, .images]

tag:
  extends: [.tag, .images]

deploy-production:
  extends: [.deploy_production_image, .images]
  environment:
    name: production/$IMAGE_NAME

# MR environment per image, on_stop cannot be specified in a matrix

deploy-mr-backend:
  extends: .deploy_mr_image
  variables: {IMAGE_NAME: backend}
  environment: {on_stop: stop-mr-backend}

stop-mr-backend:
  extends: [deploy-mr-backend, .stop_mr]

deploy-mr-frontend:
  extends: .deploy_mr_image
  variables: {IMAGE_NAME: frontend}
  environment: {on_stop: stop-mr-frontend}

stop-mr-frontend:
  extends: [deploy-mr-frontend, .stop_mr]

Customizing parameters

Take a look at the publish job template to see what variables can be overridden. The BASE_IMAGE_TAG and buildah_image_tag variables allow to run image builds with newer versions of the base and buildah container images.

Building multi-arch images

To build multi-arch container images, one job per architecture plus one job for the multi-arch manifest need to be defined:

backend:
  extends: .publish_job
  variables:
    IMAGE_NAME: backend
  parallel:
    matrix:
      - IMAGE_ARCH: [amd64, arm64, ppc64le, s390x]

backend-multi:
  extends: .publish_job
  stage: 🎁
  variables:
    IMAGE_ARCHES: "amd64 arm64 ppc64le s390x"
    IMAGE_NAME: backend

Testing container images

For all merge requests in projects that use container images, updated container images with the code in the merge request will be published to the GitLab package registry of the project with tags like mr-1234, where 1234 correspondings to the merge request ID.

Testing locally

To run container images locally, use a temporary container via

podman run \
    --rm \
    --env ENV_NAME="value" \
    --env ... \
    --workdir /code \
    registry.gitlab.com/cki-project/project/image

The --env parameters allow to set environment variables inside the container. For local development, adding --volume .:/code parameter will overlay the current directory on top of /code inside the container. This means that any changes outside the container will be immediately visible inside the container, and that it is not necessary to rebuild the container image for each code change; a restart of the container is good enough.

Testing OpenShift services

The deployment-all repository contains information on how to create non-production deployments for services.

Testing OpenShift cron jobs

Clone the cron job configuration into a new pod via oc debug while overriding the container image and IS_PRODUCTION environment variable like

oc debug \
    cronjob/acme-update-cluster-routes-daily \
    --image registry.gitlab.com/cki-project/cki-tools/cki-tools:mr-73 \
    IS_PRODUCTION=false

Testing in the pipeline

The pipeline-definition repository contains references to each container image that the pipeline uses.

To test new container images:

  • create a new branch
  • change the {image_tag} and {builder_image_tag} variables in cki_pipeline.yml and gitlab-ci.yml
  • push your branch to GitLab
  • start a merge request
  • add a comment to the merge request that contains: @cki-ci-bot test
  • wait for the pipelines to finish
  • verify everything is correct

Deploying container images

Deploying OpenShift services

All services are deployed via container images with the :production tags. When the container images are tagged with :production in a production GitLab environment, the deployment-bot will trigger a corresponding pipeline in the deployment-all repository.

The precise configuration for production deployments of services can be found in deployment-all/openshift.

Deploying OpenShift cron jobs

Cron jobs run via container images with the :production tags as well. As this tag is built automatically from the default branch for all repositories, any update to the default branch of a repository results in the immediate use of the new code the next time the cron jobs runs. This implemented via the imagePullPolicy=Always setting.

The precise configuration for production deployments of cron jobs can be found in deployment-all/schedules.

Deploying in the pipeline

The pipeline uses container images with date-based tags.

To release a tested (see above) container image, create and push a new tag via

git tag 20191031.0
git push origin 20191031.0

It is also possible to create tags in GitLab’s interface via RepositoryTagsNew tag.

All tags must be in the YYYYMMDD.release format. For example the first release on October 31, 2019 would be: 20191031.1. If a new release is needed on the same day, increment the release number by one to make 20191031.2.

Once the CI pipelines are finished, the container images should be visible in GitLab’s container repository. Verify that by clicking Packages and then Container Registry.

To deploy new container images:

  • create a new branch
  • change the {image_tag} and {builder_image_tag} variables in cki_pipeline.yml and gitlab-ci.yml
  • push your branch to GitLab
  • start a merge request
  • add a comment to the merge request that contains: @cki-ci-bot test
  • wait for the pipelines to finish
  • verify everything is correct
  • request a review of the merge request

Container image registries

Container images should be public by default, i.e. they are free for anyone to use. Container images are considered private if building them required access to internal resources, e.g. internal RHEL repositories.

registry.gitlab.com

All public container images are hosted on registry.gitlab.com. As PSI OpenShift might have transient problems to pull unauthenticated images from there, a group deploy token with read_registry scope is used for all pulls.

quay.io

Internal container images are only hosted on quay.io in the cki organization.

The cki organization also contains mirror repositories for the public container images.

Access control

The cki organization contains three teams:

  • owners: admin access, contains only CKI project members
  • repocreators: allows to create new repositories, contains the push_account robot account
  • readers: read-only access, contains external users with read-only access and the pull_account robot account

If you receive an error about not being able to access container images, talk to another team member to get your permissions adjusted.

Docker Hub

Docker Hub has introduced pull rate limits that make automated use of images hosted there very unreliable. For that reason, container images consumed from Docker Hub that are not available elsewhere are mirrored into the GitLab container image registry connected to the container image repository.

As an example, the latest alpine image is used for CI/CD in the cki-artifacts-bucket-listing repository. The image is hosted on Docker Hub and is normally referenced either via the short image name alpine:latest or the full image name docker.io/alpine:latest. To improve reliability, the image is mirrored in deployment-all into the GitLab container image registry. To use the mirrored image, the image name has been replaced by registry.gitlab.com/cki-project/containers/mirror/alpine:latest.

Last modified September 13, 2021: Document graph driver problem for cki_build_image.sh (9d0525a)