Creating Zstream Builder Images

Step-by-step guide for creating new RHEL zstream builder container images

This document provides explicit instructions for creating new RHEL zstream builder container images (RHEL 9+). It is designed to be followed by both humans and AI systems with minimal interpretation required.

Automation Note

This document has four phases, each ending with a merge request that requires human review. When following as an AI agent:

  • Execute steps within a phase without asking for confirmation
  • The build step (Phase 2, Step 3) takes several minutes - this is expected
  • STOP at the end of each phase when you see STOP: Human action required
  • Report the MR URL and wait for the user to confirm the merge before continuing
  • Only proceed to the next phase after the user confirms the previous MR is merged

Overview

When a new RHEL zstream is released (e.g., 9.7, 10.1), a corresponding builder container image must be created. This requires coordinated changes across three repositories:

  1. cki-lib (gitlab.com/cki-project/cki-lib) - CI templates that reference the builder images
  2. containers (gitlab.com/cki-project/containers) - Container image definitions
  3. kernel (gitlab.com/redhat/rhel/src/kernel) - Kernel repository that uses the builder image

The process has four phases:

  1. Phase 1: Prepare cki-lib - Add the new image reference with ONLY_DOWNSTREAM flag, merge immediately (safe because the flag prevents usage until the image exists)
  2. Phase 2: Create Container Image - Build and publish the container image
  3. Phase 3: Enable in cki-lib - Remove the ONLY_DOWNSTREAM flag to enable the image in main CI pipelines
  4. Phase 4: Update Kernel Repository - Update the kernel repository to use the new builder image

Input Parameters

The process requires the following parameters:

  • {major}: RHEL major version (e.g., 9, 10, 11)
  • {minor}: RHEL minor version (e.g., 7, 1, 0)
  • {issue_url}: URL to the GitLab issue or Jira ticket tracking this work
  • {containers_fork_remote}: Git remote name for your containers fork
  • {cki_lib_repo_path}: Path to the cki-lib repository (e.g., ../cki-lib)
  • {cki_lib_fork_remote}: Git remote name for your cki-lib fork
  • {kernel_repo_path}: Path to the kernel repository (e.g., ../kernel-rhel)
  • {kernel_upstream_remote}: Git remote for upstream kernel repo (e.g., rhel-9)
  • {kernel_fork_remote}: Git remote for your kernel repo fork (e.g., rhel-9-fork)

Note on kernel repo remotes: The kernel repository typically uses separate remotes for each RHEL major version:

  • Upstream: rhel-{major} → git@gitlab.com:redhat/rhel/src/kernel/rhel-{major}
  • Fork: e.g., rhel-{major}-fork → user’s fork of the upstream repo

Zstream branches on each remote are named {major}.{minor} (e.g., 9.6, 9.7, 10.1).

Standard MR Workflow

All phases in this guide create merge requests by pushing to a personal fork. This requires having a fork of each repository with a git remote configured.

The workflow pattern is:

  1. Navigate to the target repository (if not already there)

  2. Update to latest main: git checkout main && git pull

  3. Create a feature branch: git checkout -b {branch_name}

  4. Make changes and stage files: git add {files}

  5. Commit with issue reference: git commit -m "{message}\n\n{issue_url}"

  6. Push to your fork and create MR targeting upstream. Where {fork_remote} is the git remote pointing to your personal fork:

    git push \
    -o merge_request.create \
    -o merge_request.target=main \
    -o merge_request.title="{title}" \
    -o merge_request.description="Related issue: {issue_url}" \
    {fork_remote} {branch_name}

  7. Return to original directory (if navigated away)

Each phase below provides the concrete commands with all values filled in.

Prerequisites

Before starting, verify the following:

1. Issue URL

You must have a GitLab issue or Jira ticket URL to associate with this work.

Verification: Confirm {issue_url} has been provided.

  • If provided: Proceed to next prerequisite.
  • If NOT provided: STOP and ask the user for the issue URL before continuing.

2. Network Access

You must have access to Red Hat internal network (VPN) to reach download.devel.redhat.com.

Verification command:

curl -s -o /dev/null -w "%{http_code}" https://download.devel.redhat.com/

Success criteria: Returns 200 or 301.

3. UBI Base Image Availability

The UBI base image for the target version must exist.

Verification command:

podman pull registry.access.redhat.com/ubi{major}/ubi:{major}.{minor}

Success criteria: Image pulls successfully.

4. Repository File Exists

Check if files/rhel{major}.z.repo exists in the containers repository.

Verification command:

test -f files/rhel{major}.z.repo && echo "EXISTS" || echo "MISSING"
  • If EXISTS: No action needed, proceed to fork setup.
  • If MISSING: A new repo file must be created in Phase 2 (see 2.0 Create Repo File).

5. Containers Fork Setup

You need a fork of the containers repository with remotes configured.

Verification command:

git remote -v

Identify your fork remote: Look for a remote pointing to gitlab.com/{your-username}/containers. Note this remote name as {containers_fork_remote}.

Example output:

fork    git@gitlab.com:your-username/containers.git (fetch)
origin  git@gitlab.com:cki-project/containers.git (fetch)

In this example, {containers_fork_remote} = fork.

6. cki-lib Fork Setup

You need access to the cki-lib repository with remotes configured.

Verification command:

ls {cki_lib_repo_path}/.gitlab/ci_templates/cki-templates.yml

Success criteria: File exists.

Identify your cki-lib fork remote:

cd {cki_lib_repo_path} && git remote -v && cd -

Note the remote name pointing to your cki-lib fork as {cki_lib_fork_remote}.

7. Kernel Repository Fork Setup

You need access to the kernel repository with remotes configured for the target RHEL major version.

Verification command:

cd {kernel_repo_path} && git remote -v && cd -

Identify your remotes: Look for:

  • Upstream remote pointing to gitlab.com/redhat/rhel/src/kernel/rhel-{major} (note this as {kernel_upstream_remote})
  • Fork remote pointing to your fork (note this as {kernel_fork_remote})

Example output:

rhel-9       git@gitlab.com:redhat/rhel/src/kernel/rhel-9 (fetch)
rhel-9-fork  git@gitlab.com:your-username/rhel-9 (fetch)

In this example:

  • {kernel_upstream_remote} = rhel-9
  • {kernel_fork_remote} = rhel-9-fork

8. Tools

Required: podman.

Verification command:

podman --version

Success criteria: Returns version information.

Decision Logic: AWSCLI Flag

Based on the major version, determine the AWSCLI installation flag:

Major Version Flag Value
9 SYSTEM
10+ MATCHING

Decision rule:

IF major == 9 THEN flag = "SYSTEM"
ELSE IF major >= 10 THEN flag = "MATCHING"

Phase 1: Prepare cki-lib

This phase adds the new image reference to cki-lib with ONLY_DOWNSTREAM: 'true'. This can be merged immediately because the flag prevents the image from being used in CI until Phase 3.

1.1 Update cki-templates.yml

File: {cki_lib_repo_path}/.gitlab/ci_templates/cki-templates.yml

Location: Find the section with IMAGE_ARCHES: amd64 arm64 ppc64le s390x entries near other builder images.

Action: Add a new block for the new image with ONLY_DOWNSTREAM: 'true':

      - IMAGE_NAME:
          - builder-rhel{major}.{minor}
        IMAGE_ARCHES: amd64 arm64 ppc64le s390x
        ONLY_DOWNSTREAM: 'true'

Note: Multiple new images can be added in the same block if onboarding several at once (e.g., both 9.7 and 10.1).

1.2 Create and Merge cki-lib MR

cd {cki_lib_repo_path}
git checkout -b onboard-rhel{major}.{minor}
git add .gitlab/ci_templates/cki-templates.yml
git commit -m "gitlab-ci: add rhel{major}.{minor} builder (downstream only)

{issue_url}"
git push \
    -o merge_request.create \
    -o merge_request.target=main \
    -o merge_request.title="gitlab-ci: add rhel{major}.{minor} builder (downstream only)" \
    -o merge_request.description="Related issue: {issue_url}" \
    {cki_lib_fork_remote} onboard-rhel{major}.{minor}
cd -

Success criteria: MR is created.

STOP: Merge cki-lib preparation MR before continuing

The cki-lib MR must be reviewed and merged before proceeding. This is safe to merge immediately because ONLY_DOWNSTREAM prevents the image from being used until Phase 3.

Report the MR URL to the user and wait for confirmation that it has been merged.

Phase 2: Create Container Image

This phase creates the container image definition, builds it locally for validation, and submits an MR to the containers repository.

2.0 Create Repo File (new major versions only)

If this is the first zstream for a new major version and files/rhel{major}.z.repo does not exist, create it based on the existing pattern.

Use files/rhel9.z.repo or files/rhel10.z.repo as a template:

  1. Copy the template: cp files/rhel{existing}.z.repo files/rhel{major}.z.repo
  2. Replace all version numbers with the new major version
  3. The __Z__ placeholder will be substituted with the minor version at build time

Note: This step is only needed for the first zstream of a new major version (e.g., 11.0). Skip to 2.1 if the repo file already exists.

2.1 Create the Container Image File

Create the file builds/builder-rhel{major}.{minor}.in with the following content:

#define _AWSCLI_PIP_{flag} 1
#define _RHEL_VERSION {major}
#define _RHEL_MINOR_VERSION {minor}

#include "setup-from-rhel"
#include "builder"
#include "cleanup"

Example: RHEL 9.7

File: builds/builder-rhel9.7.in

#define _AWSCLI_PIP_SYSTEM 1
#define _RHEL_VERSION 9
#define _RHEL_MINOR_VERSION 7

#include "setup-from-rhel"
#include "builder"
#include "cleanup"

Example: RHEL 10.1

File: builds/builder-rhel10.1.in

#define _AWSCLI_PIP_MATCHING 1
#define _RHEL_VERSION 10
#define _RHEL_MINOR_VERSION 1

#include "setup-from-rhel"
#include "builder"
#include "cleanup"

2.2 Update GitLab CI Configuration

Add the new image name to the CI build matrix in .gitlab-ci.yml.

Location: Find the .images section with parallel: matrix: containing the list of IMAGE_NAME values.

Action: Add builder-rhel{major}.{minor} to the appropriate list based on visibility:

  • For RHEL images: Add to the internal visibility section alongside other builder-rhel* images

Example: To add builder-rhel10.1, find the section containing builder-rhel10.0 and add the new image:

      - IMAGE_NAME:
          - builder-rhel8
          - builder-rhel8.2
          # ... other images ...
          - builder-rhel10
          - builder-rhel10.0
          - builder-rhel10.1  # Add this line
        AUTOMATIC_DEPLOYMENT: 'false'
        VISIBILITY: internal

Verification command:

grep -q "builder-rhel{major}.{minor}" .gitlab-ci.yml && echo "FOUND" || echo "MISSING"

Success criteria: Returns FOUND.

2.3 Build the Container Image Locally

Run the build using the CKI buildah container:

podman run \
    --rm \
    --pull=newer \
    --security-opt label=disable \
    -e IMAGE_NAME=builder-rhel{major}.{minor} \
    -w /code \
    -v .:/code:z \
    -v ~/.local/share/containers:/var/lib/containers:z \
    quay.io/cki/buildah:production \
    cki_build_image.sh

Note: The --security-opt label=disable flag is required to allow buildah to access the shared container storage. This disables SELinux label enforcement for the container, which is needed because the host’s container storage files have SELinux contexts that would otherwise block access.

Success criteria: Command exits with code 0 and outputs build completion message.

Note: The local build creates the image with tag latest-amd64 (or the architecture you’re building on).

2.4 Validate the Built Image

Run validation tests on the built image:

podman run --rm localhost/builder-rhel{major}.{minor}:latest-amd64 bash -c '
    set -e
    echo "Checking gcc..." && gcc --version
    echo "Checking make..." && make --version
    echo "Checking python3..." && python3 --version
    echo "Checking build tools..." && which bison flex bc
    echo "Running compile test..."
    echo "int main(){return 0;}" > /tmp/test.c
    gcc -o /tmp/test /tmp/test.c
    /tmp/test
    echo "SUCCESS: All validation tests passed"
'

Success criteria: Output ends with SUCCESS: All validation tests passed.

2.5 Create and Submit Merge Request

2.5.1 Create a Branch

git checkout -b add-builder-rhel{major}.{minor}

2.5.2 Stage and Commit

git add builds/builder-rhel{major}.{minor}.in .gitlab-ci.yml
git commit -m "Add builder-rhel{major}.{minor} container image

{issue_url}"

2.5.3 Push to Fork and Create MR

git push \
    -o merge_request.create \
    -o merge_request.target_project=cki-project/containers \
    -o merge_request.target=main \
    -o merge_request.title="Add builder-rhel{major}.{minor} container image" \
    -o merge_request.description="Related issue: {issue_url}" \
    {containers_fork_remote} add-builder-rhel{major}.{minor}

Success criteria: Command outputs MR URL.

STOP: Merge containers MR and verify image publication

The containers MR must be reviewed, merged, and the CI pipeline must complete to build and publish the container images.

Report the MR URL to the user and wait for confirmation that:

  1. The MR has been merged
  2. The container images have been published

Verification commands (replace {mr_id} with the MR number):

# Check MR pipeline completed (image tagged with MR number)
skopeo inspect docker://quay.io/cki/builder-rhel{major}.{minor}:mr-{mr_id}

# Check production deployment completed (may require manual deployment)
skopeo inspect docker://quay.io/cki/builder-rhel{major}.{minor}:production

Both images must be available before proceeding to Phase 3.

Phase 3: Enable in cki-lib

After the containers MR is merged and the images are published, update cki-lib to move the image from the ONLY_DOWNSTREAM block to the main image list.

3.1 Update cki-templates.yml

File: {cki_lib_repo_path}/.gitlab/ci_templates/cki-templates.yml

Action: Remove the image from the ONLY_DOWNSTREAM block and add it to the main image list (the block without ONLY_DOWNSTREAM that contains other builder-rhel* images).

Before:

      - IMAGE_NAME:
          - builder-rhel9.6
          # other images...
        IMAGE_ARCHES: amd64 arm64 ppc64le s390x
      - IMAGE_NAME:
          - builder-rhel{major}.{minor}
        IMAGE_ARCHES: amd64 arm64 ppc64le s390x
        ONLY_DOWNSTREAM: 'true'

After:

      - IMAGE_NAME:
          - builder-rhel9.6
          - builder-rhel{major}.{minor}  # Moved here
          # other images...
        IMAGE_ARCHES: amd64 arm64 ppc64le s390x
      # ONLY_DOWNSTREAM block removed (or emptied if other images remain)

3.2 Create and Merge cki-lib MR

cd {cki_lib_repo_path}
git checkout main
git pull
git checkout -b enable-rhel{major}.{minor}
git add .gitlab/ci_templates/cki-templates.yml
git commit -m "gitlab-ci: enable rhel{major}.{minor} builder

{issue_url}"
git push \
    -o merge_request.create \
    -o merge_request.target=main \
    -o merge_request.title="gitlab-ci: enable rhel{major}.{minor} builder" \
    -o merge_request.description="Related issue: {issue_url}" \
    {cki_lib_fork_remote} enable-rhel{major}.{minor}
cd -

Success criteria: Command outputs MR URL.

STOP: Merge cki-lib MR before continuing

The cki-lib MR must be reviewed and merged before proceeding to Phase 4.

Report the MR URL to the user and wait for confirmation that it has been merged.

Phase 4: Update Kernel Repository

After the container image is published and enabled in cki-lib, update the kernel repository to use the new builder image. New zstream branches use the ystream builder image (e.g., builder-rhel9) as a fallback until the zstream container is available. This phase updates builder_image to the newly created zstream image.

4.1 Checkout the Zstream Branch

cd {kernel_repo_path}
git fetch {kernel_upstream_remote}
git checkout -b use-builder-{major}.{minor} {kernel_upstream_remote}/{major}.{minor}

4.2 Update builder_image in .gitlab-ci.yml

File: .gitlab-ci.yml

Location: Find the trigger pipeline section for the RHEL major version.

Action: Change builder_image from the ystream fallback to the zstream image:

Before:

.trigger_rhel{major}_pipeline:
  trigger:
    branch: rhel{major}
  variables:
    builder_image: quay.io/cki/builder-rhel{major}

After:

.trigger_rhel{major}_pipeline:
  trigger:
    branch: rhel{major}
  variables:
    builder_image: quay.io/cki/builder-rhel{major}.{minor}

4.3 Create and Submit Merge Request

The kernel repository requires a specific commit message format:

git add .gitlab-ci.yml
git commit -s -m "gitlab-ci: use rhel{major}.{minor} builder image

ubi{major}.{minor} has been released.

JIRA: INTERNAL
Upstream Status: RHEL only

Part of {issue_url}"
git push \
    -o merge_request.create \
    -o merge_request.draft \
    -o merge_request.target={major}.{minor} \
    -o merge_request.title="gitlab-ci: use rhel{major}.{minor} builder image" \
    -o "merge_request.description=ubi{major}.{minor} has been released.\n\nJIRA: INTERNAL\n\nUpstream Status: RHEL only\n\nPart of {issue_url}\n\nSigned-off-by: $(git config user.name) <$(git config user.email)>" \
    {kernel_fork_remote} use-builder-{major}.{minor}
cd -

Note: The -s flag adds the Signed-off-by line automatically. Use literal \n for newlines in the description since push options cannot contain raw newlines.

Success criteria: Command outputs MR URL.

STOP: Wait for pipeline and mark MR ready

  1. Wait for the MR pipeline to complete successfully
  2. Mark the MR as ready (remove draft status)
  3. The kernel stream maintainers will review and merge the MR on their own schedule

Report the MR URL to the user. The onboarding process is complete once the MR is submitted. The kernel maintainers will merge it when ready, after which the new builder image will be in use by the kernel CI pipelines.

Complete Example: Creating builder-rhel9.7

Assuming:

  • {issue_url} = https://gitlab.com/cki-project/cki-lib/-/issues/123
  • {containers_fork_remote} = fork
  • {cki_lib_repo_path} = ../cki-lib
  • {cki_lib_fork_remote} = fork
  • {kernel_repo_path} = ../kernel-rhel
  • {kernel_upstream_remote} = rhel-9
  • {kernel_fork_remote} = rhel-9-fork

Example: Phase 1

cd ../cki-lib
git checkout main && git pull

# Edit .gitlab/ci_templates/cki-templates.yml
# Add the following block:
#
#       - IMAGE_NAME:
#           - builder-rhel9.7
#         IMAGE_ARCHES: amd64 arm64 ppc64le s390x
#         ONLY_DOWNSTREAM: 'true'

git checkout -b onboard-rhel9.7
git add .gitlab/ci_templates/cki-templates.yml
git commit -m "gitlab-ci: add rhel9.7 builder (downstream only)

https://gitlab.com/cki-project/cki-lib/-/issues/123"
git push \
    -o merge_request.create \
    -o merge_request.target=main \
    -o merge_request.title="gitlab-ci: add rhel9.7 builder (downstream only)" \
    -o merge_request.description="Related issue: https://gitlab.com/cki-project/cki-lib/-/issues/123" \
    fork onboard-rhel9.7
cd -

# STOP: Get MR reviewed and merged before continuing

Example: Phase 2

# Prerequisites check
curl -s -o /dev/null -w "%{http_code}" https://download.devel.redhat.com/
podman pull registry.access.redhat.com/ubi9/ubi:9.7
test -f files/rhel9.z.repo && echo "Repo file exists"

# Create the .in file
cat > builds/builder-rhel9.7.in << 'EOF'
#define _AWSCLI_PIP_SYSTEM 1
#define _RHEL_VERSION 9
#define _RHEL_MINOR_VERSION 7

#include "setup-from-rhel"
#include "builder"
#include "cleanup"
EOF

# Update .gitlab-ci.yml - add builder-rhel9.7 to the internal images list
# Find the section with builder-rhel9.6 and add builder-rhel9.7 after it
sed -i '/- builder-rhel9.6$/a\          - builder-rhel9.7' .gitlab-ci.yml

# Verify the change
grep -q "builder-rhel9.7" .gitlab-ci.yml && echo "CI file updated" || echo "CI update failed"

# Build
podman run \
    --rm \
    --pull=newer \
    --security-opt label=disable \
    -e IMAGE_NAME=builder-rhel9.7 \
    -w /code \
    -v .:/code:z \
    -v ~/.local/share/containers:/var/lib/containers:z \
    quay.io/cki/buildah:production \
    cki_build_image.sh

# Validate
podman run --rm localhost/builder-rhel9.7:latest-amd64 bash -c '
    set -e
    gcc --version && make --version && python3 --version
    which bison flex bc
    echo "int main(){return 0;}" > /tmp/t.c && gcc -o /tmp/t /tmp/t.c && /tmp/t
    echo "SUCCESS: All validation tests passed"
'

# Create MR
git checkout -b add-builder-rhel9.7
git add builds/builder-rhel9.7.in .gitlab-ci.yml
git commit -m "Add builder-rhel9.7 container image

https://gitlab.com/cki-project/cki-lib/-/issues/123"
git push \
    -o merge_request.create \
    -o merge_request.target_project=cki-project/containers \
    -o merge_request.target=main \
    -o merge_request.title="Add builder-rhel9.7 container image" \
    -o merge_request.description="Related issue: https://gitlab.com/cki-project/cki-lib/-/issues/123" \
    fork add-builder-rhel9.7

# STOP: Get MR reviewed and merged, wait for CI to publish images
# Verify images are available (replace 123 with actual MR number):
#   skopeo inspect docker://quay.io/cki/builder-rhel9.7:mr-123
#   skopeo inspect docker://quay.io/cki/builder-rhel9.7:production

Example: Phase 3

cd ../cki-lib
git checkout main && git pull

# Edit .gitlab/ci_templates/cki-templates.yml
# Move builder-rhel9.7 from ONLY_DOWNSTREAM block to main image list

git checkout -b enable-rhel9.7
git add .gitlab/ci_templates/cki-templates.yml
git commit -m "gitlab-ci: enable rhel9.7 builder

https://gitlab.com/cki-project/cki-lib/-/issues/123"
git push \
    -o merge_request.create \
    -o merge_request.target=main \
    -o merge_request.title="gitlab-ci: enable rhel9.7 builder" \
    -o merge_request.description="Related issue: https://gitlab.com/cki-project/cki-lib/-/issues/123" \
    fork enable-rhel9.7
cd -

# STOP: Get MR reviewed and merged before continuing

Example: Phase 4

cd ../kernel-rhel
git fetch rhel-9
git checkout -b use-builder-9.7 rhel-9/9.7

# Edit .gitlab-ci.yml
# Find .trigger_rhel9_pipeline and change:
#   builder_image: quay.io/cki/builder-rhel9
# to:
#   builder_image: quay.io/cki/builder-rhel9.7

git add .gitlab-ci.yml
git commit -s -m "gitlab-ci: use rhel9.7 builder image

ubi9.7 has been released.

JIRA: INTERNAL
Upstream Status: RHEL only

Part of https://gitlab.com/cki-project/cki-lib/-/issues/123"
git push \
    -o merge_request.create \
    -o merge_request.draft \
    -o merge_request.target=9.7 \
    -o merge_request.title="gitlab-ci: use rhel9.7 builder image" \
    -o "merge_request.description=ubi9.7 has been released.\n\nJIRA: INTERNAL\n\nUpstream Status: RHEL only\n\nPart of https://gitlab.com/cki-project/cki-lib/-/issues/123\n\nSigned-off-by: $(git config user.name) <$(git config user.email)>" \
    rhel-9-fork use-builder-9.7
cd -

# STOP: Wait for pipeline to complete, then mark MR ready (remove draft)
# Kernel maintainers will merge on their own schedule - onboarding complete!

Troubleshooting

Build fails with network errors

Ensure you are connected to Red Hat VPN and can access download.devel.redhat.com.

UBI image not found

The UBI image for the target version may not be published yet. Wait for Red Hat to publish the UBI image before creating the builder.