Lockfiles

Lockfile architecture for reproducible container builds: RPM lockfiles, Python lockfiles, two-tier design, dependency-archive caching

All CKI container images use lockfiles to pin exact dependency versions, ensuring reproducible builds regardless of upstream mirror state.

RPM lockfiles

Input: rpms.in.yaml

Human-editable package list with repository context. Located at builds/<name>/rpms/rpms.in.yaml.

Top-level keys include:

  • installWeakDeps: false – prevents weak dependencies from being pulled in
  • zchunk: false – disables zchunk metadata downloads

Repos declare updates/updates-secondary baseurls pointing to the dependency-archive caching proxy (arr-cki-prod-lambda-dependency-archive.infra.cki-project.org) for build reproducibility. The fedora/fedora-secondary (release) baseurls stay at dl.fedoraproject.org since release repos are static snapshots.

The context.image: must match the _BASE_IMAGE_TAG value (same version + digest) with its own # renovate: comment for automated base image tracking.

Output: rpms.lock.yaml

Generated by rpm-lockfile-prototype. Pins NVRAs and full download URLs for every package.

The # builddep marker

Packages tagged with # builddep in rpms.in.yaml are autoremoved in the cleanup include (build-only deps like gcc, python3-devel).

Arch-specific package filtering

rpms.in.yaml supports per-package architecture filtering via arches: {only: [x86_64, aarch64]} or arches: {not: [s390x]}. The rpm-lockfile include’s sed/awk commands handle this at install time, skipping packages not matching the build arch.

The rpm-lockfile include

The include installs lockfile packages in 5 steps:

  1. Snapshots pre-existing base image packages
  2. Installs lockfile packages by URL (extracted via sed from rpms.lock.yaml url: fields – bypasses repository resolution entirely, downloading packages directly by URL)
  3. Marks all packages as dependency-installed
  4. Re-marks pre-existing packages as user-installed (protects base image packages from autoremove)
  5. Re-marks rpms.in.yaml non-builddep entries as user-installed

DNF 4 vs 5 compatibility: DNF 4 uses dnf mark remove (dep-installed) and dnf mark install (user-installed) – counterintuitive names. DNF 5 uses the clearer dnf mark dependency / dnf mark user. The include uses cpp macros (_DNF_MARK_DEP/_DNF_MARK_USER) conditioned on _DNF_VERSION. RHEL 7 and earlier use a yum fallback (install only, no mark tracking).

The rpm-lockfile-repos include

Used by RHEL builders to delete all existing repo files from /etc/yum.repos.d/ and generate a new repo file from the repos: section of rpms.in.yaml, ensuring subsequent dnf install commands resolve from the same repositories as the lockfile.

Cleanup behavior

Lockfile images (_RPM_LOCKFILE defined) run dnf -y autoremove in the cleanup include to remove transitive deps (including # builddep packages), then skip dnf update (all packages are at pinned versions). Non-lockfile images skip autoremove and run dnf -y update instead. Both paths end with dnf clean all and cache removal.

Python lockfiles

Two-tier architecture

pyproject.toml ---> requirements.lock (global, all extras, VCS->SHA)
                         |
                         +--constrains--> builds/<name>/requirements.lock (per-image)
                         +--constrains--> tox environments

Root requirements.lock: Generated with --all-extras --no-strip-extras. Resolves ALL deps including VCS refs (@production branch refs) to commit SHAs. Master constraint for the entire repo. Requires --exclude-newer=P7D for 7-day freshness cooldown. Per-package cooldown override: --exclude-newer-package=<pkg>=false exempts a specific package from the 7-day cooldown (e.g., --exclude-newer-package=tft-request-watch=false in kernel-qe-tools).

Per-image builds/<name>/requirements.lock: Generated with --constraint=requirements.lock (or --constraints= – both forms are valid; Renovate supports both since renovatebot/renovate#43485). Resolves only the extras needed for that specific image, constrained to the versions in the root lockfile.

Natural ordering: Renovate updates root lockfile first; per-image lockfiles can only update after root merges on main (they depend on the root via --constraints=).

VCS dependency handling

pyproject.toml declares VCS deps with @production refs (e.g., cki-lib @ git+https://...@production). uv pip compile resolves these to commit SHAs. The uv wrapper’s two-pass mechanism handles extras correctly: pass 1 discovers VCS dep extras via PyPI-only constraints, pass 2 resolves with a targeted --override containing correct extras + pinned SHAs.

--no-emit-package and RPM/pip overlap

When the RPM lockfile provides Python packages (e.g., python3-pyyaml), the Python lockfile must exclude them with --no-emit-package=pyyaml to prevent double-installation.

The verify-python-lockfiles include validates at build time:

  1. No package appears in both RPM and pip lockfiles
  2. Every --no-emit-package entry is actually installed by RPMs (checked via importlib.metadata.distribution() – verifies installed Python packages, not RPM names)

This include is only needed in images with --no-emit-package entries. Heavy use in kernel-qe-tools (~49 excluded packages covering system-library deps like libvirt-python, lxml, cryptography, koji, beaker-client, testcloud and their transitive deps).

Tox constraint integration

Standard pattern: tox environments are constrained to the root lockfile.

  • Pattern A (pip install --no-deps -r requirements.lock): For repos where the lockfile only contains tox-relevant deps (cki-tools, cki-lib, kpet).
  • Pattern B (uv pip install --override): For repos where the lockfile also contains image-only deps needing system libs (kernel-qe-tools with libvirt-python). Requires --no-strip-extras on the root lockfile so transitive extras resolve correctly.

Variant: kernel-qe-tools uses a separate requirements-tox.lock (compiles specific extras --extra=kcidb-tool --extra=ci-tools --extra=dev --extra=lint --extra=test, excluding image and all) since the root lockfile includes packages that cannot install outside the container. Note: requirements-tox.lock has no Makefile target – regenerate manually by extracting and running the command from its header.

dependency-archive caching proxy

S3-backed Lambda that durably caches RPM packages and OCI container images.

RPM caching

rpms.in.yaml updates/updates-secondary baseurls point to arr-cki-prod-lambda-dependency-archive.infra.cki-project.org instead of upstream Fedora mirrors. When Fedora supersedes a package (removing the old version from the primary mirror), the cache still has it. 14-month (426-day) S3 lifecycle retention for cached .rpm files.

OCI image caching

The Lambda also caches OCI manifests and blobs from quay.io, ensuring digest-pinned base images (<ver>@sha256:<digest>) survive upstream garbage collection. Client-side routing via registries.conf with pull-from-mirror = "digest-only" for three image families:

  • quay.io/fedora/fedora
  • quay.io/fedora/eln
  • quay.io/centos/centos

Only digest-pinned pulls are routed through the cache; tag lookups go directly to quay.io. The dependency-tools image ships this config via files/registries.conf.d/ (from the containers repo). This is what makes inline digest pinning in _BASE_IMAGE_TAG reliable for reproducible builds.

The OCI cache endpoint is oci-cache.dependency-archive.cki-project.org (separate CloudFront distribution from the RPM cache). Flat content-addressed S3 keys (cache/oci/<digest>) deduplicate across image names since OCI digests are content-addressed.

Header format requirements

All requirements.lock files embed the regeneration command on line 2 (e.g., # uv pip compile ...). Makefile targets extract and replay this command.

Header rules enforced by cki_validate_lockfiles.sh in CI:

  • Long-form options only (no short flags except -v)
  • = separator for all value-carrying options (e.g., --exclude-newer=P7D not --exclude-newer P7D)
  • --exclude-newer= required on all lockfiles
  • --no-strip-extras required on root lockfiles
  • Based on Mintmaker/Renovate pip-compile manager allowlists

cki_validate_containerfiles.sh validates that Containerfile.in and rpms.in.yaml have digest-pinned base images (either @sha256: inline or currentDigest=sha256: in the Renovate comment). Both validation scripts live in cki-lib/shell-scripts/ and are distributed via pip install into the cki-tools image.

Lockfile regeneration

Makefile pattern

Every repo with lockfiles has (or should have) make targets using dependency-tools:production via podman. Currently: cki-tools, kernel-qe-tools, datawarehouse have Makefiles; containers has RPM-only targets; reporter, kpet, cki-lib, kernel-workflow Makefiles are TODO.

The target extracts the uv pip compile command from the lockfile’s own header (sed -n 's/^# //p') and re-executes it. This makes lockfiles self-documenting.

Separate targets for:

  • Root lockfile: make requirements.lock
  • Per-image Python lockfiles: make builds/<name>/requirements.lock
  • RPM lockfiles: make builds/<name>/rpms/rpms.lock.yaml

Root lockfile must be regenerated first when bumping shared deps (per-image lockfiles depend on it via --constraints=).

The PODMAN_RUN macro mounts a $(CACHE_DIR) (default /tmp/rpm-lockfile-cache) for rpm-lockfile-prototype’s DNF metadata cache. A dedicated $(CACHE_DIR): target with @mkdir -p $@ is listed as an order-only prerequisite of every target that uses PODMAN_RUN.