Python software installation

How Python packages in the pipeline are installed and upgraded

Rationale

While GitLab pipeline jobs run in Bash, most of the work is done by CKI-specific Python software. The installation mechanism ensures:

  • All jobs in the pipeline use the same package versions
  • New versions of CKI packages can be easily deployed and rolled back
  • Code from merge requests can be tested via bot-triggered integration tests
  • Upgrades of external package dependencies are gated by tests

Consistent package versions

Every pipeline has a prepare stage with one job per container image used in the pipeline. The prepare jobs create tarballs of virtual environments (venv) with all required software and provide them to later jobs via GitLab artifacts.

Dependencies come from requirements-python.lock and requirements-builder.lock (maintained via Renovate pip-compile manager with uv pip compile). The CI variable bridge (pipeline/requirements-lock.yml with REQUIREMENTS_PYTHON_LOCK + REQUIREMENTS_BUILDER_LOCK) injects lockfile contents as CI variables. Prepare jobs use pip install --no-deps -r to install from these lockfiles.

Deployment and rollback of CKI packages

CKI packages are downloaded from the production branches of the corresponding repositories (declared as VCS deps with @production refs in pyproject.toml, resolved to commit SHAs in lockfiles). The deploy-production jobs in MR and default branch pipelines reset the production branch HEADs appropriately. The production environment page for each repository can be used to roll back to previous code versions.

Integration testing of CKI packages

For integration testing, the CKI CI Bot allows overriding package URLs when triggering pipelines. The pipeline iterates OVERRIDABLE_PACKAGES (beaker-jobwatch, cki-lib, cki-tools, kpet, reporter) to substitute MR branch refs for production refs.

Gating of dependency updates

External package versions are gated by lockfiles: requirements-python.lock and requirements-builder.lock in the pipeline-definition repository, maintained by Renovate. Updates are filed as MRs, validated by CI, and require approval before merging.

See Lockfiles for the full lockfile architecture and Renovate for the automated maintenance flow.