Python software installation

How Python packages in the pipeline are installed and upgraded

Rationale and general idea

While the GitLab pipeline jobs are running in Bash, most of the “real” work is done by CKI-specific software written in Python. The required Python and data packages can be found in the GITLAB_COM_PACKAGES and DATA_PROJECTS pipeline variables.

The current approach to software installation fulfills the following requirements:

  • all jobs in the pipeline use the same package versions
  • new versions of CKI packages can be easily deployed and rolled back
  • code from merge requests for CKI packages can be tested via bot-triggered integration tests
  • upgrades of external package dependencies are gated by tests

Consistent package versions

Every pipeline has a prepare stage with one job per container image used in the pipeline. The prepare jobs create tarballs of virtual environments (venv) with all the required software and provide them to later jobs in the pipeline via the GitLab artifacts.

Deployment and rollback of CKI packages

CKI packages are downloaded from the production branches of the corresponding repositories. The deploy-production jobs in the MR and default branch pipelines are used for continuous delivery by resetting the production branch HEADs appropriately. The production environment page for each repository can be used to roll back to previous code versions.

Integration testing of CKI packages

For integration testing, the CKI CI Bot bot allows to override package URLs when triggering pipelines.

Gating of dependency updates

To constrain external packages to “known-good” versions, the pipeline-definition repository contains a constraints.txt file that is used during installation of the CKI Python packages. For use in the pipelines, the constraints file is converted into YAML format at pipeline/constraints.yml during package upgrade.

Package upgrades in the constraints file are managed via Renovate running in an hourly cron job. The Renovate dependency dashboard issue shows all outstanding updates. Updates either already have open merge requests or are awaiting scheduling. Creation of merge requests is limited to weekends to not interfere with “normal” development work. Selecting the checkbox next to an update on the dashboard forces the creation of the corresponding merge request on the next Renovate run.