Python software installation
Rationale and general idea
While the GitLab pipeline jobs are running in Bash, most of the “real” work is done by CKI-specific software written in Python. The required Python and data packages can be found in the GITLAB_COM_PACKAGES and DATA_PROJECTS pipeline variables.
The current approach to software installation fulfills the following requirements:
- all jobs in the pipeline use the same package versions
- new versions of CKI packages can be easily deployed and rolled back
- code from merge requests for CKI packages can be tested via bot-triggered integration tests
- upgrades of external package dependencies are gated by tests
Consistent package versions
Every pipeline has a
prepare stage with one job per container image used in
the pipeline. The
prepare jobs create tarballs of virtual environments
(venv) with all the required software and provide them to later jobs in
the pipeline via the GitLab artifacts.
Deployment and rollback of CKI packages
CKI packages are downloaded from the production branches of the corresponding
deploy-production jobs in the MR and default branch
pipelines are used for continuous delivery by resetting the
HEADs appropriately. The production environment page for each repository can
be used to roll back to previous code versions.
Integration testing of CKI packages
For integration testing, the CKI CI Bot bot allows to override package URLs when triggering pipelines.
Gating of dependency updates
To constrain external packages to “known-good” versions, the pipeline-definition repository contains a constraints.txt file that is used during installation of the CKI Python packages. For use in the pipelines, the constraints file is converted into YAML format at pipeline/constraints.yml during package upgrade.
Package upgrades in the constraints file are managed via Renovate running in an hourly cron job. The Renovate dependency dashboard issue shows all outstanding updates. Updates either already have open merge requests or are awaiting scheduling. Creation of merge requests is limited to weekends to not interfere with “normal” development work. Selecting the checkbox next to an update on the dashboard forces the creation of the corresponding merge request on the next Renovate run.