CKI-007: Pipeline software installation

Efficient and stable installation of cki-project software

Problem

The pipeline needs cki-project Python packages installed. Packages can be overridden by providing a url to a repository.

Ideally:

  • All stages except prepare can install python wheels or be given a venv, rather than having to install from source.
  • Pip dependency resolution failure should not cause the pipeline to fail.
  • It is possible to manually rollback to a previously working set of Python packages

Current approach

Every pipeline has a prepare ${arch} stage for each arch, which creates a tar with the software. This has three deficiencies:

  • It creates many tarballs with identical software
  • The prepare stage can fail if pip install fails
  • For overrides, it installs the incorrect version of a package, uninstalls that version, and then installs the requested version.

Possible Approaches

In the following, cki-project repo refers to Python package repos such as cki-lib or cki-tools and data repos used in the pipeline such as kpet-db.

Build a freeze file

For each main pipeline in a cki-project repo, trigger a child pipeline in pipeline-definition to build a freeze file by installing all pipeline dependencies and calling pip freeze. On success, upload the freeze file with a unique name like cki-project-pipeline-dependencies-freeze-p-12345.txt. Run package installation unit tests for all images in pipeline-definition with this freeze file. On success, upload the freeze file in a deployment job to a well-known name such as cki-project-pipeline-dependencies-freeze-production.txt.

This freeze file is used to install all cki-project repo dependencies in the prepare jobs in the pipeline. For overrides, replace cki-project repo versions in the freeze file before installing it.

Optionally, cache the generated venv based on container image, architecture and freeze file.

Build tarballs

For each main pipeline in a cki-project repo, trigger a child pipeline in pipeline-definition to builds a venv tarball per image. On success, upload the tarball with a unique name like cki-project-pipeline-dependencies-tarball-image-arch-p-12345.tar.gz. In a deployment job, upload the tarball to a well-known name such as cki-project-pipeline-dependencies-tarball-image-arch-production.tar.gz.

This tarball is used to install all cki-project repo dependencies in the prepare jobs in the pipeline. For overrides, use the current uninstall/install hack.

Problems:

  • This will break with image upgrades that upgrade system libraries or Python which would need a rebuild of the tarball.
  • Requires building tarballs for ~22 images * 1-4 arches

Build container images

For each main pipeline in a cki-project repo, trigger a child pipeline in containers to builds corresponding -python for each pipeline container image. These images derive from the normal images, but have all pipeline dependencies installed via pip. Trigger a child pipeline in pipeline-definition to test those images.

Also build those -python images in the normal containers pipelines.

Problems:

  • A lot of images are tagged in containers. It is unclear how to determine which images need to get such a corresponding -python image, and how to easily integrate that into the containers pipelines.
  • Requires building container layers for ~22 images * 1-4 arches * number of tags

Notes

The override hack is necessary because versions cannot be used with URLs: https://www.python.org/dev/peps/pep-0508/#versions