CKI-005: Pipeline artifact storage
This document describes the different ways that access to pipeline artifacts is provided.
Pipelines default to artifacts stored on the GitLab servers. The properties of these artifacts are sometimes in conflict with the requirements of the CKI project.
Current external artifact solutions in the pipeline
Currently, the pipeline supports different approaches to publish data outside of GitLab:
|Access to artifacts for Beaker and QE
|limited to the publish stage
|S3 artifact support
|GitLab artifacts on GitLab/S3/both
|Access to artifacts in Data Warehouse/KCIDB
|Access to a prebuild venv + pipeline scripts
|per architecture in prepare stage
The following pipeline types can be distinguished and are discussed below:
Internal pipelines in private pipeline projects
These pipelines run in private pipeline projects, e.g. for MRs in private GitLab projects.
Internal RHEL5 pipelines in private pipeline projects
Additionally, these pipelines are limited by RHEL 5 which runs in Beaker.
Trusted pipelines in public pipeline projects
These pipelines run in public pipeline projects, e.g. for MRs from trusted contributors in public GitLab projects.
Untrusted pipelines in public projects
These pipelines run in public pipeline projects for MRs from untrusted contributors.
GitLab artifacts are characterized by:
- in private projects, they are only accessible with credentials, e.g. it is
not possible to host a DNF repository on them as required for Beaker and QE
- artifacts have a maximum size of 1GiB (
- artifacts are only provided via HTTPS; in RHEL 5, the TLS support is too old
- GitLab artifacts do not allow for the publishing of incremental test results
from those pipelines; this is less of a problem when not a lot of tests are
- GitLab artifacts do not require credentials for upload (
no creds); untrusted pipelines cannot upload to external artifact storage for that reason (as they have no credentials)
The table lists which pipeline types are affected by the different aspects described above:
Two storage types will be provided with the following features:
This means to rework the artifact uploading in the pipeline to unify
publish_elsewhere, S3 artifact storage support,
KCIDB file uploading and
Per pipeline, artifacts will be either stored only on GitLab, or on GitLab and S3 simultaneously.
If artifacts are also stored on S3, jobs with artifacts > 1GiB will be excluded from the GitLab artifacts on a case-by-case basis. For S3, a browseable index page will be provided, and a link will be shown in the end of the job output.
For the different pipelines, the following storage configurations will be used:
|GitLab + internal S3/https
|GitLab + internal S3/http
|GitLab + public S3/https
- One unified way to store artifacts either on GitLab and/or S3 that fulfills all requirements
- We might find another requirement that will conflict with the proposed solution
- Keep the different ways of uploading artifacts, and add a fourth one when we solve the gcov/debug kernel size problem