Regular ways to trigger pipelines
Pipelines can be triggered in multiple ways depending on what’s being tested. Most traditional ways are handled via the pipeline-trigger repository. This repository contains code to trigger pipelines for:
- Plain git repositories, after every push to the branch(es)
- Email patch series, via queries to a Patchwork instance
- Already built kernels from Fedora build systems (Koji/COPR)
Patch and git triggers are supposed to run as cron jobs, while the build system trigger needs to be constantly deployed as it acts as a listener on message buses.
Kernels living in GitLab.com
With GitLab, one can set up a
.gitlab-ci.yml file to run testing for merged
code (equivalent of the git pipelines triggered by the pipeline-trigger), as
well as proposed code changes in the form of merge requests (rather than email
patches). Such pipelines start automatically after code is merged or a merge
request is submitted.
Because we use our own resources for testing, the merge request testing is only
used for trusted contributors as running untrusted code in someone’s
infrastructure is a bad idea due to security concerns. Limited testing for
external contributors is set up via the
external CI bot; the implementation
and details can be found in the webhooks repository and the predefined
configuration used for testing lives in pipeline-data. Any changes to the
configuration used for external CI runs need to be submitted against the
pipeline-data project. Full testing of external merge requests can be
manually triggered by a trusted reviewer by clicking the
Run pipeline button
on the merge request’s
Pipelines tab. This would create a pipeline which
matches the trusted testing. Full pipeline testing uses GitLab’s feature that
allows running pipelines for forks in the parent project.
Baseline and trusted MR testing
Testing is based on a multiproject pipeline setup. The kernel repository is only used to trigger the pipeline in a different repository. The reason for this is again that we’re using our own resources, and these are shared for all GitLab kernel projects we’re testing. For maintenance, it’s way easier to add our own runners and pipeline setup into one project rather than each individual kernel repo and its forks. All of the actual pipeline runs happen in the CKI pipelines projects.
Trusted contributors are members of the appropriate CKI pipeline project which
allows them to automatically start the pipelines there. The pipelines are
started via trigger jobs defined in
.gitlab-ci.yml of the kernel repo.
.gitlab-ci.yml file uses templates and defaults imported from the
pipeline definition repository. This makes the implementation of the
.gitlab-ci.yml more straightforward (less variables to define in the kernel
YAMLs), less error prone (e.g. the CVE pipelines will be triggered in the
correct pipeline project) and easier to maintain (e.g. a single variable change
doesn’t need to be committed into every single kernel repository and branch).
Example of such a
include: - project: cki-project/pipeline-definition ref: main file: kernel_templates.yml workflow: !reference [.workflow] # Basic trigger job .trigger: trigger: branch: my-ci-branch variables: # Add any variables you need that are not defined in the template here merge_request_regular: extends: [.internal, .merge_request, .with_notifications, .trigger] # Add any more variable definitions and runs as you need
As this CI definition lives directly in the kernel repo, contributors have the ability to adjust it if their kernel changes require it. However, only trusted contributors will see their changes reflected in the running CI pipeline!
Production and testing retriggers
Both production and testing pipelines can be created by retriggering an already
existing pipeline. A production pipeline is only created when the
IS_PRODUCTION environment variable is set to a truthy value (
If that is not the case, a testing pipeline is created instead. Testing pipelines
have an extra
retrigger = true variable added to ensure correct handling.
An already existing pipeline can be retriggered by the retrigger script.
Pipelines for Fedora build systems can be retriggered or created by the brew trigger script. The task ID is obtained via the CLI instead of the message bus.
Only the brew trigger script accepts the regular config file from pipeline-trigger; the retrigger script takes all configuration from the original pipeline. However, both scripts accept trigger variable overrides in case an adjustment is needed. This allows creation of production pipelines that were not created e.g. because of networking issues during pipeline YAML downloads - such pipelines don’t have any jobs or trigger variables that can be queried and thus a regular retrigger doesn’t work there.
Bot testing done by
cki-ci-bot on various repositories uses the same code
paths as the retrigger script in the background.
To retrigger a pipeline in GitLab, you need a personal access token. The
personal access token can be generated in your GitLab user settings and needs
to be saved in the
GITLAB_PRIVATE_TOKEN environment variable. The base URL of
the GitLab instance containing the pipeline project should be saved in the
GITLAB_URL variable. Optionally, the
IS_PRODUCTION variable can be exposed.
Once you’ve done all of this, you can trigger a pipeline with the retrigger script like
python3 -m pipeline_tools retrigger \ --project redhat/red-hat-ci-tools/kernel/cki-public-pipelines \ --pipeline-id PIPELINE_ID
The example will query the pipeline
PIPELINE_ID from the
project, and create a new pipeline in the same project.
Extra variables and variable overrides can be provided via the
option. In the following example, the Beaker testing is not being skipped:
python3 -m pipeline_tools retrigger \ --project redhat/red-hat-ci-tools/kernel/cki-public-pipelines \ --pipeline-id PIPELINE_ID \ --variables skip_beaker=false
You can trigger a pipeline with the brew trigger script like
python3 -m pipeline_tools.brew_trigger 1234567
In the above example, configuration is taken from the default local file
This can be overriden with the
-c/--config <path> or
--config-url <url> options.