Pipeline triggering

This page has an internal companion page which might contain additional information.

Regular ways to trigger pipelines

Pipelines can be triggered in multiple ways depending on what’s being tested. Most traditional ways are handled via the pipeline-trigger repository. This repository contains code to trigger pipelines for:

Plain git repositories, after every push to the branch(es)
Already built kernels from Fedora build system (Koji)

Git triggers are supposed to run as cron jobs, while the build system trigger needs to be constantly deployed as it acts as a listener on message buses.

Kernels living in GitLab.com

With GitLab, one can set up a .gitlab-ci.yml file to run testing for merged code (equivalent of the git pipelines triggered by the pipeline-trigger), as well as proposed code changes in the form of merge requests. Such pipelines start automatically after code is merged or a merge request is submitted.

Because we use our own resources for testing, the merge request testing is only used for trusted contributors as running untrusted code in someone’s infrastructure is a bad idea due to security concerns. Limited testing for external contributors is set up via the external CI bot; the implementation and details can be found in the kernel-workflow repository and the predefined configuration used for testing lives in pipeline-data. Any changes to the configuration used for external CI runs need to be submitted against the pipeline-data project. Full testing of external merge requests can be manually triggered by a trusted reviewer by clicking the Run pipeline button on the merge request’s Pipelines tab. This would create a pipeline which matches the trusted testing. Full pipeline testing uses GitLab’s feature that allows running pipelines for forks in the parent project.

Git repo and trusted MR testing

Testing is based on a multiproject pipeline setup. The kernel repository is only used to trigger the pipeline in a different repository. The reason for this is again that we’re using our own resources, and these are shared for all GitLab kernel projects we’re testing. For maintenance, it’s way easier to add our own runners and pipeline setup into one project rather than each individual kernel repo and its forks. All of the actual pipeline runs happen in the CKI pipelines projects.

Trusted contributors are members of the appropriate CKI pipeline project which allows them to automatically start the pipelines there. The pipelines are started via trigger jobs defined in .gitlab-ci.yml of the kernel repo.

The .gitlab-ci.yml file uses templates and defaults imported from the pipeline definition repository. This makes the implementation of the .gitlab-ci.yml more straightforward (less variables to define in the kernel YAMLs), less error prone (e.g. the CVE pipelines will be triggered in the correct pipeline project) and easier to maintain (e.g. a single variable change doesn’t need to be committed into every single kernel repository and branch).

Example of such a .gitlab-ci.yml definition:

include:
  - project: cki-project/pipeline-definition
    ref: main
    file: kernel_templates.yml

workflow: !reference [.workflow]

# Basic trigger job
.trigger:
  trigger:
    branch: my-ci-branch
  variables:
    # Add any variables you need that are not defined in the template here

merge_request_regular:
  extends: [.internal, .merge_request, .with_notifications, .trigger]

# Add any more variable definitions and runs as you need

As this CI definition lives directly in the kernel repo, contributors have the ability to adjust it if their kernel changes require it. However, only trusted contributors will see their changes reflected in the running CI pipeline!

Production and testing retriggers

Both production and testing pipelines can be created by retriggering an already existing pipeline. A production pipeline is only created when the CKI_DEPLOYMENT_ENVIRONMENT environment variable is set to production. If that is not the case, a testing pipeline is created instead. Testing pipelines have an extra CKI_DEPLOYMENT_ENVIRONMENT=retrigger variable added to ensure correct handling.

An already existing pipeline can be retriggered by the retrigger script.

Pipelines for Fedora build systems can be retriggered or created by the koji trigger script. The task ID is obtained via the CLI instead of the message bus.

Only the koji trigger script accepts the regular config file from pipeline-trigger; the retrigger script takes all configuration from the original pipeline. However, both scripts accept trigger variable overrides in case an adjustment is needed. This allows creation of production pipelines that were not created e.g. because of networking issues during pipeline YAML downloads - such pipelines don’t have any jobs or trigger variables that can be queried and thus a regular retrigger doesn’t work there.

Bot testing done by cki-ci-bot on various repositories uses the same code paths as the retrigger script in the background.

Retriggering examples

To retrigger a pipeline in GitLab, you need a personal access token. The personal access token can be generated in your GitLab user settings and needs to be saved in the GITLAB_PRIVATE_TOKEN environment variable. The base URL of the GitLab instance containing the pipeline project should be saved in the GITLAB_URL variable. Optionally, the CKI_DEPLOYMENT_ENVIRONMENT variable can be exposed.

Once you’ve done all of this, you can trigger a pipeline with the retrigger script like

python3 -m cki.cki_tools.retrigger \
        https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-public-pipelines/-/pipelines/PIPELINE_ID

The example will query the pipeline PIPELINE_ID from the redhat/red-hat-ci-tools/kernel/cki-public-pipelines project, and create a new pipeline in the same project.

Extra variables and variable overrides can be provided via the --variables option. In the following example, the Beaker testing is not being skipped:

python3 -m cki.cki_tools.retrigger \
        https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-public-pipelines/-/pipelines/PIPELINE_ID
        --variables skip_beaker=false

You can trigger a pipeline with the koji trigger script like

python3 -m cki_tools.koji_trigger \
        --task-id 1234567 \
        --config-path brew.yaml