CKI-006: Retriggering MR pipelines

Enable the bot to retrigger pipelines from kernel repo merge requests

Abstract

This document proposes a way to enable the bot to retrigger pipelines from kernel repo merge requests.

Motivation

Once upon a time, pipelines were only triggered by either the Brew trigger microservice or the pipeline-trigger cron job. With the move of kernel development to the new kernel workflow (KWF) hosted on gitlab.com, most pipelines are now spawned “properly” via parent-child pipelines. As the bot is not able to retrigger those pipelines, test coverage decreases.

Legacy approach to triggering and retriggering pipelines

The legacy approach for triggering pipelines is via the GitLab API. To provide a suitable user interface, a new no-change commit is created on the appropriate branch with the trigger variables described in the commit message. Then, a pipeline is triggered via the API for that commit.

For retriggering such pipelines, the bot creates a temporary branch on top of the commit that should be retriggered. If required, the .gitlab-ci.yml is changed to reference the pipeline definition under test. The original trigger variables are switched to non-production mode, e.g. by adding CKI_DEPLOYMENT_ENVIRONMENT=retrigger and disabling settings related to email sending. If requested via tests_only=true, the trigger variables are adjusted to skip all stages up to the publish stage, and to reuse the publish job artifacts from the original pipeline. Then, a pipeline is triggered as described above, including the additional no-change commit.

Triggering pipelines in the brave new world of KWF

With KWF, kernel repositories are hosted on gitlab.com. The included .gitlab-ci.yml triggers both baseline (branch tip) and MR pipelines which contain bridge jobs that in turn trigger the CKI pipelines.

As eventually all kernel building (non-Brew/Koji) Fedora/CentOS/RHEL kernel pipelines will be triggered as parent-child pipelines, the bot needs to gain a way to test as much of the trigger flow above as possible.

Code under test

At the moment, the high-level trigger flow starts in the kernel repo .gitlab-ci.yml with something similar to

include:
  - project: cki-project/pipeline-definition
    ref: main
    file: kernel_templates.yml

workflow: !reference [.workflow]

.trigger:
  trigger:
    branch: rhel8

merge_request_regular:
  extends: [.internal, .merge_request, .trigger]

baseline_regular:
  extends: [.internal, .baseline, .trigger]

This goes together with some centralized templates in pipeline-definition:

.workflow:
  rules:
    - if: '$CI_MERGE_REQUEST_PROJECT_PATH =~ /^redhat.rhel/ ||
           $CI_PROJECT_PATH =~ /^redhat.rhel/'
      variables:
        REQUESTED_PIPELINE_TYPE: 'internal'

.merge_request:
  variables:
    commit_hash: ${CI_COMMIT_SHA}
    git_url: ${CI_MERGE_REQUEST_PROJECT_URL}
    branch: ${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}
    mr_id: ${CI_MERGE_REQUEST_IID}
    mr_url: ${CI_MERGE_REQUEST_PROJECT_URL}/-/merge_requests/${CI_MERGE_REQUEST_IID}
  rules:
    - if: '$PIPELINE_TYPE == $REQUESTED_PIPELINE_TYPE && $CI_MERGE_REQUEST_ID'

.baseline:
  variables:
    commit_hash: ${CI_COMMIT_SHA}
    git_url: ${CI_PROJECT_URL}
    branch: ${CI_COMMIT_BRANCH}
  rules:
    - if: '$PIPELINE_TYPE == $REQUESTED_PIPELINE_TYPE && $CI_COMMIT_BRANCH'

.internal:
  trigger:
    project: redhat/red-hat-ci-tools/kernel/cki-internal-pipelines/cki-internal-contributors
    strategy: depend
  variables:
    PIPELINE_TYPE: 'internal'

In the pipeline repository, this is combined with a copy of pipeline-definition and the checkout of various Python modules.

To summarize, the following table lists the different pieces of code involved in triggering and running CKI pipelines which should get tested by the bot.

Code under test Description
.gitlab-ci.yml Kernel repo GitLab CI jobs
kernel_templates.yml Kernel repo job templates and workflow
trees/*.yml CKI pipeline jobs depending on kernel type
cki_pipeline.yml CKI pipeline job templates and default variable values
kpet-db KPET database
python modules cki-lib, cki-tools, skt, kpet, upt, reporter

Approach

In general, to test both the .gitlab-ci.yml and kernel_templates.yml files, (at least partial) testing needs to happen at the level of the kernel repos. Those pieces are not used when spawning child pipelines directly.

One-time only: Create necessary infrastructure

Prepare the pipeline repository

  • create a temporary branch that refers to the correct pipeline-definition repo/branch

Triggering baseline parent pipelines

  • determine the last successful kernel repo baseline pipeline where the commit is present in the corresponding CI kernel repo
  • use the GitLab API to create a temporary branch from the associated commit
  • use the GitLab API to add a commit that modifies the .gitlab-ci.yml file by
    • modifying the kernel template include as appropriate
    • adding CI jobs that derive from the production jobs, and modify them so they trigger CI pipelines with the correct variables

Triggering MR parent pipelines

  • determine the last successful kernel repo MR pipeline where the merge commit is present in the corresponding CI kernel repo
  • use the GitLab API to create a temporary target branch from the associated target commit
  • use the GitLab API to create a temporary source branch from the associated source commit
  • use the GitLab API to add a commit to the source branch that modifies the .gitlab-ci.yml file by
    • modifying the kernel template include as appropriate
    • adding CI jobs that derive from the production jobs, and modify them so they trigger CI pipelines with the correct variables
  • file an MR for merging source into target

Cleanup

  • after the pipelines started, delete all temporary branches

Benefits

Changes to pipeline-definition and Python modules can be properly tested before merging.

Drawbacks

Not implementing bot support will make retriggering such pipelines very cumbersome, which will result in developers not testing their changes properly. This will introduce regressions and increase down time of the CKI service.

Alternatives

Trigger child pipelines directly

The way child pipelines are triggered mean that those could be triggered manually to appear to come from a baseline revision or an MR. This would mean that the code in the .gitlab-ci.yml and kernel_templates.yml files would not be tested.

Trigger pipelines in the production kernel repositories

There are two issues with this approach:

  • at the moment, it is not possible to trigger MR pipelines with custom trigger variables
  • retriggered pipelines in the production repositories would appear in the UI for revisions and merge requests, which would be confusing for kernel developers and might affect gating/MR readiness checks
  • filing separate MRs for retriggered pipelines might result in notification spam