Testing CKI Builds externally

Some Kernel Merge Requests contain changes to subsystems’ files that require additional testing. This is done by external CI systems. The external CI systems are declared in the external_tests in the owners.yaml file, and assigned in the readyForMergeDeps property to applicable subsystems.

Given the limited infrastructure available to some external CI systems, we allow them to test CKI builds on their own discretion, by notifying them with eng.cki.ready_for_test messages in UMB.

ExternalCI Messaging Workflow

The complete workflow for test results and external CI integration follows these steps:

1. Merge Request applicability to external CI

   • Merge Request is created or updated
   • CKI starts running a pipeline for each variant of the Kernel
   • KWF’s “subsystems” webhook runs and adds the ExternalCI label to the MR, if applicable, based on the owners.yaml file

2. CKI Builds

   • CKI builds are complete and published successfully in each variant pipeline
   • CKI sends an eng.cki.ready_for_test message with cki_finished=false. They are not sent for pipelines that failed to build.

3. CKI Tests and automatic triaging

   • CKI finishes testing a build
   • CKI triager processes error/fail results and waives them based on known issues regexes
   • CKI sends an eng.cki.ready_for_test message to UMB with cki_finished=true, once automatic triaging finishes. This message contains the result of CKI testing after triaging in the status field, which can be success, error, or fail.
   • [ckihook] updates the CKI::OK/CKI::Failed labels in the Merge Request

   Note

   A build where all FAILed and ERRORed tests have been triaged will be reported as success. For downstream pipelines that care about CKI test status via cki_finished=true, it should be safe to trigger for any builds that report as success or error, because the latter means just that a test was aborted, probably due to a bug in the test itself.

4. External CI Trigger

   • External CI systems listen for eng.cki.ready_for_test UMB messages
   • When a message is received, they can initiate their testing process, optionally checking fields in the message, like the subsystems and external_ci lists, or even the modified_files list.

5. External CI Results Management

   • External CIs take CKI checkout and build entries from the message like {"origin": "redhat", "id": "redhat:123456789_x86_64"}, and replace the redhat origin used by CKI with their own, for example LNST would use lnst, e.g. {"origin": "lnst", "id": "lnst:123456789_x86_64"}. We call these “shadow” checkout and build entries.
   • They can indicate that they have started testing a build using the missing_test_plan field in a build. For example, they could send a message containing something like: {"origin": "lnst", "id": "lnst:123456789_x86_64", "misc": {"missing_test_plan": true}}. Alternatively, they could send the actual test plan in the form of tests without status, as long as they fill all status fields later, once the testing is complete.
   • The message should be in KCIDB v4 format, which has a detailed schema documented by kcidb-io complemented by the CKI’s misc fields schema.
   • External CIs submit their results to CKI using the umb.VirtualTopic.eng.cki.results topic.
   • datawarehouse_submitter forwards these results to DataWarehouse.

6. Result Processing and Status Updates

   • Once again, data becomes visible in DataWarehouse and undergoes automatic triaging
   • [ckihook] processes these results, and updates the ExternalCI::<origin>::<status> MR labels, based on the origin of the shadow checkouts defined by the external CI.
     • Updates label to Fail if there are untriaged tests with status="FAIL".
     • Otherwise, updates label to OK, representing that all tests passed or were waived.

7. Manual Intervention and Control

   • Kernel Maintainers, QE, and External CI teams can waive failed tests by:
     • Waiving tests in DataWarehouse using regexes or manually, following triaging process, which will render as “OK” if all failed tests get waived.
     • Override results via ExternalCI::<origin>::Waived MR label manually.
   • Any other manual interaction with any ExternalCI label triggers their reevaluation, i.e. removing a label or switching OK/Failed labels.

Ready for test message schema

{
    "ci": {
      "name": "CKI (Continuous Kernel Integration)",
      "team": "CKI",
      "docs": "https://cki-project.org",
      "url": "https://gitlab.com/cki-project",
      "email": "cki-project@redhat.com",
      "irc": "Slack #team-kernel-cki"
    },
    "run": {
      "url": "<PIPELINE_URL>"
    },
    "artifact": {
      "type": "cki-build",
      "issuer": "<PATCH_AUTHOR_EMAIL or CKI>",  # Always CKI for merge request testing
      "source_package_name": "<kernel-source-package-name>",
      "source_package_version": "<kernel-source-package-version>",  # optional
      "source_package_release": "<kernel-source-package-release>"   # optional
    },
    "checkout_id": "<KCIDB checkout_id>",
    "tree_name": "<KCIDB tree name>",
    "build_info": [
      {
        "architecture": "<ARCH>",
        "build_id": "<KCIDB build_id>",
        "package_name": "<kernel-binary-package-name>",
        "package_version": "<kernel-binary-package-version>",  # optional
        "package_release": "<kernel-binary-package-release>",  # optional
        "kernel_package_url": "<LINK_TO_REPO>",
        "debug_kernel": bool
      },
      ...
    ],
    "patch_urls": ["list", "of", "strings", "or", "empty", "list"],
    "merge_request": {
      "merge_request_url": "link-or-empty-string",
      "is_draft": bool,
      "subsystems": ["list", "of", "SSTs", "based", "on", "Subsystem", "labels"],
      "jira": ["list", "of", "jira", "links"],
      "bugzilla": ["list", "of", "bz", "links"],
      "qa_contacts": ["list", "of", "jira", "qa", "contact", "emails"],
      "external_ci": ["list", "of", "raw", "ExternalCI", "labels"],
    },
    "branch": "name-of-branch-or-empty-string",
    "modified_files": ["list", "of", "strings", "or", "empty", "list"],
    "pipelineid": <ID>,
    "cki_finished": bool,
    "status": "success|error|fail", # status NOT present if cki_finished is false
    "category": "kernel-build",
    "namespace": "cki",
    "type": "build",
    "generated_at": "<DATETIME_STRING>",
    "version": "0.1.0"
}

For documentation of tree_name, see Standardized CKI KCIDB tree names.

Installing merge request kernels

Kernels from merge requests can be installed for testing in Beaker via the kernelinstall task:

<job>
  <recipeSet>
    <recipe ks_meta="redhat_ca_cert">
      <distroRequires>
        <distro_arch op="=" value="x86_64"/>
        <variant op="=" value="BaseOS"/>
        <distro_family op="=" value="CentOSStream9"/>
      </distroRequires>
      <hostRequires/>
      <task name="Kernel installation" role="STANDALONE">
        <fetch url="https://gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/production/kernel-tests-production.zip#distribution/kernelinstall"/>
        <params>
          <param name="CHECK_CALLTRACE" value="1"/>
          <param name="KERNELARGEXTRAMODULES" value="1"/>
          <param name="KERNELARGINTERNALMODULES" value="1"/>
          <param name="KERNELARGNAME" value="{{ build_info.package_name }}"/>
          <param name="KERNELARGVERSION" value="{{ build_info.package_version }}-{{ build_info.package_release }}"/>
          <param name="KERNELARGPERMREPO" value="{{ build_info.kernel_package_url }}"/>
        </params>
      </task>
    </recipe>
  </recipeSet>
</job>

The distroRequires configuration needs to be adjusted based on the build_info.architecture and tree_name fields.