DataWarehouse services & events

The CKI DataWarehouse (DW) is the central store for kernel test results. Around it, a set of independent microservices form an event-driven pipeline connected through RabbitMQ and Kafka messaging. This page describes how those services fit together: their queues, APIs, and interactions, including the MR label lifecycle driven by kernel-workflow and the ExternalCI boundary.

For the RabbitMQ message schemas and DLX retry architecture see Communication fabric. For the published event reference see DataWarehouse microservices.

Related pages: Architecture overview, MR pipeline, Triaging guide, Testing CKI builds externally, Debug triaging.

MR lifecycle: from merge request to CI results, labels, and blocking

This section walks the complete top-down flow, starting from a developer opening a merge request all the way to CI results, MR labels and how the MR can be blocked by CI failures.

A single MR run triggers one parent GitLab pipeline, which spawns multiple downstream pipelines, one per kernel variant checkout (e.g. kernel, kernel-debug, kernel-64k, kernel-automotive). Each downstream pipeline gets its own DW checkout id (redhat:<pipeline_id>, the downstream GitLab pipeline id). Most of the flow below runs per checkout (build -> test -> DW -> notify -> external CI) and repeats for every variant pipeline. Exceptions on the MR itself (once per MR update, not per checkout): the subsystems hook sets Subsystem: and ExternalCI::…::NeedsTesting labels when the MR is opened or updated. The diagram shows one checkout; imagine it repeated N times.

flowchart TD
  MROpen["Developer opens / updates MR on GitLab"]

  subgraph kwf1 [kernel-workflow: on MR event]
    SubsysCompute("subsystems hook\nmatches changed files -> externalci.yaml")
    SubsysLabels["sets Subsystem:<name> labels\nsets ExternalCI::<name>::NeedsTesting\n(only when MR not draft)"]
  end

  subgraph kwfMR [kernel-workflow: MR labels]
    CkiHook(["ckihook: redhat (CKI) origin"])
    CKILabel["sets CKI::OK / CKI::Failed label\nposts pipeline status comment"]
    SubsysRTR(["subsystems hook\n(external-CI origin only)\nGET /api/1/kcidb/nvrs/{nvr}/status?origin=name"])
    ExtCILabel["sets ExternalCI::origin::OK/Failed label\non the MR"]
  end

  subgraph pipeline [CKI pipeline: GitLab multi-project downstream]
    PipelineTrigger["parent pipeline triggered\n(MR push / bot / reviewer)"]
    PipelineFanout["downstream pipeline per variant\ne.g. kernel, kernel-debug,\nkernel-64k, kernel-automotive …"]
    PipelineTrigger --> PipelineFanout
  end

  subgraph perCheckout ["per checkout (×N, one downstream pipeline each)"]
    PipelineBuild["build jobs per architecture\nredhat:<pipeline_id>"]
    PipelineTest["CKI test jobs\nBeaker / Testing Farm"]
    CheckResults["kernel-results: check_results job\ncki.triager --build-id --to-dw (sync)"]
    JobSubmit["each job after_script:\ndatawarehouse_submitter\n(best-effort, progressive)"]
    Herder("pipeline-herder\ninfra retries;\nherder.build when settled")
    subgraph extCI ["externalCI system (one per system: LNST, kqe-tier0, …)"]
      ExtCISub["subscribe"]
      ExtCIPrePost{"pre or post?\ncki_finished"}
      ExtCILabels{"ExternalCI::<origin>::\nNeedsTesting in Kafka?\n(absent while MR draft)"}
      ExtCIRun["runs tests on CKI-built kernel"]
      ExtCISubmit["submits shadow checkout\nvia Kafka cki.results"]
      ExtCISkip["skip"]
      ExtCISub --> ExtCIPrePost
      ExtCIPrePost -->|"false (pre-test)"| ExtCILabels
      ExtCIPrePost -->|"true (post-test)"| ExtCIRun
      ExtCILabels -->|yes| ExtCIRun
      ExtCILabels -->|no| ExtCISkip
      ExtCIRun --> ExtCISubmit
    end
    PipelineBuild --> PipelineTest
    PipelineBuild --> JobSubmit
    PipelineTest --> JobSubmit
    PipelineTest --> CheckResults
    PipelineBuild --> Herder
    PipelineTest --> Herder
  end

  subgraph dwAll [DataWarehouse, per checkout id]
    AmqpBridge("amqp-bridge\nkafka.rx.cki.results.# -> RabbitMQ")
    Submitter("datawarehouse-submitter\nPOST /api/1/kcidb/submit")
    DWStore("DataWarehouse\nstores Checkout -> Build -> Test -> TestResult\nby origin: redhat, lnst, …\npublishes datawarehouse.* events")
    Triager("datawarehouse-triager\nlinks IssueOccurrences, marks triaged")
    BSFCron{{"BuildSetupsFinishedCheckouts cron\nevery 10 min"}}
    BSFGate{"already\nbuild_setups_finished?"}
    BSFEvent["datawarehouse.checkout.build_setups_finished"]
    NotifierPre("cki-notifier: pre-test\nKafka cki.ready-for-test\ncki_finished=false")
    Cron{{"ReadyToReportCheckouts cron\nevery 10 min"}}
    RTRGate{"already\nready_to_report?"}
    RTREvent["datawarehouse.checkout.ready_to_report"]
    ICOEvent["datawarehouse.checkout.checkout_issueoccurrences_changed"]
    NotifierPost("cki-notifier: post-test\nKafka cki.ready-for-test\ncki_finished=true + status")
  end

  DraftReady["cki.kwf.subsystems.checkout.mr_out_of_draft"]

  Reporter(["reporter\n(non-MR checkouts only)"])

  %% MR-wide: labels once per MR update
  MROpen --> SubsysCompute --> SubsysLabels
  SubsysLabels -.->|"merge_request.external_ci\n(GitLab labels at publish time)"| NotifierPre
  SubsysLabels -.->|"no NeedsTesting labels\nwhile MR draft"| ExtCILabels

  MROpen -->|"MR leaves draft"| DraftReady
  DraftReady -.->|"subsystems applies\nExternalCI labels first"| SubsysLabels
  DraftReady --> NotifierPre

  MROpen --> PipelineTrigger
  PipelineFanout --> PipelineBuild

  %% per-checkout: result submission (HTTP API)
  JobSubmit -->|"HTTP POST progressive"| DWStore
  Herder -->|"herder.build"| Submitter
  Submitter -->|"HTTP POST from job artifact"| DWStore

  %% per-checkout: BSF -> pre-test notifier -> Kafka
  DWStore --> BSFCron
  BSFCron --> BSFGate
  BSFGate -->|"no + all setups done"| BSFEvent
  BSFEvent --> NotifierPre
  NotifierPre -->|"Kafka cki.ready-for-test"| ExtCISub

  ExtCISubmit -->|"Kafka cki.results"| AmqpBridge
  AmqpBridge --> Submitter

  %% per-checkout: triage + ready-to-report -> post-test notifier -> Kafka
  DWStore -->|"RabbitMQ async"| Triager
  CheckResults -->|"HTTP sync triage"| Triager
  Triager -->|"issue occurrences\nchanged / waived"| ICOEvent
  Triager --> Cron
  Cron --> RTRGate
  RTRGate -->|"no + finished + triaged"| RTREvent

  %% ready_to_report -> RabbitMQ consumers (same bus; post-test notifier publishes Kafka)
  RTREvent -->|"retry check-kernel-results"| CkiHook
  RTREvent --> SubsysRTR
  RTREvent --> Reporter
  RTREvent --> NotifierPost
  NotifierPost -->|"Kafka cki.ready-for-test"| ExtCISub

  ICOEvent --> NotifierPost
  ICOEvent -->|"retry check-kernel-results"| CkiHook
  ICOEvent --> SubsysRTR

  MROpen -->|"GitLab MR webhook"| CkiHook
  CheckResults -->|"pipeline webhook"| CkiHook
  CkiHook -->|"GitLab API"| CKILabel
  SubsysRTR -->|"GitLab API"| ExtCILabel

  classDef serviceNode fill:#E3F2FD,stroke:#1565C0,color:#1a1a1a
  classDef actionNode fill:#FAFAFA,stroke:#757575,color:#1a1a1a
  classDef cronNode fill:#FFF9C4,stroke:#F9A825,color:#1a1a1a
  class SubsysCompute,CkiHook,SubsysRTR,Herder,AmqpBridge,Submitter,DWStore,Triager,NotifierPre,NotifierPost,Reporter serviceNode
  class MROpen,SubsysLabels,CKILabel,ExtCILabel,PipelineTrigger,PipelineFanout,PipelineBuild,PipelineTest,CheckResults,JobSubmit,ExtCISub,ExtCIRun,ExtCISubmit,ExtCISkip,BSFEvent,RTREvent,ICOEvent,DraftReady actionNode
  class BSFCron,Cron cronNode

  %% RabbitMQ edges (blue): 13,19,23,28,31,32,34,38,39,40,41,43,44,45,46,47
  linkStyle 13 stroke:#0288D1,stroke-width:2px
  linkStyle 19 stroke:#0288D1,stroke-width:2px
  linkStyle 23 stroke:#0288D1,stroke-width:2px
  linkStyle 28 stroke:#0288D1,stroke-width:2px
  linkStyle 31 stroke:#0288D1,stroke-width:2px
  linkStyle 32 stroke:#0288D1,stroke-width:2px
  linkStyle 34 stroke:#0288D1,stroke-width:2px
  linkStyle 38 stroke:#0288D1,stroke-width:2px
  linkStyle 39 stroke:#0288D1,stroke-width:2px
  linkStyle 40 stroke:#0288D1,stroke-width:2px
  linkStyle 41 stroke:#0288D1,stroke-width:2px
  linkStyle 43 stroke:#0288D1,stroke-width:2px
  linkStyle 44 stroke:#0288D1,stroke-width:2px
  linkStyle 45 stroke:#0288D1,stroke-width:2px
  linkStyle 46 stroke:#0288D1,stroke-width:2px
  linkStyle 47 stroke:#0288D1,stroke-width:2px

  %% Kafka edges (orange): 29,30,42
  linkStyle 29 stroke:#E65100,stroke-width:2px
  linkStyle 30 stroke:#E65100,stroke-width:2px
  linkStyle 42 stroke:#E65100,stroke-width:2px

  %% HTTP API edges (green): 21,22,24,33,48,49
  linkStyle 21 stroke:#388E3C,stroke-width:2px
  linkStyle 22 stroke:#388E3C,stroke-width:2px
  linkStyle 24 stroke:#388E3C,stroke-width:2px
  linkStyle 33 stroke:#388E3C,stroke-width:2px
  linkStyle 48 stroke:#388E3C,stroke-width:2px
  linkStyle 49 stroke:#388E3C,stroke-width:2px

Legend

Arrows:

• blue = RabbitMQ message bus (incl. GitLab webhook -> receiver -> RabbitMQ)
• orange = Kafka topic cki.ready-for-test / cki.results (broadcast; each subscriber applies its own filters)
• green = HTTP API / GitLab pipeline trigger
• default (black) = in-process logic
• dashed = GitLab MR metadata / label policy

Nodes:

• rounded blue = CKI/KWF microservice
• square gray = action, state, event, side-effect
• hexagon = cron polling job
• diamond = policy decision (CKI policy or external CI system policy)

Stage-by-stage description

1. MR opened / updated

The GitLab webhook fires to RabbitMQ. The subsystems hook computes which kernel subsystems are touched by the diff and which external CI systems are needed (via external_tests in externalci.yaml). It sets Subsystem:<name> and ExternalCI::<name>::NeedsTesting labels on the MR (plus an overall ExternalCI::NeedsTesting rollup when any external CI applies). Note that draft MRs do not get ExternalCI:: labels until the MR is ready for review; see step 7 for the mr_out_of_draft rebroadcast when that happens. See merge_request.external_ci for how those labels appear in the Kafka payload.

2. CKI pipeline runs (parent -> downstream per variant)

Opening or updating the MR triggers a parent GitLab pipeline (trusted user, bot, or reviewer). That parent spawns one downstream multi-project pipeline per kernel variant (kernel, kernel-debug, kernel-64k, kernel-automotive, …). Each downstream pipeline is a separate DW checkout (redhat:<pipeline_id>); the stages below run in parallel and independently for every checkout. Each checkout holds builds per architecture (e.g. redhat:12345_x86_64_kernel).

Within each checkout, build jobs compile the kernel for each configured architecture; CKI test jobs run suites in Beaker / Testing Farm against those builds.

3. Result submission (progressive + herder fallback, per job)

Pipeline jobs run datawarehouse_upload_kcidb in after_script, which calls datawarehouse-submitter with the local kcidb_all.json artifact. That upload is best-effort and progressive, failures are ignored and the job does not block on DataWarehouse being reachable. DW receives partial updates as the pipeline runs so triage, crons, and the UI can work on in-flight data.

In parallel, pipeline-herder consumes GitLab job webhooks (gitlab.*.build). Its primary role is infra failure handling: it matches known transient failures (image pull, empty trace, missed tests, …) and retries jobs. When a job is settled (no infra retry pending, or a real failure to report), herder publishes herder.build with the job URL. datawarehouse-submitter picks that up, fetches kcidb_all.json from the GitLab job artifact, and POSTs again. This path is the reliable fallback when the in-pipeline upload failed or DW was temporarily unavailable. Most jobs may already be in DW from after_script; herder ensures eventual consistency once the job is really done.

Both paths call the same POST /api/1/kcidb/submit API via datawarehouse-submitter.

4. DW stores and broadcasts (per checkout)

DW saves each checkout’s objects (Checkout -> Build -> Test -> TestResult) in order. A Django post_save signal fires for each save, queuing a datawarehouse.{type}.new or updated message to cki.exchange.webhooks.

MR metadata is embedded in each checkout’s misc.related_merge_request field, injected by the pipeline at build time. When issue occurrences on a checkout change (waive, unwaive, triage update), DW also publishes datawarehouse.checkout.checkout_issueoccurrences_changed on the same exchange (picked up in steps 12–13).

5. Automatic triage (per checkout)

Triage runs the same cki.triager code on two paths:

1. Sync (pipeline): the check_results job in the kernel-results stage (once per architecture build, after tests finish) runs cki.triager --build-id … --to-dw in the pipeline container. Failures are logged and do not fail the job; the async service is the fallback. This is the only triage trigger for CVE pipelines.
2. Async (RabbitMQ): the datawarehouse-triager service consumes every new / updated / needs_triage event published when DW stores objects (step 4). This catches progressive after_script uploads and herder resubmits as they land.

On each path, for every failure triager fetches log files from DW, runs them against IssueRegex rules, links matching IssueOccurrence rows, and marks the object as triaged.

6. Ready-to-report barrier (per checkout)

The ReadyToReportCheckouts cron job polls every 10 minutes for every checkout id separately, regardless of origin. filter_ready_to_report excludes checkouts already ready_to_report and only considers those where every object is both finished and triaged. That avoids re-emitting ready_to_report on every poll (infinite notify loop). On first match the cron sets ready_to_report=True and publishes datawarehouse.checkout.ready_to_report with the checkout id in the body (e.g. redhat:12345 or lnst:12345). If later KCIDB ingestion changes outcomes, DW clears the flag so the checkout can cross the barrier again. Downstream consumers can filter by origin, for example ckihook and cki-notifier act on redhat checkouts, while subsystems acts on external-CI origins listed in owners.yaml (e.g. lnst).

7. Pre-test Kafka notification (per checkout, builds ready)

When all builds in this checkout finish, the BuildSetupsFinishedCheckouts cron (every 10 minutes) emits build_setups_finished to RabbitMQ. cki-notifier reacts immediately, fetches the full checkout from DW, reads labels from the MR via GitLab API, and embeds them in the Kafka message (merge_request.subsystems, merge_request.external_ci, see merge_request.external_ci). It then publishes cki.ready-for-test with cki_finished=false to Kafka. CKI does not route this message to specific consumers, it is a broadcast message. External CI systems (LNST, kqe-tier0, etc.) can subscribe and decide independently whether to act.

8. External CI test run and result submission (per checkout)

External CI systems subscribe to the same Kafka topic; each cki.ready-for-test message carries one checkout’s build_info. Whether they start testing on cki_finished=false, on a later message, or not at all is each external CI system’s policy, not enforced by CKI. In practice many systems (e.g. LNST) act or when builds are ready (cki_finished=false) or when CKI tests are finished (cki_finished=true, maybe even status=success), and require ExternalCI::<origin>::NeedsTesting in merge_request.external_ci before starting a shadow checkout. When they run, they submit results back via Kafka topic cki.results. The amqp-bridge forwards these onto the CKI RabbitMQ exchange (kafka.rx.cki.results.#), where datawarehouse-submitter ingests them as a separate checkout with an external-CI origin (e.g. lnst:12345).

9. External-CI origin results stored and triaged (per shadow checkout)

From here the shadow checkout rejoins the common per-checkout path at step 4 (DW stores and broadcasts), step 5 (automatic triage), and step 6 (ready-to-report barrier). Same mechanics as a redhat checkout; only the checkout origin differs (e.g. lnst:12345 instead of redhat:12345). datawarehouse-submitter ingestion in step 8 feeds step 4; datawarehouse-triager and ReadyToReportCheckouts then run as described there. Step 10 below picks up when that shadow checkout’s ready_to_report fires.

10. ExternalCI labels set (per external-CI checkout -> MR-wide)

Each time an external-CI origin checkout’s ready_to_report fires (e.g. lnst:…), the subsystems hook extracts the origin from the checkout ID and calls GET /api/1/kcidb/nvrs/{nvr}/status?origin=<name>. This request returns the aggregate pass/fail (and readiness) across all variants (all checkouts) for that origin. The hook then sets the appropriate ExternalCI::<origin>::Running/OK/Failed (or _EXPERIMENTAL variant) on the MR.

11. CKI labels and reporting (per redhat checkout -> MR-wide)

ckihook sets CKI::OK / CKI::Failed labels and posts the pipeline status comment on GitLab webhooks. A pipeline event triggers label evaluation when a downstream pipeline or check-kernel-results finishes; MR events re-evaluate while CI is in flight, and note events handle manual request-cki-evaluation retriggers. It evaluates variant pipelines via GitLab GraphQL and enriches the comment with DW failure links. When a redhat origin checkout’s ready_to_report or checkout_issueoccurrences_changed fires, ckihook uses the checkout’s GitLab pipeline URLs (object.misc.provenance) to retry failed results-stage jobs (check-results, kernel-results, or triager); label updates follow when the retried job completes and GitLab emits another pipeline webhook. reporter sends an email report to mailing lists on ready_to_report (skipped for MR-associated checkouts).

12. Post-test Kafka notification (per redhat checkout)

On the same ready_to_report for a redhat origin checkout, cki-notifier publishes cki.ready-for-test with cki_finished=true and status (success / error / fail, from compute_overall_status) to the same Kafka topic as step 7. A later checkout_issueoccurrences_changed event from step 4 may trigger another post-test message when triage changes the overall outcome from fail/error to success (but not the reverse). The same external CI subscribers decide whether to act on this message, for example kqe-tier0 typically filters on cki_finished=true via qe-pipeline-definition / message-trigger, while others may act on cki_finished=false (pre-test).

13. MR blocking and label re-evaluation (MR-wide)

MR blocking is enforced at two levels:

• The CKI::Failed label applied by ckihook prevents the MR from being merged. GitLab merge request rules require the label to be absent (or CKI::OK to be present) before the merge button is enabled.
• The check-kernel-results pipeline job queries DW for the final pipeline status and sets the downstream pipeline to failed or passed accordingly, blocking the parent MR pipeline until results are known.

A CKI::Failed can be resolved by fixing the regression, by the triager linking a known issue (which changes the label to CKI::Waived), or by a manual waive in DW. If an issue is manually waived or unwaived in DW at any point, a checkout_issueoccurrences_changed event re-triggers ckihook and subsystems to re-evaluate labels.

End-to-end data flow

Simplified view of the end-to-end data flow for the MR lifecycle.

flowchart LR
  subgraph external [External]
    GitLabWH("GitLab webhooks")
    KafkaRFT("Kafka cki.ready-for-test")
    KafkaRes("Kafka cki.results")
    KernelCI("KernelCI\npublic DB")
    GitLabMR("GitLab MR\nlabels & comments")
  end

  subgraph ingest [Ingest]
    Jobs["GitLab pipeline jobs"]
    Herder("pipeline-herder\ninfra retries")
    AmqpBridge("amqp-bridge\nkafka.rx.cki.results.#")
    Submitter("datawarehouse-submitter")
  end

  subgraph dw [DataWarehouse]
    DWAPI("REST API\n/api/1/kcidb/submit")
    DWDB("PostgreSQL")
    DWMQ("RabbitMQ\ncki.exchange.webhooks")
  end

  subgraph consumers [Consumers]
    Triager("datawarehouse-triager")
    Kwaier("kwaier")
    Forwarder("datawarehouse-kcidb-forwarder")
    Reporter("reporter")
    Notifier("cki-notifier")
    CkiHook("kernel-workflow\nckihook")
    SubSystems("kernel-workflow\nsubsystems")
  end

  GitLabWH -->|"gitlab.*.build"| Herder
  GitLabWH -->|"MR / note"| SubSystems
  GitLabWH -->|"MR / note / pipeline"| CkiHook

  Jobs -->|"after_script: submitter\n(best-effort, progressive)"| DWAPI
  Herder -->|"herder.build"| Submitter
  KafkaRes --> AmqpBridge
  AmqpBridge --> Submitter
  Submitter -->|"POST /api/1/kcidb/submit"| DWAPI
  DWAPI --> DWDB
  DWDB -->|"Django signals"| DWMQ

  DWMQ --> Triager
  DWMQ --> Kwaier
  DWMQ --> Forwarder
  DWMQ --> Reporter
  DWMQ --> Notifier
  DWMQ --> CkiHook
  DWMQ --> SubSystems

  Forwarder --> KernelCI
  Notifier --> KafkaRFT
  CkiHook --> GitLabMR
  SubSystems --> GitLabMR

  classDef serviceNode fill:#E3F2FD,stroke:#1565C0,color:#1a1a1a
  classDef actionNode fill:#FAFAFA,stroke:#757575,color:#1a1a1a
  classDef externalNode fill:#E8F5E9,stroke:#2E7D32,color:#1a1a1a
  class Herder,AmqpBridge,Submitter,DWAPI,DWDB,DWMQ,Triager,Kwaier,Forwarder,Reporter,Notifier,CkiHook,SubSystems serviceNode
  class Jobs actionNode
  class GitLabWH,KafkaRFT,KafkaRes,KernelCI,GitLabMR externalNode

CKI service catalog

Most CKI microservices are AMQP consumers on cki.exchange.webhooks, each with a dedicated queue and routing key filter. A few also cross the external boundary via Kafka: amqp-bridge ingests cki.results into RabbitMQ, cki-notifier publishes cki.ready-for-test, and external CI systems subscribe on that side.

Ingest services

• datawarehouse-submitter: ingests KCIDB from pipeline jobs (herder.build) and external CI results (kafka.rx.cki.results.#); POSTs to the DW API.
• pipeline-herder: consumes GitLab job webhooks, retries infra failures, publishes herder.build when a job is settled.
• amqp-bridge: bridges Kafka (and other external buses) into the CKI RabbitMQ exchange.

DataWarehouse core

• datawarehouse-webservice: Django REST API, dashboard, and RabbitMQ publisher for datawarehouse.* events on KCIDB saves.
• datawarehouse-webservice-readonly: read-only API replica for ops and debugging.

Post-ingest consumers

• datawarehouse-triager: regex-based auto-triage; links IssueOccurrences and marks objects triaged.
• kwaier: kwai AI analysis on test failures; re-submits enriched KCIDB to DW (feature-flagged).
• datawarehouse-kcidb-forwarder: forwards public checkout data upstream to KernelCI.
• reporter: email reports on ready_to_report (non-MR checkouts only).
• cki-notifier: publishes Kafka cki.ready-for-test for external CI subscribers.

MR consumers (kernel-workflow)

• ckihook: CKI::OK / CKI::Failed labels and pipeline status comments (final labels on GitLab pipeline events; MR/note for in-flight re-evaluation and manual request-cki-evaluation); retries check-kernel-results on DW ready_to_report and checkout_issueoccurrences_changed (redhat origin).
• subsystems: Subsystem: and ExternalCI:: labels on MR open/update; external-origin OK/Failed labels from DW NVR status.

Events: downstream effect matrix

Here is how main events map to downstream effects:

Full event body schema is documented in DataWarehouse microservices.

Triage flow

datawarehouse-triager consumes datawarehouse.{checkout,build,test}.{new,updated,needs_triage} messages from RabbitMQ. The pipeline check_results job runs the same triager synchronously (cki.triager --build-id … --to-dw) once per architecture build in the kernel-results stage; see Architecture overview. This is a fallback mechanism, if the synchronous triage fails for any reason (e.g. datawarehouse momentarily unreachable), the async triager will be triggered to triage the object. Object types are checkout, build, and test only (not testresult directly).

How the triager processes objects

sequenceDiagram
  participant MQ as RabbitMQ
  participant T as triager
  participant DW as DataWarehouse API

  MQ->>T: datawarehouse.{type}.{new|updated|needs_triage}<br/>(object in message body)
  T->>T: download logs from object URLs<br/>(log_url, output_files)
  loop for each IssueRegex
    T->>T: run regex filters
    alt match found
      T->>DW: POST /api/1/kcidb/{type}/{id}/issues
      DW-->>T: 201 Created (IssueOccurrence)
    end
  end
  T->>DW: POST /api/1/kcidb/{type}/{id}/actions/triaged

Re-triage paths

When something changes, the triager is triggered again on one of two paths:

Object changed (pipeline re-ran, new logs): DW publishes new or updated for that object; triager runs all regexes. Old issue links may be cleared first (if CLEAR_ISSUEOCCURRENCES_ON_SAVE is on and the object changed enough).

Regex changed (someone edited an IssueRegex): after a 5-minute debounce, DW re-queues failed objects from the last 15 days with needs_triage; triager runs only the changed regexes and keeps existing links.

Both paths end up as RabbitMQ messages to the triager. The only difference in delivery is message priority: needs_triage uses priority 0 (high) and new/updated use priority 5 (normal). If the triager queue backs up, regex re-triage messages are consumed first. This does not change triage logic, only queue ordering.

flowchart LR
  subgraph objectChange [Object changed]
    A1[DW object save] --> A2["new / updated"] --> A3["triager: all regexes"]
  end

  subgraph regexChange [Regex changed]
    B1[IssueRegex save] --> B2["needs_triage\n(debounced)"] --> B3["triager: changed regex only"]
  end

Ready-to-report barrier

No service directly signals “triage is complete.” The ReadyToReportCheckouts cron job (every 10 minutes) selects checkouts where every object satisfies both finished and triaged (filter_ready_to_report), sets ready_to_report=True, and publishes datawarehouse.checkout.ready_to_report. If later KCIDB ingestion changes outcomes, DW clears the flag so the checkout can cross the barrier again.

Each checkout crosses the barrier on its own schedule; consumers filter by origin prefix in the checkout id (e.g. redhat: vs lnst:).

flowchart LR
  subgraph cron [DW cron: every 10 min]
    Poll["ReadyToReportCheckouts\nfilter_ready_to_report\n+ select_for_update"]
    Set["set ready_to_report=True"]
    Msg["datawarehouse.checkout.ready_to_report"]
    Poll --> Set --> Msg
  end

  subgraph consumers [RabbitMQ consumers]
    Reporter2["reporter\n(non-MR email)"]
    Notifier2["cki-notifier\nKafka post-test notify"]
    CkiHook2["ckihook\n(retry check-kernel-results, CKI:: labels)"]
    SubSystems2["subsystems\n(ExternalCI:: labels)"]
  end

  Msg --> Reporter2
  Msg --> Notifier2
  Msg --> CkiHook2
  Msg --> SubSystems2

Earlier intermediate events from sibling crons (also every 10 minutes, staggered within the window):

kernel-workflow hooks

Both hooks are AMQP consumers on cki.exchange.webhooks (cki.queue.webhooks.kernel-workflow-ckihook and cki.queue.webhooks.kernel-workflow-subsystems).

ckihook: CKI pipeline status labels

ckihook listens to GitLab and DataWarehouse routing keys on the same queue. The two input paths have different effects:

GitLab webhooks (MR open/update, notes, push, pipeline status): on MR, note, or pipeline events for kernel and CKI pipeline projects, ckihook evaluates downstream pipelines via GitLab GraphQL (check-kernel-results / check-results job status per variant), generates scoped labels (CKI::OK, CKI::Failed, CKI::Missing, CKI::Running, CKI::Waived, or per-variant equivalents such as CKI_RT::Missing, CKI_Automotive::Missing, …), applies label changes via the GitLab API, and posts or updates the CKI Pipelines Status comment (failure details link to DW).

DataWarehouse events (ready_to_report, checkout_issueoccurrences_changed): the DataWarehouseEvent handler does not directly set labels. It instead extracts GitLab pipeline URLs from object.misc.provenance (service_name="gitlab") and retries failed results-stage jobs (check-results, kernel-results, or triager) in the associated pipeline. When the retried job finishes, a GitLab pipeline webhook re-triggers ckihook for label evaluation. See README.ckihook.md.

subsystems: ExternalCI and subsystem labels

The DataWarehouseEvent class (kernel-workflow/webhook/session_events.py) filters incoming messages for status ∈ {ready_to_report, checkout_issueoccurrences_changed} and object_type == checkout, then extracts the MR URL from object.misc.related_merge_request.url.

On ready_to_report for an external-CI origin (e.g. lnst, any origin in owners.yaml external_tests, not redhat):

1. Extracts the origin from the checkout ID (e.g. lnst:1234 -> origin lnst).
2. Calls GET /api/1/kcidb/nvrs/{nvr}/status?origin=lnst to aggregate pass/fail across all variants.
3. Sets ExternalCI::<origin>_EXPERIMENTAL::{OK|Failed|Running} on the MR.

On GitLab MR open/update, sets Subsystem:<name> labels and ExternalCI::<name>::NeedsTesting (plus ExternalCI::NeedsTesting) from external_tests path matching, see merge_request.external_ci.

On draft -> ready for review: publishes cki.kwf.subsystems.checkout.mr_out_of_draft so cki-notifier rebroadcasts cki.ready-for-test with is_draft=false.

ExternalCI boundary

Third-party CI systems join the MR pipeline through GitLab labels, Kafka cki.ready-for-test broadcasts, optional shadow testing, and results ingested back into DataWarehouse. The MR lifecycle diagram above covers the full per-checkout flow.

flowchart TB
  NeedsLabels["ExternalCI::NeedsTesting labels\n(subsystems)"]
  OkFailedLabels["ExternalCI:: OK/Failed labels\n(subsystems, from DW NVR status)"]

  NotifierPre("cki-notifier\nKafka cki.ready-for-test\ncki_finished=false")
  NotifierPost("cki-notifier\nKafka cki.ready-for-test\ncki_finished=true + status")

  subgraph extCI ["externalCI system (one per system: LNST, kqe-tier0, …)"]
    ExtCISub["subscribe"]
    ExtCIPrePost{"pre or post?\ncki_finished"}
    ExtCILabels{"ExternalCI::<origin>::\nNeedsTesting in payload?"}
    ExtCIRun["runs tests on CKI-built kernel"]
    ExtCISubmit["submits shadow checkout\nvia Kafka cki.results"]
    ExtCISkip["skip"]
    ExtCISub --> ExtCIPrePost
    ExtCIPrePost -->|"false (pre-test)"| ExtCILabels
    ExtCIPrePost -->|"true (post-test)"| ExtCIRun
    ExtCILabels -->|yes| ExtCIRun
    ExtCILabels -->|no| ExtCISkip
    ExtCIRun --> ExtCISubmit
  end

  NeedsLabels -.->|"merge_request.external_ci\nat publish time"| ExtCILabels
  NotifierPre -->|"Kafka cki.ready-for-test"| ExtCISub
  NotifierPost -->|"Kafka cki.ready-for-test"| ExtCISub
  ExtCISubmit -->|"Kafka cki.results"| OkFailedLabels

  classDef serviceNode fill:#E3F2FD,stroke:#1565C0,color:#1a1a1a
  classDef actionNode fill:#FAFAFA,stroke:#757575,color:#1a1a1a
  class NotifierPre,NotifierPost serviceNode
  class NeedsLabels,OkFailedLabels,ExtCISub,ExtCIRun,ExtCISubmit,ExtCISkip actionNode

merge_request.external_ci

Labels on the MR

The subsystems hook matches changed files against external_tests in owners.yaml / rh_webhooks.yaml. When the MR is not a draft and path (and optional tree/branch) filters match, it applies:

If the diff changes after a prior pipeline, per-origin labels reset to NeedsTesting. Draft MRs do not get ExternalCI:: labels until ready for review. When the MR leaves draft, subsystems applies the labels and publishes cki.kwf.subsystems.checkout.mr_out_of_draft; cki-notifier rebroadcasts cki.ready-for-test with is_draft=false so subscribers see the new labels.

Unlike status labels, membership labels (Subsystem:…, ExternalCI::<name>::NeedsTesting) follow the MR diff and only change when the code changes.

Snapshot in Kafka

cki-notifier reads current GitLab MR labels at send time and embeds them in each cki.ready-for-test message:

• merge_request.external_ci: every label starting with ExternalCI: (raw GitLab strings)
• merge_request.subsystems: same for Subsystem: labels

The snapshot is fixed for that message. Later label changes on the MR are only visible in the next notify (another build_setups_finished, ready_to_report, or mr_out_of_draft).

Subscriber policy

CKI does not route Kafka per consumer. Each external CI system (one box in the diagram) applies its own filters on the broadcast:

• Pre-test (cki_finished=false): many systems require ExternalCI::<your-origin>::NeedsTesting in merge_request.external_ci, plus local rules on tree_name, modified_files, merge_request.subsystems, etc. (e.g. LNST with origin lnst).
• Post-test (cki_finished=true): often ignores NeedsTesting; filters on cki_finished, status, tree_name, Jira links, etc. (e.g. kernel-ci-pipelines via qe-pipeline-definition / message-trigger).

The array lists all current ExternalCI: labels at send time, not only NeedsTesting. After testing, OK, Failed, or Waived labels appear on the MR via subsystems (from DW NVR status) or manual updates, but an already-sent Kafka message is not updated retroactively.

Results loop

When a subscriber runs, it submits a shadow checkout via Kafka cki.results. amqp-bridge forwards to RabbitMQ; datawarehouse-submitter ingests it under an external origin (e.g. lnst:12345). datawarehouse-triager triages the shadow checkout; when it crosses the ready-to-report barrier, subsystems calls GET /api/1/kcidb/nvrs/{nvr}/status?origin=<name> and sets ExternalCI::<origin>::OK/Failed (or _EXPERIMENTAL) on the MR.

Example pre-test payload fragment:

"merge_request": {
  "external_ci": [
    "ExternalCI::NeedsTesting",
    "ExternalCI::lnst::NeedsTesting",
    "ExternalCI::kqe_tier0::NeedsTesting"
  ]
}

Full message schema: Testing CKI builds externally.