cki_tools.datawarehouse_kcidb_forwarder
Forward KCIDB data upstream into KernelCI database
This page has an internal companion page which might contain additional information.
The datawarehouse_kcidb_forwarder listens for messages from DataWarehouse
with the topics listed below,
discards messages about anything that is restricted or irrelevant for the upstream community,
cleans the payload to the KCIDB format and forwards them to KernelCI database,
which is where everyone testing Linux kernel upstream can share their data.
datawarehouse.checkout.newdatawarehouse.checkout.updateddatawarehouse.build.newdatawarehouse.build.updateddatawarehouse.test.newdatawarehouse.test.updateddatawarehouse.issue.updateddatawarehouse.incident.updated
NOTE
This list of topics is currently defined in the restricted repository:deployment-all/vars.yml
The service can be deployed locally. First install some extra dependencies with:
pip install .[datawarehouse_kcidb_forwarder]
Then start it by running:
python3 -m cki_tools.datawarehouse_kcidb_forwarder
Note that unless you set the proper environment variables to listen for an active queue, it won’t do much, as there are no alternative actions in this tool other than that.
Environment variables
| Name | Secret | Required | Description |
|---|---|---|---|
KCIDB_PROJECT_ID |
no | yes | KCIDB Google Cloud project. 1 |
KCIDB_TOPIC_NAME |
no | yes | KCIDB Submission queue topic. 1 |
GOOGLE_APPLICATION_CREDENTIALS |
yes | yes | KCIDB-provided json with Google Cloud Application credentials. |
MAX_BATCH_SIZE |
no | no | How many objects should be submitted to KCIDB at once. Defaults to 500. |
CKI_METRICS_ENABLED |
no | no | Enable prometheus metrics. Default: false |
CKI_METRICS_PORT |
no | no | Port where prometheus metrics are exposed. Default: 8000 |
RABBITMQ_HOST |
no | yes | AMQP host |
RABBITMQ_PORT |
no | yes | AMQP port, TLS is used for port 443 |
RABBITMQ_USER |
no | yes | AMQP user |
RABBITMQ_PASSWORD |
yes | yes | AMQP password |
RABBITMQ_CAFILE |
no | yes | AMQP CA file path |
RABBITMQ_CERTFILE |
no | yes | AMQP certificate + private key file path |
WEBHOOK_RECEIVER_EXCHANGE |
no | yes | AMQP exchange to receive messages |
DATAWAREHOUSE_KCIDB_FORWARDER_QUEUE |
no | yes | AMQP queue name that gets hooked up to the exchange |
DATAWAREHOUSE_KCIDB_FORWARDER_ROUTING_KEYS |
no | yes | AMQP routing keys for the messages sent to the queue |
-
The values are listed in KCIDB’s Submitters’ guide. ↩︎ ↩︎