cki_tools.datawarehouse_submitter
Submit KCIDB data to DataWarehouse
This page has an internal companion page which might contain additional information.
CKI build and test results are stored as KCIDB formatted data.
The datawarehouse_submitter
listens to herder.build
and
VirtualTopic.eng.cki.results
messages and submits the corresponding results
to DataWarehouse.
The service can be deployed locally by running
python -m cki_tools.datawarehouse_submitter
Environment variables
Name | Secret | Required | Description |
---|---|---|---|
CKI_DEPLOYMENT_ENVIRONMENT |
no | no | Define the deployment environment (production/staging) |
GITLAB_TOKENS |
no | yes | URL/environment variable pairs of GitLab instances and private tokens as a JSON object |
GITLAB_TOKEN |
yes | yes | GitLab private tokens as configured in gitlab_tokens above |
CKI_METRICS_ENABLED |
no | no | Enable prometheus metrics. Default: false |
CKI_METRICS_PORT |
no | no | Port where prometheus metrics are exposed. Default: 8000 |
DATAWAREHOUSE_URL |
no | yes | URL to DataWarehouse. |
DATAWAREHOUSE_TOKEN_SUBMITTER |
yes | yes | Token for DataWarehouse. |
RABBITMQ_HOST |
no | yes | AMQP host |
RABBITMQ_PORT |
no | yes | AMQP port, TLS is used for port 443 |
RABBITMQ_USER |
no | yes | AMQP user |
RABBITMQ_PASSWORD |
yes | yes | AMQP password |
RABBITMQ_CAFILE |
no | yes | AMQP CA file path |
RABBITMQ_CERTFILE |
no | yes | AMQP certificate + private key file path |
WEBHOOK_RECEIVER_EXCHANGE |
no | yes | AMQP exchange to receive messages |
DATAWAREHOUSE_SUBMITTER_QUEUE |
no | yes | AMQP queue name that gets hooked up to the exchange |
DATAWAREHOUSE_SUBMITTER_ROUTING_KEYS |
no | yes | AMQP routing keys for the messages sent to the queue |
Manual triggering
Results can also be submitted manually for a single GitLab job via
CKI_LOGGING_LEVEL=INFO \
CKI_DEPLOYMENT_ENVIRONMENT=production \
python3 -m cki_tools.datawarehouse_submitter --job-url JOB_URL
For directly submitting a kcidb_all.json
file:
CKI_LOGGING_LEVEL=INFO \
CKI_DEPLOYMENT_ENVIRONMENT=production \
python3 -m cki_tools.datawarehouse_submitter --kcidb-file kcidb_all.json