cki_tools.datawarehouse_submitter
Submit KCIDB data to DataWarehouse
CKI build and test results are stored as KCIDB formatted data.
The datawarehouse_submitter
listens to herder.build
and
VirtualTopic.eng.cki.results
messages and submits the corresponding results
to DataWarehouse.
The service can be deployed locally by running
python -m cki_tools.datawarehouse_submitter
Environment variables
Name | Secret | Required | Description |
---|---|---|---|
CKI_DEPLOYMENT_ENVIRONMENT |
no | no | Define the deployment environment (production/staging) |
GITLAB_TOKENS |
no | yes | URL/environment variable pairs of GitLab instances and private tokens as a JSON object |
GITLAB_TOKEN |
yes | yes | GitLab private tokens as configured in gitlab_tokens above |
CKI_METRICS_ENABLED |
no | no | Enable prometheus metrics. Default: false |
CKI_METRICS_PORT |
no | no | Port where prometheus metrics are exposed. Default: 8000 |
DATAWAREHOUSE_URL |
no | yes | URL to DataWarehouse. |
DATAWAREHOUSE_TOKEN_SUBMITTER |
yes | yes | Token for DataWarehouse. |
RABBITMQ_HOST |
no | yes | AMQP host |
RABBITMQ_PORT |
no | yes | AMQP port, TLS is used for port 443 |
RABBITMQ_USER |
no | yes | AMQP user |
RABBITMQ_PASSWORD |
yes | yes | AMQP password |
RABBITMQ_CAFILE |
no | yes | AMQP CA file path |
RABBITMQ_CERTFILE |
no | yes | AMQP certificate + private key file path |
WEBHOOK_RECEIVER_EXCHANGE |
no | yes | AMQP exchange to receive messages |
DATAWAREHOUSE_SUBMITTER_QUEUE |
no | yes | AMQP queue name that gets hooked up to the exchange |
DATAWAREHOUSE_SUBMITTER_ROUTING_KEYS |
no | yes | AMQP routing keys for the messages sent to the queue |
Configuration
Example:
batch_size: 100 # number of results per submission
web_url: # allow list of URL prefixes
- https://gitlab.instance/group/project/
Manual triggering
Results can also be submitted manually for a single GitLab job via
CKI_LOGGING_LEVEL=INFO \
CKI_DEPLOYMENT_ENVIRONMENT=production \
python3 -m cki_tools.datawarehouse_submitter --job-url JOB_URL
For directly submitting a kcidb_all.json
file:
CKI_LOGGING_LEVEL=INFO \
CKI_DEPLOYMENT_ENVIRONMENT=production \
python3 -m cki_tools.datawarehouse_submitter --kcidb-file kcidb_all.json