cki_tools.datawarehouse_submitter

Submit KCIDB data to DataWarehouse

This page has an internal companion page which might contain additional information.

CKI build and test results are stored as KCIDB formatted data.

The datawarehouse_submitter listens to herder.build and VirtualTopic.eng.cki.results messages and submits the corresponding results to DataWarehouse.

The service can be deployed locally by running

python -m cki_tools.datawarehouse_submitter

Environment variables

Name	Secret	Required	Description
`CKI_DEPLOYMENT_ENVIRONMENT`	no	no	Define the deployment environment (production/staging)
`GITLAB_TOKENS`	no	yes	URL/environment variable pairs of GitLab instances and private tokens as a JSON object
`GITLAB_TOKEN`	yes	yes	GitLab private tokens as configured in `gitlab_tokens` above
`CKI_METRICS_ENABLED`	no	no	Enable prometheus metrics. Default: false
`CKI_METRICS_PORT`	no	no	Port where prometheus metrics are exposed. Default: 8000
`DATAWAREHOUSE_URL`	no	yes	URL to DataWarehouse.
`DATAWAREHOUSE_TOKEN_SUBMITTER`	yes	yes	Token for DataWarehouse.
`RABBITMQ_HOST`	no	yes	AMQP host
`RABBITMQ_PORT`	no	yes	AMQP port, TLS is used for port 443
`RABBITMQ_USER`	no	yes	AMQP user
`RABBITMQ_PASSWORD`	yes	yes	AMQP password
`RABBITMQ_CAFILE`	no	yes	AMQP CA file path
`RABBITMQ_CERTFILE`	no	yes	AMQP certificate + private key file path
`WEBHOOK_RECEIVER_EXCHANGE`	no	yes	AMQP exchange to receive messages
`DATAWAREHOUSE_SUBMITTER_QUEUE`	no	yes	AMQP queue name that gets hooked up to the exchange
`DATAWAREHOUSE_SUBMITTER_ROUTING_KEYS`	no	yes	AMQP routing keys for the messages sent to the queue

Configuration

Example:

batch_size: 100  # number of results per submission
web_url:         # allow list of URL prefixes
  - https://gitlab.instance/group/project/

Manual triggering

Results can also be submitted manually for a single GitLab job via

CKI_LOGGING_LEVEL=INFO \
  CKI_DEPLOYMENT_ENVIRONMENT=production \
  python3 -m cki_tools.datawarehouse_submitter --job-url JOB_URL

For directly submitting a kcidb_all.json file:

CKI_LOGGING_LEVEL=INFO \
  CKI_DEPLOYMENT_ENVIRONMENT=production \
  python3 -m cki_tools.datawarehouse_submitter --kcidb-file kcidb_all.json