cki.kcidb.datawarehouse_submitter

The datawarehouse_submitter listens to a message queue, which receives a notification every time a job successfully passes through the pipeline-herder. It then retrieves the corresponding results and submits them to the DataWarehouse.

The results are stored in an artifact called kcidb_all.json using the KCIDB format schemed data.

It can also be used to manually retrieve and optionally submit the results from a single job or every job in a single pipeline.

SYNOPSIS

python3 -m cki.kcidb.datawarehouse_submitter [-h]
    [--push]
    [--datawarehouse-url  DATAWAREHOUSE_URL]
    [--datawarehouse-token  DATAWAREHOUSE_TOKEN]

    [--output-file OUTPUT_FILE]

    [--output-stdout]
    {queue,single} ...

• --push: Push output data to Datawarehouse. Defaults to True on production.
• --datawarehouse-url DATAWAREHOUSE_URL: URL to the Datawarehouse instance. Defaults to env DATAWAREHOUSE_URL.
• --datawarehouse-token DATAWAREHOUSE_TOKEN: Token for the Datawarehouse instance. Defaults to env DATAWAREHOUSE_URL.
• --output-file OUTPUT_FILE: Save output to a file.
• --output-stdout: Echo output to stdout. Defaults to False on production.

Single

Parse a single job or pipeline:

python3 -m cki.kcidb.datawarehouse_submitter single [-h]
    {pipeline,job} gitlab_url project id

• {pipeline,job}: choose between pipeline or job
• gitlab_url: Gitlab host URL.
• project: Gitlab project. Id or url-encoded path.
• id: Gitlab pipeline or job id.

Queue

Monitor queue for new finished jobs:

python3 -m cki.kcidb.datawarehouse_submitter queue [-h]
    [--rabbitmq-exchange RABBITMQ_EXCHANGE]
    [--rabbitmq-queue RABBITMQ_QUEUE]
    [--rabbitmq-host RABBITMQ_HOST]
    [--rabbitmq-port RABBITMQ_PORT]
    [--rabbitmq-user RABBITMQ_USER]
    [--rabbitmq-password RABBITMQ_PASSWORD]

Environment variables