Microservices

DataWarehouse provides a way to let external services KCIDB objects updates.

Design

To communicate between the DataWarehouse and the services running, RabbitMQ is used as the message broker.

DataWarehouse publishes all the events described on this page to the cki.exchange.datawarehouse exchange, which is forwarded to cki.exchange.webhooks. Services that want to consume these messages must create a queue and subscribe to this exchange. Using the cki-lib MessageQueue implementation is strongly recommended.

                 application
              +---------------+
              | DataWarehouse |
              +-------+-------+
                      |
              publish | exchange
        +-------------+--------------+
        + cki.exchange.datawarehouse |
        +-------------+--------------+
                      |
                      v
          +-----------+-----------+
+---------+ cki.exchange.webhooks |
|         +-----------------------+
|
|
|               queues                     services
|  +--------------------------------+    +-----------+
+->+ cki.queue.webhooks.{service_1} +--->+ service_1 |
|  +--------------------------------+    +-----------+
|  +--------------------------------+    +-----------+
+->+ cki.queue.webhooks.{service_2} +--->+ service_2 |
|  +--------------------------------+    +-----------+
.
.
|  +--------------------------------+    +-----------+
+->+ cki.queue.webhooks.{service_n} +--->+ service_n |
   +--------------------------------+    +-----------+

Events

Different events are broadcasted over this method:

New new

This event is sent when a new object is received by the DataWarehouse.

Needs Triage needs_triage

This message indicates that an object needs to be triaged. These are sent when a change is done on the Issue Regexes, and the last 2 weeks objects are requeued to be analyzed for failures.

Ready to Report ready_to_report

This message is emitted when a checkout satisfies the “ready to report” rules defined in filter_ready_to_report in DataWarehouse:

• Not be already tagged ready_to_report (ready_to_report is False).
• Checkout valid != None (finished) and last_triaged_at != None (triaged).
• If Builds exist, all Builds valid != None (finished) and last_triaged_at != None (triaged), additionally, if there are no failed builds, there’s also no builds where test_plan_missing is True.
• If Test exist, all Tests status != None (finished) and last_triaged_at != None (triaged).

At that moment the cron job sets the checkout’s ready_to_report to true and publishes the message.

If KCIDB ingestion later overwrites outcome fields (valid or status), DataWarehouse clears the ready_to_report flag back to false on the checkout so the job can evaluate the checkout again and, if appropriate, send another ready_to_report message after re-evaluation.

Updated updated

This message is sent when a existing object is modified by the DataWarehouse.

Build setups finished build_setups_finished

This message is sent when all the setup for all the builds of a checkout are finished. Note: This event always relates to a checkout.

Tests finished tests_finished

This message is sent when all the tests for a checkout are finished. Note: This event always relates to a checkout.

Issue occurrences changed within a Checkout checkout_issueoccurrences_changed

This message is sent when any issue occurrence is created or removed. Note: This event always relates to a checkout.

Messages

The body of the message contains a JSON payload as it follows:

message = {
    'timestamp': 'Timestamp of the message in ISO format',
    'status': '(new|needs_triage|ready_to_report|..)'
    'object_type': '(checkout|build|test)',
    'object': { the object itself },
}

Check Checkout, Build and Test API docs to get more information.

Routing keys

Messages are sent with a routing key of datawarehouse.{object_type}.{status}.

Acking messages

The action of acking a message should only be done when the service finished processing the pipeline. This allows that if a failure happens while working with the message, the message will be still available on the queue the next time.