Microservices

DataWarehouse provides a way to let external services KCIDB objects updates.

Design

To communicate between the DataWarehouse and the services running, RabbitMQ is used as the message broker.

DataWarehouse publishes all the events described on this page to the cki.exchange.datawarehouse exchange, which is forwarded to cki.exchange.webhooks. Services that want to consume these messages must create a queue and subscribe to this exchange. Using the cki-lib MessageQueue implementation is strongly recommended.

                 application
              +---------------+
              | DataWarehouse |
              +-------+-------+
                      |
              publish | exchange
        +-------------+--------------+
        + cki.exchange.datawarehouse |
        +-------------+--------------+
                      |
                      v
          +-----------+-----------+
+---------+ cki.exchange.webhooks |
|         +-----------------------+
|
|
|               queues                     services
|  +--------------------------------+    +-----------+
+->+ cki.queue.webhooks.{service_1} +--->+ service_1 |
|  +--------------------------------+    +-----------+
|  +--------------------------------+    +-----------+
+->+ cki.queue.webhooks.{service_2} +--->+ service_2 |
|  +--------------------------------+    +-----------+
.
.
|  +--------------------------------+    +-----------+
+->+ cki.queue.webhooks.{service_n} +--->+ service_n |
   +--------------------------------+    +-----------+

Events

Different events are broadcasted over this method:

New new

This event is sent when a new object is received by the DataWarehouse.

Needs Triage needs_triage

This message indicates that an object needs to be triaged. These are sent when a change is done on the Issue Regexes, and the last 2 weeks objects are requeued to be analyzed for failures.

Ready to Report ready_to_report

This message indicates that the conditions to report this object were met. Note: The conditions under which this message needs to be sent are still undefined.

Updated updated

This message is sent when a existing object is modified by the DataWarehouse.

Build setups finished build_setups_finished

This message is sent when all the setup for all the builds of a checkout are finished. Note: This event always relates to a checkout.

Tests finished tests_finished

This message is sent when all the tests for a checkout are finished. Note: This event always relates to a checkout.

Issue occurrences changed within a Checkout checkout_issueoccurrences_changed

This message is sent when any issue occurrence is created or removed. Note: This event always relates to a checkout.

Messages

The body of the message contains a JSON payload as it follows:

message = {
    'timestamp': 'Timestamp of the message in ISO format',
    'status': '(new|needs_triage|ready_to_report|..)'
    'object_type': '(checkout|build|test)',
    'object': { the object itself },
}

Check Checkout, Build and Test API docs to get more information.

Routing keys

Messages are sent with a routing key of datawarehouse.{object_type}.{status}.

Acking messages

The action of acking a message should only be done when the service finished processing the pipeline. This allows that if a failure happens while working with the message, the message will be still available on the queue the next time.