cki_tools.pipeline_herder

Configuration via environment variables

RabbitMQ setup

The herder will delay the restart of jobs via RabbitMQ dead-letter queues. This needs to be setup as described in the resilient message queue documentation.

Checking a single job

It is possible to run all matchers against a single job to see whether anything matches by specifying the job URL via

python3 -m cki_tools.pipeline_herder.main \
    --job-url https://instance/project/-/jobs/012345

Analyzing coverage

The batch_check module can be used to analyze coverage of successfully recovered jobs. In other words, jobs that failed and were restarted successfully should be detected by the herder.

First, the database of jobs in ~/.cache/pipeline-herder/ needs to be populated per project with

python3 -m cki_tools.pipeline_herder.batch_check \
    --project-url PROJECT_URL \
    --private-token PRIVATE_TOKEN \
    --days DAYS \
    --update

Next, the current coverage can be analyzed with --check.

Interactive review of recovered/failed jobs

Traces of recovered jobs can be interactively reviewed with --review. Ad-hoc regular expressions can be added that can later be integrated into the matchers module. This mode stores a configuration file in ~/.cache/pipeline-herder/review.yaml.

In the main loop, a table is shown that displays the currently available matchers/expressions together with the number of matching recovered jobs (true positive) and matching failed jobs (false positive).

Generally speaking, the number of matching failed jobs should be kept close to zero. These jobs were not restarted (or at least not successfully), and therefore represent jobs that should not be touched by the herder. Any false positives for a matcher/expression should be inspected to make sure that jobs are not needlessly restarted by the herder. This is especially important for test jobs where a restart might take a long time.

The following commands are accepted:

• q: quit the main loop
• [tf]123: load all traces for true/false positive jobs for a given matcher/expression in gvim
• n: open all traces of the next missed recovered job in gvimdiff, and allow the addition of regular expressions

When inspecting the next missed recovered job, new expressions can be entered that are directly tested on all missed job traces. An expression can be stored in the configuration file with y. If the job failed because of an issue that cannot be solved by a restart, the job can be ignored with i so it will not show up again.

Supported failure conditions

Pod resource exhaustion (exit code 137)

Pods might get terminated prematurely with exit code 137. Most of the times, this indicates that the pod has exhausted its ephemeral storage. One example where this can happen is during kernel RPM building. In this case, the job can be retried directly, hopefully moving it to another node. A proper fix would be to provide a larger scratch NFS volume.

Prometheus Metrics

If CKI_METRICS_ENABLED is true, Prometheus metrics are exposed on the CKI_METRICS_PORT port.

The exposed data is the following: