cki.deployment_tools.gitlab_runner_config

Ensure a managed and reproducible gitlab-runner setup

CLI interface

python3 -m cki.deployment_tools.gitlab_runner_config \
    [-c CONFIG] [options] OBJECT ACTION

A gitlab-runner setup can be quite nicely split into the following distinct objects that can be managed separately:

  • configurations: gitlab-runner configurations, i.e. the config.toml files used to configure the gitlab-runner processes, optionally embedded in a Kubernetes Secret
  • registrations: gitlab-runner registrations with GitLab projects or groups
  • activations: control which gitlab-runner registrations are active/paused
  • variables: CI/CD variables for GitLab projects or groups
  • webhooks: webhooks for GitLab projects

The following actions can be performed on them:

  • dump: get the currently deployed setup, and output it in YAML format
  • generate: generate the new setup from the configuration, and output it in YAML format
  • diff: show the difference between the currently deployed and the newly generated setup
  • apply: adjust the deployed setup to match the configuration

To register new runners with a GitLab instance, the --create-missing parameter must be specified. The printed runner authorization token must be added to the runner_tokens field of the configuration file. It is considered a secret and cannot be obtained from the GitLab instance later on! To add the token to a secrets store, provide a helper command via --add-token-helper HELPER that will get token name and value passed via the command line as positional parameters.

Configuration file

The desired gitlab-runner setup is described by a YAML configuration file. In the following sections, the configuration sections are described based on a non-trivial example which needs to be preprocessed as a Jinja template.

The example describes a gitlab-runner setup across two GitLab instances, gitlab.com and gitlab.corp.com. On the first (public) instance, all projects in the external-group group should have access to a set of gitlab runners. On the second (internal) instance, all projects in the internal-group group should have access to a set of identically configured gitlab runners. Three runners should be configured: two in an OpenShift 3 project (high and normal), and one privileged docker runner for building container images. There is one project project1 in the internal group that needs a slightly different setup. For this project, the OpenShift runners should be deployed into a different OpenShift 4 project. Additionally, this project should have access to a password via a CI/CD variable.

runner_tokens

This dictionary contains the runner authentication tokens indexed by the generated full name.

Example:

runner_tokens:
  com-ocp3-kubernetes-high: {{ COM_OCP3_KUBERNETES_HIGH_RUNNER_TOKEN }}
  com-ocp3-kubernetes-normal: {{ COM_OCP3_KUBERNETES_NORMAL_RUNNER_TOKEN }}
  com-server-docker-priv: {{ COM_SERVER_DOCKER_PRIV_RUNNER_TOKEN }}
  corp-ocp3-kubernetes-high: {{ CORP_OCP3_KUBERNETES_HIGH_RUNNER_TOKEN }}
  corp-ocp3-kubernetes-normal: {{ CORP_OCP3_KUBERNETES_NORMAL_RUNNER_TOKEN }}
  corp-server-docker-priv: {{ CORP_SERVER_DOCKER_PRIV_RUNNER_TOKEN }}
  project1-ocp4-kubernetes-normal: {{ PROJECT1_OCP4_KUBERNETES_NORMAL_RUNNER_TOKEN }}
  project1-server-docker-priv: {{ PROJECT1_SERVER_DOCKER_PRIV_RUNNER_TOKEN }}

gitlab_instances

This dictionary contains general information and access tokens for the configured GitLab instances.

The key is an abbreviation for the instance that is used in other places in the configuration and in the automatically generated names of the runners.

The value is a dictionary with the following fields:

Field Type Required Description
url string yes GitLab instance URL
api_token string yes GitLab personal access token with maintainer/owner access on the configured projects/groups

Example:

gitlab_instances:
  com:
    url: https://gitlab.com/
    api_token: {{ COM_GITLAB_TOKEN_ADMIN_BOT }}
  corp:
    url: https://gitlab.corp.com/
    api_token: {{ CORP_GITLAB_TOKEN_ADMIN_BOT }}

runner_deployments

This dictionary contains information about configured gitlab-runner deployments. One deployment corresponds to one config.toml, i.e. one gitlab-runner process. Each gitlab-runner can have multiple configurations specified in runner_configurations.

The key is an abbreviation for the deployment that is used in other places in the configuration and in the automatically generated names of the runners.

The value is a dictionary used for templating the global keys of the config.toml file.

Additionally, it can contain the following fields:

Field Type Required Description
.configfile filepath (yes) Filename for the gitlab-runner configuration, e.g. config.toml (conflicts with .secret)
.secret filepath (yes) Filename for a Kubernetes Secret, e.g. secret.yaml (conflicts with .configfile)
.filename filename (yes) Filename within the Secret (only with .secret)
.metadata string no Meta data for the Secret (only with .secret)
.template_overrides dict no Runner configuration entries that will only be applied to this deployment

Example:

runner_deployments:
  .default:
    check_interval: 0
    log_level: info
  .kubernetes:
    .filename: config.toml
    .metadata:
      labels:
        app: {{ PROJECT_NAME }}
      name: {{ PROJECT_NAME }}
    concurrent: 100
  ocp3:
    .extends: .kubernetes
    .secret: ocp3-secret.yaml
    .template_overrides:
      kubernetes:
        namespace: {{ OPENSHIFT_PROJECT }}
  ocp4:
    .extends: .kubernetes
    .secret: ocp4-secret.yaml
    .template_overrides:
      kubernetes:
        namespace: {{ OPENSHIFT4_PROJECT }}
  server:
    .configfile: server-config.toml
    concurrent: 10

runner_templates

This dictionary contains templates for gitlab-runner configurations, i.e. for a [runner] section of a config.toml. gitlab-runner process.

The key is an abbreviation for the runner name that is used in other places in the configuration and in the automatically generated names of the runners.

The value is a dictionary used for templating the keys of a [runner] section of a config.toml file.

Additionally, it can contain the following fields:

Field Type Required Description
description string no Executor description (only for documentation)
rationale string no Executor rationale (only for documentation)
.tag_list list no List of gitlab-runner tags
.cache string no Deployment-all style bucket specification for the distributed runner cache in S3
.access_level string no The access_level of the runner; not_protected or ref_protected (default not_protected)
.locked bool no Whether the Runner should be locked for current project (default true)
.maximum_timeout int no Maximum timeout set when this Runner will handle the job in seconds (default 1 week)
.run_untagged bool no Whether the Runner should handle untagged jobs (default false)
machine/.MachineOptions dict no Docker machine configuration that gets converted into the key=value array needed for machine/MachineOptions
machine/.MachineOptions/.amazonec2-tags dict no AWS EC2 tags that get converted into the key,value1,key,value2 string needed for machine/MachineOptions/amazonec2-tags, can also be provided as YAML/JSON string
.environment dict no Environment variables that get converted into the key=value array needed for environment

Example:

runner_templates:
  .default:
    .cache: {{ BUCKET_RUNNER_CACHE }}
  .docker:
    executor: docker
    docker:
      tls_verify: false
  .kubernetes:
    executor: kubernetes
    kubernetes:
      privileged: false
  docker-priv:
    .extends: .docker
    .tag_list: image-builder
    docker:
      privileged: true
  kubernetes-high:
    .extends: .kubernetes
    .tag_list: build
    limit: 4
    kubernetes:
      cpu_request: "20"
      cpu_limit: "20"
      memory_request: 8Gi
      memory_limit: 8Gi
  kubernetes-normal:
    .extends: .kubernetes
    .tag_list: test
    limit: 20
    kubernetes:
      cpu_request: "4"
      cpu_limit: "4"
      memory_request: 4Gi
      memory_limit: 4Gi

variable_groups

This dictionary contains environment variables that can be configured as trigger variables on a GitLab group or project or in the environment key for a gitlab-runner configuration.

The key is an abbreviation for the variable group that is used in other places in the configuration.

The value is a dictionary of environment variables.

Example:

variable_groups:
  home:
    HOME: /tmp
  group-default:
    DEFAULT_EMAIL: email@example.com
  project1:
    .extends: group-default
    GITLAB_PASSWORD: {{ GITLAB_PASSWORD }}

runner_configurations

This dictionary contains groups of runners that are configured with the same variables and their registration with GitLab groups and projects. Those runners can be part of multiple gitlab-runner deployments.

The key is an abbreviation for the configuration that is used in the automatically generated names of the runners.

The value is a dictionary with the following fields:

Field Type Required Description
runner_deployments dict yes Mapping from runner deployments to list of instantiated runner templates
variable_group string no Environment variables set in the environment field of the corresponding [runner] sections
active bool no Whether the runners should be active on creation (default true)
gitlab_groups list (yes) GitLab groups where runners should be registered
gitlab_projects list (yes) GitLab projects where runners should be registered

Environment variables from the runner template override values from the variable group.

Example:

runner_configurations:
  com:
    runner_deployments:
      ocp3:
        - kubernetes-high
        - kubernetes-normal
      server:
        - docker-priv
    variable_group: home
    gitlab_groups: com/external-group
  corp:
    runner_deployments:
      ocp3:
        - kubernetes-high
        - kubernetes-normal
      server:
        - docker-priv
    variable_group: home
    gitlab_groups: corp/internal-group
  project1:
    runner_deployments:
      ocp4:
        - kubernetes-normal
      server:
        - docker-priv
    variable_group: home
    gitlab_projects: corp/internal-group/project1

gitlab_variables

This list contains the assignment of variable groups to GitLab groups and projects. In other words, this list configures the project/group CI/CD variables.

Each item in the list is a dictionary with the following fields:

Field Type Required Description
variable_group list yes Variables that should be set in the CI/CD variables of the GitLab groups or projects
gitlab_groups list (yes) GitLab groups where variables should be set
gitlab_projects list (yes) GitLab projects where variables should be set

Example:

gitlab_variables:
  - gitlab_groups:
      - corp/internal-group
      - com/external-group
    variable_group: group-default
  - gitlab_projects:
      - corp/internal-group/project1
    variable_group: project1

webhook_endpoints

This list contains the assignment of webhook endpoints to GitLab projects.

Each item in the list is a dictionary with the following fields:

Field Type Required Description
url string yes The hook URL
.pattern regex no Regular expression for the URL to match a hook, defaults to the URL
.secret_token string yes Secret token to validate received payloads
.routing_keys list yes space-separated lists of webhook-receiver routing keys, see below
.active bool no when false, all events are removed, defaults to true
enable_ssl_verification bool no Do SSL verification when triggering the hook, defaults to true

Example:

webhook_endpoints:
  - url: https://webhook.endpoint/gitlab
    .pattern: https://(webhook.endpoint|old.endpoint)/gitlab
    .secret_token: {{ WEBHOOK_SECRET }}
    .routing_keys:
      - {{ CONSUMER1_ROUTING_KEYS }}
      - {{ CONSUMER2_ROUTING_KEYS }}
      - gitlab.corp.com.internal-group.project1.build
      - gitlab.corp.com.internal-group.#.deployment

The routing keys are analyzed per GitLab project and webhook endpoint. Only the referenced events will be enabled. To configure group events, use routing keys with * (without subgroups) or # wildcards (with subgroups).

Preprocessing

The following preprocessing steps are performed (in order) when reading the configuration.

Inheritance

For some dictionaries, basic inheritance and defaults modeled after the default and extends keywords in the GitLab CI/CD job descriptions can be used.

  • items with a key starting with a dot (.) are not processed, but can be used for inheritance
  • the .default item will be inherited by all other items
  • the .extends key followed by a name or list of names specifies items to inherit from
  • if the items are directly used for templating, items with a key starting with a dot (.) are removed beforehand

This is enabled for runner_templates, runner_deployments and variable_groups.