cki_tools.credentials

Manage credentials for service accounts and their metadata. Two CLI tools provide access (secrets) and lifecycle management (manager). Metadata is stored in a git-tracked YAML file; actual secret values are stored in a pluggable backend (currently HashiCorp Vault, recorded per-secret as backend: hv).

Where a service supports ephemeral credentials — OIDC federation, JWT-based delegated auth, dynamic secret issuance — prefer those; no rotation machinery is needed. This system is for long-lived shared credentials where rotation is unavoidable.

Core concepts

Secrets file and backends

Every secret has a YAML entry in the secrets file (pointed to by CKI_SECRETS_FILE). This entry holds metadata — token type, timestamps, flags — but never the secret value itself. Secret values live in a backend; the backend is recorded in the YAML:

MY_TOKEN/1:
  backend: hv
  meta:
    token_type: ssh_private_key
    active: true
    deployed: true
    created_at: '2025-01-22T10:54:04+00:00'
    comment: my-server
    key_size: 4096

Currently the only backend is HashiCorp Vault (hv). The backend is set automatically when a secret value is written.

Token groups and versions

Secrets are organized as token groups with versions. MY_TOKEN is the group; MY_TOKEN/1 is a versioned entry. The version string is arbitrary — the rotation tooling generates unix timestamps, but a simple index like 1 works fine. Best practice is to always use a version, even for single-version secrets (MY_TOKEN/1 rather than the bare MY_TOKEN).

active and deployed flags

Every version carries two boolean metadata flags:

• active — the token is valid on the remote service (not revoked/expired).
• deployed — this is the version currently consumed by services.

Exactly one version per group must be deployed at any time.

Provider / consumer pattern

When deploying rotatable secrets, two perspectives matter:

• Provider (the service using the credential to authenticate): should use the deployed version.
• Consumer (the service accepting the credential): should trust all active versions.

This enables zero-downtime rotation: a new version is created and accepted before the provider switches to it.

Lifecycle models

Multiple workloads consume the same credential. They load it at startup, cache it in memory, or refresh on their own schedule — there is no deterministic moment when all consumers have switched to a new value. Revoking the old credential immediately after creating a replacement breaks any consumer that has not yet switched.

The tooling addresses this by ensuring overlapping validity during rotation: old and new credentials both work during the cutover window. It does this either by explicitly keeping both versions active or by relying on the service’s built-in grace period. It does not require application-level retries or coordinated rollouts/restarts as part of the rotation contract; those remain optional operational complements.

The specific rotation model depends on what the remote service supports. The key differences are how many versions coexist in steady state (1 or 2) and how many steps a rotation takes (1, 2, or 3).

Two-slot rotation (no native rotation)

The service has no rotation API — creating a new token does not touch existing ones (e.g. SSH keys, AWS keys, deploy tokens). In steady state only one version exists. Rotation temporarily creates a second version: first create a new non-deployed version, then flip which one is deployed, then revoke the old one — 3 steps across 3 commits. Between the first and last step, both versions are active and consumers must trust both.

The manager CLI automates these as prepare, switch, and clean.

Service-side rotation (native rotation API)

The service provides an atomic rotation API that replaces a token in a single call — the old token is invalidated and a new one is returned (e.g. GitLab rotate() API). Because the call immediately invalidates the old token, you cannot rotate the deployed version directly — that would kill the live token before services can switch. Instead, two versions are kept in steady state: one deployed (live), one spare. Rotate the spare (which atomically revokes it and creates a new one), then flip deployed to the new version. Because the atomic call already revoked the old spare, no separate revocation step is needed — 2 steps across 2 commits.

The manager CLI automates these as prepare and switch.

Grace-period rotation

Like service-side rotation, the service replaces the token in a single call — but unlike it, the old key remains valid for a configured grace period (e.g. Testing Farm keeps old keys valid for 7 days). This removes the constraint that blocked rotating the live token: since the old one still works, you can rotate the deployed version directly. No spare slot is needed — one version in steady state, single step. The new version must be deployed before the grace period expires.

The manager CLI automates this as prepare.

Token age and rotation window

Tokens should be rotated before they expire. A common default is 365 days, with rotation flagged 30 days before the limit to give a buffer. The manager CLI exposes this via --token-age (default: 365 days).

Token types (managed vs unmanaged)

Managed secrets have a token_type in their metadata that determines which lifecycle operations the manager CLI can automate (create, destroy, rotate, update, validate) and what metadata fields are required — see the Manager CLI reference for the full list. Unmanaged secrets have no token type; they follow the same rotation patterns but each step is performed manually with cki_edit_secret.

How-to guides

Reading secrets and metadata

Retrieve a secret value (printed as a raw string, no quoting):

$ cki_secret MY_TOKEN/1
s3cr3t-value

Retrieve metadata — a single field returns its value, an empty field (#) returns all metadata as key-value pairs, one per line:

$ cki_secret MY_TOKEN/1#token_type
ssh_private_key
$ cki_secret MY_TOKEN/1#
ssh_private_key
True
True
2025-01-22T10:54:04+00:00
...

Retrieve the raw secret data from the backend — same pattern with ::

cki_secret MY_TOKEN/1:private_key    # single field
cki_secret MY_TOKEN/1:               # all key-value pairs

By default, dict and list results are flattened with one value per line (keys are not shown). Use --json for structured output that preserves keys, types, and nesting:

$ cki_secret --json MY_TOKEN/1#
{"token_type": "ssh_private_key", "active": true, "deployed": true, ...}

Location syntax

Conditions

Conditions filter multiple secrets via path[meta-key-1,!meta-key-2,...]. For all secrets starting with the given path (delimited by /), metadata fields are checked with a default of False. A secret is included only if all conditions match: True for meta-key-1, False for !meta-key-2.

The result is always a list — one entry per matching secret. This wraps each value in an extra level compared to the non-condition form:

$ cki_secret MY_TOKEN/1                # single string
s3cr3t-value

$ cki_secret MY_TOKEN[deployed]        # list of strings (one per match)
s3cr3t-value

$ cki_secret --json MY_TOKEN[deployed]
["s3cr3t-value"]

Combined with # or :, each entry is the corresponding dict or field:

$ cki_secret --json MY_TOKEN[active]#
[{"token_type": "ssh_private_key", "active": true, ...}, {"token_type": ...}]

$ cki_secret --json MY_TOKEN[active]:
[{"private_key": "...", "public_key": "..."}, {"private_key": "...", ...}]

Without --json, the list is flattened and each value is printed on its own line (dicts print as their Python repr — use --json when querying structured data with conditions).

Adding an unmanaged secret

For secrets without a token_type (not automated by the manager CLI):

1. Add the secret interactively (opens $EDITOR):

   cki_edit_secret MY_SECRET
   

   This creates the value in the backend and adds default metadata to the secrets file:

   MY_SECRET:
     backend: hv
     meta:
       active: true
       created_at: '2025-01-22T10:54:04+00:00'
       deployed: true
   

2. Add additional metadata if needed:

   cki_edit_secret MY_SECRET#some_field some-value
   

3. Commit the secrets file changes.

Removing an unmanaged secret

For managed secrets, use manager destroy instead (see below).

1. Remove all metadata and the backend reference:

   cki_edit_secret MY_SECRET# ''
   cki_edit_secret MY_SECRET: ''
   

2. Commit the secrets file changes.

3. Optionally, delete the secret from HashiCorp Vault:

   vault kv metadata delete apps/cki/MY_SECRET
   

Creating a managed secret

For secrets with a token_type that supports create:

1. Check the token type reference for required fields.

2. Add the required metadata to the secrets file. For example, an SSH key:

   SERVER_SSH_KEY/1:
     meta:
       token_type: ssh_private_key
       comment: server-name
       key_size: 4096
   

3. Create the secret:

   python3 -m cki_tools.credentials.manager create --token-name SERVER_SSH_KEY/1
   

4. Commit the secrets file changes.

Destroying a managed secret

python3 -m cki_tools.credentials.manager destroy --token-name SERVER_SSH_KEY/1

This revokes the token on the remote service (if applicable) and sets active: false in the metadata. Destroying a deployed version requires --force.

Checking rotation status

Show which managed tokens need attention:

python3 -m cki_tools.credentials.manager status
python3 -m cki_tools.credentials.manager table

Use --token-name to check a single token, --token-type-prefix to scope to a token type, or --token-age to override the age threshold:

python3 -m cki_tools.credentials.manager status --token-type-prefix gitlab_project
python3 -m cki_tools.credentials.manager status --token-age 180d

The table command shows a matrix of token health. Column abbreviations:

Rotating tokens (three-step)

Before starting: the prepare step may create new tokens and revoke old non-deployed ones. This can cause downstream validation to fail, so coordinate with your team to merge rotation changes quickly.

When operating on multiple tokens (no --token-name), prepare, switch, and clean prompt for confirmation before proceeding. Use --token-type-prefix to scope to a specific token type, or --force to bypass the age check (tokens not yet due for rotation are skipped by default).

Step 1: prepare

python3 -m cki_tools.credentials.manager prepare --token-name MY_TOKEN

Creates a new version (or rotates the spare slot). The secrets file changes:

# Before:
MY_TOKEN/1:
  meta: {token_type: gitlab_project_token, active: true, deployed: true, ...}

# After:
MY_TOKEN/1:
  meta: {token_type: gitlab_project_token, active: true, deployed: true, ...}
MY_TOKEN/1748123456:
  meta: {token_type: gitlab_project_token, active: true, deployed: false, ...}

Commit and deploy the secrets file before proceeding.

Step 2: switch

python3 -m cki_tools.credentials.manager switch --token-name MY_TOKEN

Flips deployed to the newest version:

MY_TOKEN/1:
  meta: {..., deployed: false, ...}
MY_TOKEN/1748123456:
  meta: {..., deployed: true, ...}

Commit and deploy before proceeding.

Step 3: clean

python3 -m cki_tools.credentials.manager clean --token-name MY_TOKEN

Destroys excess non-deployed versions and marks them active: false.

Commit and deploy.

Optional: purge

python3 -m cki_tools.credentials.manager purge --token-name MY_TOKEN

Removes metadata and backend entries for inactive versions.

Rotating tokens (grace period)

For token types that support grace-period rotation (e.g. Testing Farm), only the prepare step is needed — it rotates the deployed version directly:

python3 -m cki_tools.credentials.manager prepare --token-name MY_TF_TOKEN

The old key remains valid for the grace period (default: 7 days). No switch or clean step is needed.

Validating tokens

manager validate checks that active managed tokens are still valid on the remote service (API call per token):

python3 -m cki_tools.credentials.manager validate
python3 -m cki_tools.credentials.manager validate --token-type-prefix gitlab_project

cki_secrets_validate checks consistency between the secrets file and the backend for all secrets (managed and unmanaged, no remote service calls):

cki_secrets_validate

Exit codes: 0 = fully consistent, 1 = secrets exist in the backend without a metadata entry (informational — this is normal, e.g. after purging metadata), 2 = metadata references a secret missing from the backend, or required metadata (created_at) is absent.

Troubleshooting

validate reports missing secrets in Vault

The token was purged or never created. Check if the metadata still references it; if so, re-run create to recreate it.

prepare says “not preparing without –force”

The token is not old enough to need rotation. Use --token-age to lower the threshold, or --force to override.

Rotation errors mid-way (meta/error field)

When a creation or rotation fails after writing to the backend, the error is saved in meta/error. Inspect the error message, fix the underlying issue (expired auth, network failure, permission denied), and re-run the operation. The secret data may already be in the backend — check before re-creating.

Invariant violations

• “No deployed token”: no version has deployed: true. Manually set deployed: true on the correct version in the secrets file.
• “More than one deployed token”: multiple versions have deployed: true. Set all but the intended version to deployed: false.

Authentication errors

• GitLab 401 Unauthorized: you are likely using a personal token instead of a service account token. Re-check GITLAB_TOKENS.
• Vault permission denied: re-run cki_secrets_login --oidc.

Secrets CLI reference

$ python3 -m cki_tools.credentials.secrets --help
usage: secrets.py [-h] {secret,variable,edit,validate,login,logout} ...

Access CKI variables and secrets

positional arguments:
  {secret,variable,edit,validate,login,logout}
    secret              Retrieve a secret value
    variable            Retrieve a variable value
    edit                Edit a secret value
    validate            Validate stored secrets
    login               Log into secrets storage
    logout              Log out of secrets storage

  -h, --help            show this help message and exit
  --json                output in json format

CLI aliases

cki_secret

usage: cki_secret [-h] [--json] key

Retrieve a secret value using the location syntax.

cki_variable

usage: cki_variable [-h] [--json] key

Retrieve an unencrypted variable from the variables file (CKI_VARS_FILE).

Values are printed as-is by default. With --json, output is properly JSON-encoded (strings quoted, dicts serialized, etc.).

cki_edit_secret

usage: cki_edit_secret [-h] key [value]

Edit secrets and metadata. All location syntax forms are supported except conditions. When editing metadata, the secrets file is rewritten. When editing secret values, the backend is updated.

If value is omitted, $EDITOR (or vi) is opened. Quit with a nonzero exit code to abort (e.g. :cq in vi).

cki_secrets_validate

usage: cki_secrets_validate [-h]

Compare the secrets available in the backend with the secrets metadata. Exit codes: 0 = fully consistent, 1 = secrets exist in the backend without a metadata entry (informational), 2 = metadata references a secret missing from the backend, or required metadata (created_at) is absent.

cki_secrets_login

usage: cki_secrets_login [-h] [--duration DURATION] (--oidc | --jwtrole JWTROLE | --approle APPROLE)

Log into HashiCorp Vault. Authentication methods:

• --oidc: for human users (requires a Kerberos ticket for SSO).
• --jwtrole ROLE: for service accounts (requires VAULT_ID_TOKEN).
• --approle ROLE: for service accounts (requires VAULT_APPROLE_SECRET_ID).

The --duration parameter optionally extends the token validity beyond its default lifetime.

The client token is stored in ~/.vault-token (compatible with the official vault CLI).

cki_secrets_logout

usage: cki_secrets_logout [-h]

Revoke the client token and remove ~/.vault-token.

Manager CLI reference

usage: python3 -m cki_tools.credentials.manager
            [--token-name TOKEN_NAME]
            [--token-type-prefix TOKEN_TYPE_PREFIX]
            [--token-age TOKEN_AGE]
            [--force]
            {create,destroy,rotate,update,validate,status,table,prepare,switch,clean}

Low-level commands

Three-step rotation commands

Miscellaneous commands

The validate command also checks these invariants:

• all versions have active, deployed and created_at fields
• only one version is marked as deployed

Atlassian API tokens

• Operations: (none — externally managed)
• Features: validate
• Steady-state versions: 1
• Grace-period rotation: no

Validates that the API token works and the associated Atlassian account is active via the Jira /rest/api/2/myself endpoint.

Atlassian API tokens required fields

The secret value (API token) must be stored in the backend manually.

Atlassian API tokens managed fields

GitHub personal access tokens

• Operations: (none — externally managed)
• Features: update, validate
• Steady-state versions: 1
• Grace-period rotation: no

Validates that the token works via the GitHub /user endpoint. Cross-checks user_name if present in metadata. Update populates scopes from the X-OAuth-Scopes response header and expires_at from the GitHub-Authentication-Token-Expiration header.

GitHub personal access tokens required fields

The secret value (personal access token) must be stored in the backend manually.

GitHub personal access tokens managed fields

GitLab project access tokens

• Operations: create, destroy, rotate
• Features: three-step rotation, update, validate
• Steady-state versions: 2
• Grace-period rotation: no

See the API description for details.

Validate authenticates with the token and checks that it is not revoked.

GitLab project access tokens required fields

GitLab project access tokens managed fields

GitLab group access tokens

• Operations: create, destroy, rotate
• Features: three-step rotation, update, validate
• Steady-state versions: 2
• Grace-period rotation: no

See the API description for details.

Validate authenticates with the token and checks that it is not revoked.

GitLab group access tokens required fields

GitLab group access tokens managed fields

GitLab personal access tokens

• Operations: destroy, rotate
• Features: three-step rotation, update, validate
• Steady-state versions: 2
• Grace-period rotation: no

See the API description for details.

Validate authenticates with the token and checks that it is not revoked.

Dependency: rotation requires another personal access token on the same instance and user with api or read_api scope. The tool finds a suitable token automatically from the secrets file.

GitLab personal access tokens required fields

GitLab personal access tokens managed fields

GitLab project deploy tokens

• Operations: create, destroy
• Features: three-step rotation, update, validate
• Steady-state versions: 1
• Grace-period rotation: no

See the API description for details.

Validate checks that the token is not revoked or expired.

GitLab project deploy tokens required fields

GitLab project deploy tokens managed fields

GitLab group deploy tokens

• Operations: create, destroy
• Features: three-step rotation, update, validate
• Steady-state versions: 1
• Grace-period rotation: no

See the API description for details.

Validate checks that the token is not revoked or expired.

GitLab group deploy tokens required fields

GitLab group deploy tokens managed fields

GitLab runner authentication tokens

• Operations: rotate, destroy
• Features: three-step rotation, update, validate
• Steady-state versions: 2
• Grace-period rotation: no

See the API description for details.

Validate calls the runner verify endpoint.

GitLab runner authentication tokens required fields

The secret value (runner token) must be stored in the backend initially.

GitLab runner authentication tokens managed fields

LDAP keytabs

• Operations: (none — externally managed)
• Features: update
• Steady-state versions: 1
• Grace-period rotation: no

Update fetches LDAP object attributes from the directory server.

LDAP keytabs required fields

The secret value (keytab, base64-encoded) must be stored in the backend manually.

LDAP keytabs managed fields

LDAP passwords

• Operations: (none — externally managed)
• Features: update, validate
• Steady-state versions: 1
• Grace-period rotation: no

Update fetches LDAP object attributes. Validate performs a simple bind to the LDAP server.

LDAP passwords required fields

The secret value (password) must be stored in the backend manually.

LDAP passwords managed fields

AWS secret access keys

• Operations: create, destroy
• Features: three-step rotation, update
• Steady-state versions: 1
• Grace-period rotation: no

Note: the optional profile_name field selects the AWS profile to use. If missing, prepare and rotate silently skip the token.

AWS secret access keys required fields

AWS secret access keys managed fields

Dogtag certificates

• Operations: create
• Features: three-step rotation, update, validate
• Steady-state versions: 1
• Grace-period rotation: no

Validate verifies that the private key matches the certificate.

Dependency: authentication uses an ldap_password secret matching the uid field. This LDAP password must exist and be valid.

Dogtag certificates required fields

Dogtag certificates managed fields

Splunk HEC tokens

• Operations: (none — externally managed)
• Features: validate
• Steady-state versions: 1
• Grace-period rotation: no

Validate POSTs to the HEC endpoint; response codes 5 (no data) and 12 (event field required) are treated as success, confirming the token is valid.

Splunk HEC tokens required fields

The secret value (HEC token) must be stored in the backend manually.

Splunk HEC tokens managed fields

SSH keys

• Operations: create
• Features: three-step rotation, update, validate
• Steady-state versions: 1
• Grace-period rotation: no

Validate verifies that the private and public key form a matching pair. key_size must be at least 4096.

SSH keys required fields

SSH keys managed fields

Kubernetes ServiceAccount tokens

• Operations: create, destroy
• Features: three-step rotation, update, validate
• Steady-state versions: 1
• Grace-period rotation: no

Creates long-lived ServiceAccount tokens using Kubernetes Secrets with automatic token population. The ServiceAccount must exist beforehand.

Since Kubernetes 1.24, Secrets are no longer auto-created for ServiceAccounts. Creating an annotated kubernetes.io/service-account-token Secret is the explicit way to obtain a long-lived token. Managing these Secrets through the credentials manager (rather than just templating them directly) adds lifecycle management on top: three-step rotation via prepare/switch/clean, validation against the cluster API via SelfSubjectReview, and created_at tracking exposed as cki_token_created_at Prometheus metrics.

Validate performs a SelfSubjectReview against the cluster API to confirm the token is accepted.

Kubernetes ServiceAccount tokens required fields

Kubernetes ServiceAccount tokens managed fields

Generic passwords

• Operations: create
• Features: three-step rotation
• Steady-state versions: 1
• Grace-period rotation: no

Passwords are created via diceware.

Generic passwords required fields

Generic passwords managed fields

HashiCorp Vault secret IDs

• Operations: create, destroy
• Features: three-step rotation, validate
• Steady-state versions: 1
• Grace-period rotation: no

Only AppRole auth secret IDs are supported. Validate attempts a login with the secret ID.

HashiCorp Vault secret IDs required fields

HashiCorp Vault secret IDs managed fields

RabbitMQ passwords

• Operations: create, destroy
• Features: three-step rotation, update, validate
• Steady-state versions: 1
• Grace-period rotation: no

See the API description for details.

Validate establishes an actual AMQP connection (not the management API) to verify the credentials work end-to-end.

Dependency: management operations authenticate using another rabbitmq_password secret with administrator tags. At least one admin RabbitMQ user must exist.

RabbitMQ passwords required fields

RabbitMQ passwords managed fields

Testing Farm tokens

• Operations: create, destroy, rotate
• Features: single-step rotation (grace period), update, validate
• Steady-state versions: 1
• Grace-period rotation: yes (default 7 days)

Requires the requests-gssapi optional extra. Authenticates via Kerberos/SPNEGO using the ldap_password secret matching the uid derived from principal.

The Testing Farm API supports a grace period during token regeneration: after the API key is regenerated, the previous key remains valid for the configured duration (default: 7 days). This allows single-step rotation via prepare — see Grace-period rotation.

Validate calls /whoami, checks that the token is enabled and the token ID matches.

See the API description for details.

Dependency: authentication uses an ldap_password secret matching the uid derived from principal. This LDAP password must exist and be valid.

Testing Farm tokens required fields

Testing Farm tokens managed fields

Metrics

python3 -m cki_tools.credentials.metrics

Outputs Prometheus metrics about stored credentials:

Environment variables