cki_tools.dependency_archive

The dependency-archive consists of two AWS Lambda functions that provide a transparent caching proxy for Fedora RPM downloads and OCI container image artifacts. Requests are served from the upstream origin when available and fall back to S3-cached copies when the origin no longer has the artifact (e.g. after upstream garbage collection). All non-cacheable requests are forwarded to the origin without caching.

Lambda functions

The dependency-archive image provides two Lambda handlers selected via LAMBDA_HANDLER:

• cki_tools.dependency_archive.handler_lambda: API Gateway entry point that routes RPM requests through the cache strategy described below and OCI requests through the v2 distribution API handler.
• cki_tools.dependency_archive.uploader_lambda: async worker invoked by the handler on cache miss; downloads from origin and uploads to S3.

RPM cache behavior

The handler uses a HEAD-first strategy to minimise S3 egress costs:

• .rpm requests (origin has it): handler sends a HEAD request to origin first; if the origin returns 200, the client is redirected there directly (no S3 egress). If the RPM is not yet cached, the uploader is triggered asynchronously to populate the cache for future fallback.
• .rpm requests (gone from origin, cached): when the HEAD returns 404 or times out and the RPM exists in S3, the handler redirects to a time-limited presigned S3 URL.
• .rpm requests (gone from origin, not cached): handler redirects to origin as a last resort (the client will see the origin’s error).
• Non-.rpm requests: handler returns 302 to origin (no caching).
• Errors: any S3/Lambda error falls back to a 302 redirect to origin, so the cache is never in the critical path.

OCI cache behavior

Requests with a v2/ path prefix are handled as OCI Distribution API v2 requests. The cache only processes digest-based manifest and blob pulls for images listed in OCI_ALLOWED_PREFIXES; everything else (tag lookups, other API endpoints, images not in the allowlist) is forwarded to the upstream registry transparently.

• GET /v2/: health check, returns 200 with Docker-Distribution-API-Version: registry/2.0.
• GET /v2/<name>/manifests/sha256:<digest> (allowed image): same strategy as RPMs – redirect to origin when available (triggering the uploader on cache miss), fall back to serving the manifest inline from S3 when the origin no longer has it.
• GET /v2/<name>/blobs/sha256:<digest> (allowed image): same strategy as RPMs – redirect to origin when available, fall back to a presigned S3 URL when the origin no longer has it.
• All other requests: forwarded to the upstream registry (302 redirect). This includes tag-based pulls, tag listings, and images not in OCI_ALLOWED_PREFIXES.
• Errors: S3 failures are logged and the handler falls through to fetch from the upstream registry, so the cache is never in the critical path.

Image allowlisting

OCI caching is scoped to specific images via two independent mechanisms:

Server-side (OCI_ALLOWED_PREFIXES): the cache only stores and serves artifacts for image names matching this allowlist. Requests for other images are forwarded to the upstream registry without touching S3. This prevents the cache from filling up with unwanted images.

Client-side (registries.conf prefix): the prefix field in registries.conf controls which pulls the container runtime routes through the cache in the first place. Using a specific prefix (e.g. quay.io/fedora/fedora) ensures that only matching images are sent to the mirror.

Both layers should agree: the client prefix should match (or be a subset of) the server-side OCI_ALLOWED_PREFIXES.

Client configuration

To use the cache as a transparent mirror for quay.io, configure registries.conf to redirect digest-based pulls through the cache:

[[registry]]
prefix = "quay.io/fedora/fedora"
location = "quay.io/fedora/fedora"

[[registry.mirror]]
location = "dependency-archive.example.com/fedora/fedora"
pull-from-mirror = "digest-only"

The location must include the full repository path (matching prefix), not just the hostname. The location field replaces the entire prefix portion of the reference, so location = "quay.io" would rewrite quay.io/fedora/fedora:tag into the invalid quay.io:tag.

With this configuration, buildah and skopeo will try the cache first for any digest-pinned pull of quay.io/fedora/fedora, falling back to quay.io/fedora/fedora directly if the cache is unavailable. Tag-based pulls bypass the mirror entirely due to pull-from-mirror = "digest-only".

To test without modifying the system-wide configuration, use a local config file:

buildah --registries-conf=/path/to/registries.conf pull quay.io/fedora/fedora@sha256:...
# or
CONTAINERS_REGISTRIES_CONF=/path/to/registries.conf skopeo copy \
    docker://quay.io/fedora/fedora@sha256:... oci:image:latest

Alternatively, address the cache directly as a registry (useful for ad-hoc testing without any registries.conf changes):

skopeo copy --src-tls-verify=false --src-no-creds \
    docker://cache-host:8080/fedora/fedora@sha256:... oci:image:latest

Request and response format

The handler receives API Gateway v2 (HTTP API) events with a {path+} catch-all route parameter. The path mirrors the origin URL structure:

• Input: event["pathParameters"]["path"] – everything after the hostname, e.g. pub/fedora/linux/updates/44/Everything/x86_64/Packages/f/foo-1.0.fc44.x86_64.rpm or v2/fedora/fedora/manifests/sha256:abc123
• 302 redirect: RPM and blob responses are redirects via the Location header, pointing to either the origin URL or a presigned S3 URL
• 200: OCI health check and manifest responses are returned inline
• 400: returned only when pathParameters or path is missing

Deployment

• Container image: quay.io/cki/dependency-archive
• S3 key prefix: RPMs are stored under cache/pub/...; OCI artifacts use flat content-addressed keys under cache/oci/<digest> (deduplicating across image names since OCI digests are content-addressed)
• Deployment config: lives in the deployment-all repo as an Ansible playbook using the cki_aws_lambda role (same pattern as receiver)

HTTP serve mode

The image can also be run as a standalone HTTP server via gunicorn, suitable for both integration testing and production OCP deployments where the Lambda runtime is not used:

python -m gunicorn --config cki_tools/dependency_archive/_gunicorn_conf.py cki_tools.dependency_archive:_wsgi_app

In this mode:

• Gunicorn serves requests on port 8080 using gevent workers for cooperative I/O multiplexing.
• Prometheus metrics are served at /metrics on the same port using multiprocess aggregation (each worker’s counters are combined).
• Cache misses are uploaded in a background thread per worker via an internal queue, so HTTP responses are not blocked by S3 uploads.

Sizing

DEPENDENCY_ARCHIVE_POOL_SIZE (default 1000) is the single concurrency knob. It sets:

• gunicorn worker_connections (max concurrent connections per worker)
• urllib3 pool_maxsize (HTTP keep-alive connections to origin)
• botocore max_pool_connections (connections to S3/Lambda API)

The default is 2 gevent workers per pod. Two workers outperform one on a 1 CPU pod because the workload is ~95% I/O wait: while one worker does CPU work (SSL processing, response parsing), the other sits in epoll_wait consuming zero CPU. The kernel interleaves their CPU bursts more efficiently than a single event loop can (+27% RPS, 2× tighter p95 in load tests).

For a 1 CPU / 512 MiB pod, DEPENDENCY_ARCHIVE_POOL_SIZE=200 is a good starting point: 200 connections × 2 workers × 2 replicas = 800 max concurrent requests, well within the memory budget (~8 MiB SSL state per worker on top of ~150 MiB base).

All gunicorn settings (workers, worker class, timeout, bind address) have defaults in _gunicorn_conf.py and can be overridden via the standard GUNICORN_CMD_ARGS environment variable. For example, to switch to threaded workers with 4 processes:

GUNICORN_CMD_ARGS="--workers 4 -k gthread --threads 200"

Testing

• Unit tests: python -m pytest tests/test_dependency_archive.py
• Integration tests: inttests/images/dependency-archive/ (runs via CI image inttest job, or locally with tox -e image -- inttests/images/dependency-archive). The OCI integration test deploys the image in HTTP serving mode and uses skopeo copy to pull a real Fedora image through the cache.

Design rationale

This module intentionally avoids depending on cki-lib to keep the Lambda image small (12 pip packages vs 41+). See the module docstring in cki_tools/dependency_archive/ for details.