cki_tools.cki_mr_viewer

CKI MR Viewer - Web dashboard for viewing GitLab merge requests and pipeline statistics

Overview

The CKI MR Viewer is a Flask-based web application that provides a user-friendly interface for viewing merge requests and their pipeline/job statistics from Red Hat kernel projects on gitlab.com. It supports persistent caching and real-time pipeline tracking.

Features

  • Flexible Date Filtering: Query MRs from last N days or custom date range
  • Cross-Project Pipelines: Track parent-child pipelines across different GitLab projects
  • Comprehensive Job Tracking: Monitor 6 job states (passed, failed, running, pending, skipped, other)
  • Persistent Cache: Dramatically speeds up queries with 8-hour TTL
  • Modern Web UI: Real-time progress tracking and collapsible pipeline views
  • Production Ready: Built-in Gunicorn support for production deployment

Installation

Install with the optional dependencies:

pip install -e .[cki-mr-viewer]

CLI Usage

$ python -m cki_tools.cki_mr_viewer.cli --help
usage: cli.py [-h] [--version] {serve,health-check} ...

GitLab MR Viewer CLI

positional arguments:
  {serve,health-check}  Available commands
    serve               Start the dashboard server
    health-check        Perform health check of the application

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

Serve Command (Web Dashboard)

$ python -m cki_tools.cki_mr_viewer.cli serve --help
usage: cli.py serve [-h] [--host HOST] [--port PORT] [--debug] [--production]
                    [--workers WORKERS] [--open-browser]

options:
  -h, --help         show this help message and exit
  --host HOST        Host to bind to
  --port PORT        Port to bind to
  --debug            Enable debug mode
  --production       Run with Gunicorn production server
  --workers WORKERS  Number of Gunicorn workers (production mode only)
  --open-browser     Open browser automatically

Health Check Command

$ python -m cki_tools.cki_mr_viewer.cli health-check --help
usage: cli.py health-check [-h] [--host HOST] [--port PORT]
                           [--timeout TIMEOUT]

options:
  -h, --help         show this help message and exit
  --host HOST        Host to check
  --port PORT        Port to check
  --timeout TIMEOUT  Timeout in seconds

Deployment Options

Development Server

python -m cki_tools.cki_mr_viewer.cli serve --host 0.0.0.0 --port 5000 --debug

Production Server (Gunicorn)

export GITLAB_TOKEN='glpat-your-token-here'
python -m cki_tools.cki_mr_viewer.cli serve --production --host 0.0.0.0 --port 5000 --workers 4

Container Deployment

Run with gitlab.com token (all CKI projects are on gitlab.com):

podman run -d -p 5000:5000 \
  -e GITLAB_TOKEN='your-token-here' \
  -v gitlab-mr-cache:/data/cache:Z \
  --name cki-mr-viewer \
  quay.io/cki/cki-mr-viewer:production

Explanation:

  • -e GITLAB_TOKEN='...' - GitLab.com API token for authentication
  • -v gitlab-mr-cache:/data/cache:Z - Persistent cache for faster queries

Verify Container is Running

# Check container status
podman ps | grep cki-mr-viewer

# Check health endpoint
curl http://localhost:5000/health

# View logs
podman logs -f cki-mr-viewer

Environment Variables

  • GITLAB_TOKEN: GitLab API token for authentication (required)
  • CACHE_ENABLED: Enable/disable cache (default: true)
  • CACHE_TTL_HOURS: Cache TTL in hours (default: 8)
  • CACHE_REFRESH_DAYS: Days to refresh in background (default: 180)
  • SENTRY_DSN: Sentry DSN for error tracking (optional)
  • SENTRY_ENVIRONMENT: Sentry environment name (optional, default: production)

Error Handling & Resilience

The application includes comprehensive error handling for transient API failures:

  • Automatic Retry Logic: Rate limits (429), server errors (5xx), and network issues are automatically retried with exponential backoff
  • Smart Error Detection: Distinguishes between temporary errors (retries) and permanent errors (alerts)
  • Sentry Integration: Reports persistent failures to Sentry for monitoring and alerting
  • Cache Preservation: Transient errors don’t invalidate existing cache data
  • Graceful Degradation: Failed projects are retried without blocking other projects

Retry Behavior

Error Type Retryable Default Retry After Max Attempts
429 Rate Limit ✅ Yes 60s (from header) 3
500-504 Server Error ✅ Yes 30s 3
408 Timeout ✅ Yes 10s 3
Network Error ✅ Yes 30s 3
401/403 Auth Error ❌ No N/A 0
404 Not Found ❌ No N/A 0

Supported Projects

The application monitors Red Hat kernel projects on gitlab.com:

  • redhat/centos-stream/src/kernel/centos-stream-10
  • redhat/centos-stream/src/kernel/centos-stream-9
  • redhat/rhel/src/kernel/rhel-10
  • redhat/rhel/src/kernel/rhel-9
  • redhat/rhel/src/kernel/rhel-8
  • redhat/rhel/src/kernel/rhel-7
  • redhat/rhel/src/kernel/rhel-6

API Endpoints

Get Merge Requests

GET /api/mrs?project=<project>&days=<days>
GET /api/mrs?project=<project>&start_date=<YYYY-MM-DD>&end_date=<YYYY-MM-DD>

Example:

curl "http://localhost:5000/api/mrs?project=redhat/centos-stream/src/kernel/centos-stream-10&days=7"

Get MR Pipelines

GET /api/mr/<mr_iid>/pipelines?project=<project>&max_pipelines=<limit>

Example:

curl "http://localhost:5000/api/mr/123/pipelines?\
project=redhat/centos-stream/src/kernel/centos-stream-10"

Bulk Pipeline Fetch

POST /api/mrs/pipelines/bulk
Body: {"mr_iids": [1, 2, 3], "project": "...", "max_pipelines": 5}

Example:

curl -X POST http://localhost:5000/api/mrs/pipelines/bulk \
  -H "Content-Type: application/json" \
  -d '{"mr_iids": [123, 124], "project": "redhat/centos-stream/src/kernel/centos-stream-10"}'

Cache Management

GET  /api/cache/info        # View cache statistics
GET  /api/cache/entries     # List cache entries  
POST /api/cache/clear       # Remove expired entries
POST /api/cache/clear-all   # Clear entire cache

Example:

# Get cache info
curl http://localhost:5000/api/cache/info | python3 -m json.tool

# Clear expired cache
curl -X POST http://localhost:5000/api/cache/clear

Health Check

GET /health

Example:

curl http://localhost:5000/health

Module Structure

cki_tools/cki_mr_viewer/
├── __init__.py          # Package initialization
├── app.py               # Flask application and API endpoints
├── viewer.py            # GitLab API client and data fetching logic
├── cache_manager.py     # Persistent cache management
├── cli.py               # Command-line interface
└── templates/
    └── index.html       # Web UI template

How it Works

  1. Data Fetching: Uses GitLab REST API v4 to fetch MRs and pipeline data
  2. Caching: MR and pipeline data are cached with 8-hour TTL for fast repeated queries
  3. Background Refresh: Optional background scheduler refreshes cache every 8 hours
  4. Rate Limiting: Intelligent request throttling (max 10 concurrent) to avoid API limits

Common Use Case

Track Red Hat kernel MRs from gitlab.com:

podman run -d -p 5000:5000 \
  -e GITLAB_TOKEN='glpat-xxx' \
  -v gitlab-mr-cache:/data/cache:Z \
  --name cki-mr-viewer \
  quay.io/cki/cki-mr-viewer:production

Access the dashboard at: http://localhost:5000

Performance Notes

  • First Query: May take 1-2 minutes for 180 days of data
  • Cached Queries: Nearly instant (<1 second) for same date range
  • Cache TTL: 8 hours (configurable via CACHE_TTL_HOURS)
  • Rate Limits: With token: ~10x more requests than anonymous

Troubleshooting

Issue: Rate Limited

Solution: Ensure GITLAB_TOKEN is set with a valid personal access token.

Issue: Cache Not Persisting

Solution: Ensure volume is mounted: -v gitlab-mr-cache:/data/cache:Z

Development

Run tests:

cd /home/bgoncalv/repos/cki/cki-tools
python -m unittest discover -s tests/cki_mr_viewer -p 'test_*.py' -v

Run locally:

pip install -e .[cki-mr-viewer]
export GITLAB_TOKEN='your-token'
python -m cki_tools.cki_mr_viewer.cli serve --debug