Adding a Mirror Repository

Step-by-step guide for adding a new kernel.org mirror to GitLab

This document provides explicit instructions for adding a new kernel.org repository mirror to GitLab. It is designed to be followed by both humans and AI systems with minimal interpretation required.

Automation Note

This document has three phases, each ending with a verification step or merge request. When following as an AI agent:

  • Execute steps within a phase without asking for confirmation
  • STOP at the end of each phase when you see STOP: Human action required
  • Report the MR URL and wait for the user to confirm the merge before continuing
  • Only proceed to the next phase after the user confirms the previous step is complete

Overview

CKI maintains mirrors of kernel.org repositories on GitLab to limit load on upstream servers and provide reliable access for CI pipelines. The mirrors are configured at:

The process has three phases:

  1. Phase 1: Update mirrors.yml - Add the new repository URL to the mirror list
  2. Phase 2: Create GitLab groups/repo - Run script to create the mirror repository
  3. Phase 3: Deploy configuration - Apply the mirroring configuration

Input Parameters

The process requires the following parameters:

  • {kernel_org_url}: The kernel.org URL to mirror (e.g., https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git)
  • {issue_url}: URL to the GitLab issue or Jira ticket tracking this work
  • {pipeline_data_repo_path}: Path to the pipeline-data repository (e.g., ../pipeline-data)
  • {pipeline_data_fork_remote}: Git remote name for your pipeline-data fork
  • {deployment_all_repo_path}: Path to the deployment-all repository (internal, e.g., ../deployment-all)

Prerequisites

Before starting, verify the following:

1. Issue URL

You must have a GitLab issue or Jira ticket URL to associate with this work.

Verification: Confirm {issue_url} has been provided.

  • If provided: Proceed to next prerequisite.
  • If NOT provided: STOP and ask the user for the issue URL before continuing.

2. Source Repository Accessibility

The kernel.org repository must be publicly accessible.

Verification command:

git ls-remote --heads {kernel_org_url} | head -5

Success criteria: Returns branch references without error.

3. Repository Not Already Mirrored

Check that the repository is not already in the mirrors list.

Verification command:

grep -F "{kernel_org_url}" {pipeline_data_repo_path}/mirrors.yml

Success criteria: Returns no output (repository not found).

  • If output is empty: Proceed to next prerequisite.
  • If output shows URL: STOP - repository is already mirrored.

4. Pipeline-data Fork Setup

You need a fork of the pipeline-data repository with remotes configured.

Verification command:

cd {pipeline_data_repo_path} && git remote -v && cd -

Identify your fork remote: Look for a remote pointing to your personal fork. Note this remote name as {pipeline_data_fork_remote}.

5. GitLab Access

You need maintainer access to the mirror group on gitlab.com.

Verification: Confirm you can access https://gitlab.com/redhat/red-hat-ci-tools/kernel/mirror.

Phase 1: Update mirrors.yml

This phase adds the new repository URL to the mirrors list in pipeline-data.

1.1 Determine the Googlesource URL

Kernel.org repositories are mirrored via kernel.googlesource.com for reliability. Convert the kernel.org URL to the googlesource equivalent.

Conversion rule:

https://git.kernel.org/pub/scm/{path}.git
→
https://kernel.googlesource.com/pub/scm/{path}.git

Example:

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
→
https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net.git

1.2 Add URL to mirrors.yml

File: {pipeline_data_repo_path}/mirrors.yml

Location: Find the source_urls: list.

Action: Add the googlesource URL to the list in alphabetical order by path.

Example addition:

source_urls:
  # ... existing entries ...
  - https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net.git
  # ... existing entries ...

1.3 Create and Submit Merge Request

cd {pipeline_data_repo_path}
git checkout main && git pull
git checkout -b add-mirror-{repo_name}
git add mirrors.yml
git commit -m "mirrors: add {repo_name} repository

{issue_url}"
git push \
    -o merge_request.create \
    -o merge_request.target=main \
    -o merge_request.title="mirrors: add {repo_name} repository" \
    -o merge_request.description="Related issue: {issue_url}" \
    {pipeline_data_fork_remote} add-mirror-{repo_name}
cd -

Success criteria: MR is created.

STOP: Human action required

The pipeline-data MR must be reviewed and merged before proceeding.

Report the MR URL to the user and wait for confirmation that it has been merged.

Phase 2: Create GitLab Groups and Repository

After the mirrors.yml MR is merged, create the GitLab groups and repository using the script below.

2.1 Run the Repository Creation Script

The following Python script creates the necessary GitLab groups and the mirror repository for a single new mirror.

Prerequisites:

  • cki-tools package installed (pip install cki-tools)
  • GitLab token with API access to gitlab.com (maintainer on mirror group)
  • Environment variables configured:
    • GITLAB_TOKENS: JSON dict mapping GitLab host names to environment variable names, e.g., {"gitlab.com": "GITLAB_TOKEN"}
    • GITLAB_TOKEN: Your GitLab private token

Script (save as create_mirror_repo.py):

"""Create GitLab group hierarchy and repo for a single kernel mirror."""

import subprocess
import sys

from cki_lib.gitlab import get_instance
from cki_lib.misc import only_log_exceptions

# Configuration
TARGET_MIRROR_GROUP_PATH = "redhat/red-hat-ci-tools/kernel/mirror/"
ORIGINAL_MIRROR_PREFIX = "https://git.kernel.org/pub/scm/"
KERNEL_REPOS_PREFIX = "https://kernel.googlesource.com/pub/scm/"

def create_mirror_repo(googlesource_url: str) -> None:
    """Create GitLab groups and repository for a single mirror."""
    gl = get_instance("https://gitlab.com")

    # Extract path from URL
    repo_path = googlesource_url.removeprefix(KERNEL_REPOS_PREFIX).removesuffix('.git')
    path_parts = repo_path.split('/')

    # Create any missing intermediary groups
    for i in range(1, len(path_parts)):
        group_path = TARGET_MIRROR_GROUP_PATH + '/'.join(path_parts[:i])
        try:
            gl.groups.get(group_path)
            print(f"Group exists: {group_path}")
        except Exception:
            parent_path = (TARGET_MIRROR_GROUP_PATH + '/'.join(path_parts[:i-1])).rstrip('/')
            parent = gl.groups.get(parent_path)
            gl.groups.create({
                'parent_id': parent.id,
                'name': path_parts[i-1],
                'path': path_parts[i-1]
            })
            print(f"Created group: {group_path}")

    # Create the repository
    project_path = TARGET_MIRROR_GROUP_PATH + repo_path
    try:
        project = gl.projects.get(project_path)
        print(f"Repository exists: {project_path}")
    except Exception:
        parent_path = (TARGET_MIRROR_GROUP_PATH + '/'.join(path_parts[:-1])).rstrip('/')
        parent = gl.groups.get(parent_path)
        project = gl.projects.create({
            'namespace_id': parent.id,
            'path': path_parts[-1]
        })
        print(f"Created repository: {project_path}")

    # Set default branch based on the original repository
    kernel_org_url = ORIGINAL_MIRROR_PREFIX + repo_path
    result = subprocess.run(
        ['git', 'ls-remote', '--symref', kernel_org_url, 'HEAD'],
        check=True, capture_output=True, encoding='utf8'
    )
    default_branch = result.stdout.split()[1].replace('refs/heads/', '')

    if project.default_branch != default_branch:
        project.default_branch = default_branch
        with only_log_exceptions():
            project.save()
        print(f"Set default branch: {default_branch}")

    print(f"\nMirror repository ready: https://gitlab.com/{project_path}")

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print(f"Usage: {sys.argv[0]} <googlesource_url>")
        print(f"Example: {sys.argv[0]} https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net.git")
        sys.exit(1)
    create_mirror_repo(sys.argv[1])

Run the script:

python3 create_mirror_repo.py {googlesource_url}

Example:

python3 create_mirror_repo.py https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net.git

Expected output:

Group exists: redhat/red-hat-ci-tools/kernel/mirror/linux
Group exists: redhat/red-hat-ci-tools/kernel/mirror/linux/kernel
Group exists: redhat/red-hat-ci-tools/kernel/mirror/linux/kernel/git
Group exists: redhat/red-hat-ci-tools/kernel/mirror/linux/kernel/git/netdev
Created repository: redhat/red-hat-ci-tools/kernel/mirror/linux/kernel/git/netdev/net
Set default branch: main

Mirror repository ready: https://gitlab.com/redhat/red-hat-ci-tools/kernel/mirror/linux/kernel/git/netdev/net

2.2 Verify Repository Creation

Verification command:

# Extract the path from the googlesource URL
REPO_PATH=$(echo "{googlesource_url}" | sed 's|https://kernel.googlesource.com/pub/scm/||' | sed 's|\.git$||')
curl -s "https://gitlab.com/api/v4/projects/redhat%2Fred-hat-ci-tools%2Fkernel%2Fmirror%2F${REPO_PATH//\//%2F}" | jq -r '.web_url'

Success criteria: Returns the GitLab project URL.

STOP: Human action required

Verify the repository was created correctly at the URL shown above.

Report the repository URL to the user and wait for confirmation.

Phase 3: Deploy Mirroring Configuration

After the repository is created, deploy the gitlab-repo-config to enable mirroring. The deployment scripts are in the deployment-all repository.

3.1 Deploy Configuration

cd {deployment_all_repo_path}
git checkout main && git pull
REPOS=cki ./gitlab-repo-config/deploy.sh
cd -

3.2 Verify Mirroring is Active

Verification command:

# Check mirror status via GitLab API
REPO_PATH=$(echo "{googlesource_url}" | sed 's|https://kernel.googlesource.com/pub/scm/||' | sed 's|\.git$||')
curl -s "https://gitlab.com/api/v4/projects/redhat%2Fred-hat-ci-tools%2Fkernel%2Fmirror%2F${REPO_PATH//\//%2F}" | jq -r '.mirror, .mirror_user_id'

Success criteria: Returns true and a user ID.

3.3 Verify Content is Mirrored

Wait a few minutes for the initial mirror sync, then verify:

Verification command:

REPO_PATH=$(echo "{googlesource_url}" | sed 's|https://kernel.googlesource.com/pub/scm/||' | sed 's|\.git$||')
git ls-remote "https://gitlab.com/redhat/red-hat-ci-tools/kernel/mirror/${REPO_PATH}.git" | head -5

Success criteria: Returns branch/tag references matching the source repository.

Complete Example: Adding netdev/net mirror

Assuming:

  • {kernel_org_url} = https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
  • {issue_url} = https://gitlab.com/cki-project/infrastructure/-/issues/123
  • {pipeline_data_repo_path} = . (current directory is pipeline-data)
  • {pipeline_data_fork_remote} = fork
  • {deployment_all_repo_path} = ../deployment-all

Example: Phase 1

# Verify not already mirrored
grep -F "netdev/net" mirrors.yml

# Edit mirrors.yml - add the URL in alphabetical order:
#   - https://kernel.googlesource.com/pub/scm/linux/kernel/git/netdev/net.git

git checkout main && git pull
git checkout -b add-mirror-netdev-net
git add mirrors.yml
git commit -m "mirrors: add netdev/net repository

https://gitlab.com/cki-project/infrastructure/-/issues/123"
git push \
    -o merge_request.create \
    -o merge_request.target=main \
    -o merge_request.title="mirrors: add netdev/net repository" \
    -o merge_request.description="Related issue: https://gitlab.com/cki-project/infrastructure/-/issues/123" \
    fork add-mirror-netdev-net

# STOP: Get MR reviewed and merged before continuing

Example: Phase 2

# After MR is merged, run the creation script
python3 create_mirror_repo.py

# Verify
curl -s "https://gitlab.com/api/v4/projects/redhat%2Fred-hat-ci-tools%2Fkernel%2Fmirror%2Flinux%2Fkernel%2Fgit%2Fnetdev%2Fnet" | jq -r '.web_url'

# STOP: Verify repository exists at the URL

Example: Phase 3

# Deploy configuration
cd ../deployment-all
git checkout main && git pull
REPOS=cki ./gitlab-repo-config/deploy.sh
cd -

# Verify mirroring is enabled
curl -s "https://gitlab.com/api/v4/projects/redhat%2Fred-hat-ci-tools%2Fkernel%2Fmirror%2Flinux%2Fkernel%2Fgit%2Fnetdev%2Fnet" | jq -r '.mirror'

# Wait a few minutes, then verify content
git ls-remote https://gitlab.com/redhat/red-hat-ci-tools/kernel/mirror/linux/kernel/git/netdev/net.git | head -5

# Done! Mirror is active.

Troubleshooting

Repository creation fails with permission error

Ensure you have maintainer access to the mirror group.

Mirroring not starting

Check that the deploy was successful and the mirror configuration was applied. The GitLab mirror user must have appropriate permissions.

Content not syncing

GitLab pull mirroring runs periodically. Force a sync from the repository settings page: Settings → Repository → Mirroring repositories → Update now.