connecting external Beaker labs via VPN connections

How the CKI setup interfaces with external Beaker labs via VPN tunnels

To run pipelines that use external labs for testing, the CKI setup can use VPN tunnels to connect to the Beaker instance.

Constraints

  • external labs host Beaker servers on their intranet, which requires a VPN connection for testing
  • multiple VPN connections for different external labs are not allowed to interfere with each other
  • when using static keys, OpenVPN credentials can only be used by one client, i.e. it is not possible to access the intranet with the same credentials from multiple machines at the same time

General idea

Pipelines connecting to external labs should run completely separated from the rest of CKI setup in a separate pipeline repository. This enables custom GitLab runners to be attached and configured per external lab. The GitLab runners are hosted on multiple subnets across availability zones in AWS. Each runner will spawn workers into a worker subnet per availability zone.

To access the Beaker instance, VPN connections will be provided by the GitLab runners which will used by all respective workers. This requires the GitLab runners to function as NAT gateways for the intranet CIDR, and provide DNS services for the workers to be able to resolve hosts of the intranet domain.

Components

In the following, foo is used as a placeholder for the lab name.

  • pipeline repository: https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-foo-pipelines
  • pipeline trigger configuration in https://gitlab.com/cki-project/pipeline-data/-/blob/main/foo.yaml and associated CronJob
  • S3 buckets: arr-cki-prod-foo-…
  • VPC subnets: arr-cki.prod.subnet.foo-…
  • GitLab runners deployed into the runner subnets: foo-…

VPN connection sharing

The VPN connection is created on the GitLab runners via NetworkManager and OpenVPN. To share the VPN connection, the following aspects are configured.

AWS setup:

  • rule in the security group for the GitLab runners to allow ingress traffic on any port from the worker subnets
  • routes added to worker subnet route tables that use the runner EC2 instances for the intranet CIDR
  • disabled source/destination checks on the runner EC2 instances

GitLab runner instances:

  • IPv4 forwarding enabled
  • OpenVPN NetworkManager VPN connection to the intranet
  • VPN connection added as secondary connection to the default Ethernet connection so it gets started automatically
  • NetworkManager dispatcher script to configure masquerading as soon as the VPN connection is started, taking Docker iptables rules into account
  • systemd-resolved exposed on the default network interface via DNSStubListenerExtra
  • instance-specific worker user script that points systemd-resolved on the workers to systemd-resolved on the GitLab runner instances