Reproducing and debugging test runs

How to reproduce or debug kernel testing done in the CKI pipeline

The CKI tests are not meant to be rerun locally. This is (mostly) because CKI runs testing on specific hardware and setup to get appropriate coverage.

Most often the tests are being ran in the Red Hat Beaker instance, however different labs and cloud providers are supported.

Other teams can also send their test results to CKI to propagate them to the developers; in these cases CKI has no control over which tests are being executed, how, and in what environment.

Rerunning a pipeline or job via CKI bot

Use the bot testing on an opened MR in the kernel test repository as described in the bot documentation. This also allows modifying the test to include debug code.

Rerunning tests in Beaker directly

This option is limited to test execution done in the Red Hat Beaker instance.

Obtaining an appropriate Beaker XML file

The Beaker XML files are available in the builds and tests at DataWarehouse.

Selecting tests to run

For each test, a minimal Beaker XML with only one recipe that does not include any later tests is available from the DataWarehouse entry of the test:

Test entry in DataWarehouse

The complete Beaker XML including all recipes and tests is available in the output files of the Build.

It is possible to remove previous tests (<task>) from the XML file as well. But keep in mind that tests that ran before the failing test can be the reason for the failure.

Testing a different kernel

If you wish to execute the testing against a different kernel version, e.g. to check if the failures occur with base kernel, you can do so by editing the KPKG_URL parameter:

  1. Search for KPKG_URL in the XML.
  2. Replace the part in front of # by a valid yum or dnf repository containing the kernel you wish to install.
  3. If you picked a different kernel variant to install (e.g. kernel vs kernel-rt, debug vs non-debug), adjust the options after # accordingly. Most of the time, you want to keep them the same.

You can also use the full KPKG_URL parameter (including the options) from a different CKI Beaker XML to install a kernel from that run.

Reserving the machine after the test

If you want to keep the machine after the tests have finished, add <reservesys/> right before </tasks>. This will reserve the machine for 24 hours.

Modifying host requirements

It might be necessary to modify the <hostrequires> section, e.g. to select exactly the same machine as was used for the failing test. See the Beaker documentation and XML schema for details.

NOTE: In some cases, the machines needed for testing are not available to all developers. In such case, reach out to the test maintainer(s) (their contact information is available on the test details in DataWarehouse) or the CKI team to get temporary access.

Submitting the job

The job can be either submitted via the command line or via the web interface.

Submitting the job via the command line

This step assumes the following Beaker configuration stored in ~/.beaker_client/config:

HUB_URL = "https://beaker.engineering.redhat.com"
AUTH_METHOD = "krbv"
KRB_REALM = "REDHAT.COM"

Submit the job via

bkr job-submit <FILENAME>

You can get the bkr command by installing the beaker-client package. Full documentation of the command is available by running man bkr job-submit.

Submitting the job via the web interface

  1. Go to https://beaker.engineering.redhat.com and click on Log in in the top right corner.
  2. Open the Scheduler dropdown from the top left side menu. Pick the New Job option.
  3. Click on the Browse button and upload your file.
  4. Click on the Submit Data button below the Browse button.
  5. Click on the Queue button on the right side.

Accessing test machines via SSH and manually running tests

It is possible to access the test machines via ssh root@machine-name during the test execution. With <reservesys/> as described above, this is even possible after the tests have finished.

To manually rerun tests when logged in via SSH, locate the appropriate harness.log output file for the failing in DataWarehouse, and change into the directory printed on top of it like

** Fetching task: 160743445 [/mnt/tests/gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip/rt-tests/us/stalld]

There, the test code can be run via bash runtest.sh.