Reproducing and debugging test runs
The CKI tests are not meant to be rerun locally. This is (mostly) because CKI runs testing on specific hardware and setup to get appropriate coverage.
Most often the tests are being ran in the Red Hat Beaker instance, however different labs and cloud providers are supported.
Other teams can also send their test results to CKI to propagate them to the developers; in these cases CKI has no control over which tests are being executed, how, and in what environment.
Rerunning a pipeline or job via CKI bot
Use the bot testing on an opened MR in the kernel test repository as described in the bot documentation. This also allows modifying the test to include debug code.
Rerunning tests in Beaker directly
This option is limited to test execution done in the Red Hat Beaker instance.
Obtaining an appropriate Beaker XML file
The Beaker XML files are available in the builds and tests at DataWarehouse.
Selecting tests to run
For each test, a minimal Beaker XML with only one recipe that does not include any later tests is available from the DataWarehouse entry of the test:
The complete Beaker XML including all recipes and tests is available in the output files of the Build.
It is possible to remove previous tests (
<task>) from the XML file as well.
But keep in mind that tests that ran before the failing test can be the reason
for the failure.
Testing a different kernel
If you wish to execute the testing against a different kernel version, e.g.
to check if the failures occur with base kernel, you can do so by editing
- Search for
KPKG_URLin the XML.
- Replace the part in front of
#by a valid
dnfrepository containing the kernel you wish to install.
- If you picked a different kernel variant to install (e.g. kernel vs
kernel-rt, debug vs non-debug), adjust the options after
#accordingly. Most of the time, you want to keep them the same.
You can also use the full
KPKG_URL parameter (including the options) from a
different CKI Beaker XML to install a kernel from that run.
Reserving the machine after the test
If you want to keep the machine after the tests have finished, add
<reservesys/> right before
</tasks>. This will reserve the machine for 24
Modifying host requirements
It might be necessary to modify the
<hostrequires> section, e.g. to select
exactly the same machine as was used for the failing test. See the Beaker
documentation and XML schema for details.
NOTE: In some cases, the machines needed for testing are not available to all developers. In such case, reach out to the test maintainer(s) (their contact information is available on the test details in DataWarehouse) or the CKI team to get temporary access.
Submitting the job
The job can be either submitted via the command line or via the web interface.
Submitting the job via the command line
This step assumes the following Beaker configuration stored in
HUB_URL = "https://beaker.engineering.redhat.com" AUTH_METHOD = "krbv" KRB_REALM = "REDHAT.COM"
Submit the job via
bkr job-submit <FILENAME>
You can get the
bkr command by installing the
beaker-client package. Full
documentation of the command is available by running
man bkr job-submit.
Submitting the job via the web interface
- Go to https://beaker.engineering.redhat.com and click on
Log inin the top right corner.
- Open the
Schedulerdropdown from the top left side menu. Pick the
- Click on the
Browsebutton and upload your file.
- Click on the
Submit Databutton below the
- Click on the
Queuebutton on the right side.
Accessing test machines via SSH and manually running tests
It is possible to access the test machines via
ssh root@machine-name during
the test execution. With
<reservesys/> as described above, this is even
possible after the tests have finished.
To manually rerun tests when logged in via SSH, locate the appropriate
harness.log output file for the failing in DataWarehouse, and change into
the directory printed on top of it like
** Fetching task: 160743445 [/mnt/tests/gitlab.com/redhat/centos-stream/tests/kernel/kernel-tests/-/archive/main/kernel-tests-main.zip/rt-tests/us/stalld]
There, the test code can be run via