cki_test.sh

Wrapper to run Python unit tests and check coverage

This page has an internal companion page which might contain additional information.

Runs pytest under coverage. The script takes care to install the needed dependencies via pip beforehand. For pytest, only the packages specified as parameters will be checked.

When run in a GitLab CI/CD pipeline for a merge request, the script tries to determine the previous code coverage and will fail if code coverage decreased. This can be skipped by adding [skip coverage check] with an explanation to the merge request description.

Environment variables

Environment variable	Description
`CKI_SKIP_PREPARE`	`true` if Python package installation should be skipped
`CKI_COVERAGE_ENABLED`	`false` if coverage calculation should be disabled
`CKI_COVERAGE_CHECK_ENABLED`	`false` if the coverage percentage calculation should be disabled
`CKI_PYTEST_ARGS`	Additional arguments for pytest, e.g. custom selection of tests
`CKI_PYTEST_IGNORELIST`	Directories to ignore for test discovery

NOTE

CKI_SKIP_PREPARE=true can be useful to run checks without network. When using tox, you should probably call: tox --skip-pkg-install; when using podman, you should probably call podman run --net none ...args....

Running Python tests and linting

To run both Python tests and linting, execute the following command:

tox

To run only the testing use the test tox environment:

tox -e test

Selecting which tests to run

To run only a custom selection of the tests, use:

TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS \
    CKI_PYTEST_ARGS="tests/file1.py[::classname[::test_name]] [tests/file2.py...]" \
    tox -e test

For example, the following command runs test_get_pipeline in tests/test_gitlab.py and all the tests in tests/test_cronjob.py:

TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS \
    CKI_PYTEST_ARGS="tests/test_gitlab.py::TestGitLabParseURL::test_get_pipeline tests/test_cronjob.py" \
    tox -e test

Usually test cases are unique, so you can often select tests by substring with -k, for example:

TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS \
    CKI_PYTEST_ARGS="-k test_get_pipeline" \
    tox -e test

Finally, another useful set of filters is:

--lf, --last-failed: Rerun only the tests that failed at the last run (or all if none failed)
--ff, --failed-first: Run all tests, but run the last failures first.

Printing captured logs and summary

Pytest will capture any output generated during execution by default. If you want to read stdout, stderr, or even specifically logs, you’ll need to combine a few options:

--capture=method Per-test capturing method: one of fd|sys|no|tee-sys. -s is a shortcut for --capture=no, which makes any output to be printed when it happens, mixed with the standard pytest logs; useful when you want to print() something in the testing code. Default: fd.
--show-capture={no,stdout,stderr,log,all} Controls how captured stdout/stderr/log is shown on failed tests. Note that only the root logger is captured, so make sure other loggers are propagating during tests. Default: all.
-r chars Show extra test summary info (including the captured text) as specified by chars: (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed, (p)assed, (P)assed with output, (a)ll except passed, or (A)ll. (w)arnings are enabled by default (see –disable-warnings), ‘N’ can be used to reset the list. Default: ‘fE’; cki_lint.sh default: ’s’.
--log-cli-level=DEBUG to see colored logs under the live log call section. Differently from the --show-capture=log, which shows all the captured logs in the summary at the end of the run according to the -r filter, --log-cli-level will shows logs as they happen, between each test. Useful to human-check which test is generating ERROR logs. Beware that it will only capture logs greater or equal to CKI_LOGGING_LEVEL.

Subtest caveats: if all failed assertions reside under a subtest context, pytest will count the test as passed even if we’re using pytest-subtests. You will see the subtests failures as “SUBFAIL” in the summary, but their logs won’t be included if you are using -rf. In that case, if you want to read logs from failed subtest in the summary, you will need to use -rA instead.

Note also that tests doing self.assertLogs will override the logging handlers, which means they won’t be captured by pytest, and therefore won’t be reported.

Example, showing warning and errors logs as they happen (which ideally shouldn’t exist and instead be caught by assertLogs), reporting all logs in the summary, this time including debug and info, about everything but passed tests:

TOX_OVERRIDE="testenv.passenv+=CKI_PYTEST_ARGS,CKI_LOGGING_LEVEL" \
    CKI_LOGGING_LEVEL="DEBUG" \
    CKI_PYTEST_ARGS="--log-cli-level=WARNING --show-capture=log -ra" \
    tox -e test

Disable coverage report

An additional useful environment variable is CKI_COVERAGE_ENABLED, which can be used to disable the coverage report, leaving a cleaner output from tests.

TOX_OVERRIDE=testenv.passenv+=CKI_COVERAGE_ENABLED \
    CKI_COVERAGE_ENABLED="false" \
    tox -e test