cki_test.sh
Runs pytest
under coverage
. The script takes care to install the needed dependencies
via pip beforehand. For pytest
, only the packages specified as parameters will be checked.
When run in a GitLab CI/CD pipeline for a merge request, the script tries to
determine the previous code coverage and will fail if code coverage decreased.
This can be skipped by adding [skip coverage check]
with an explanation to
the merge request description.
Environment variables
Environment variable | Description |
---|---|
CKI_COVERAGE_ENABLED |
false if coverage calculation should be disabled |
CKI_COVERAGE_CHECK_ENABLED |
false if the coverage percentage calculation should be disabled |
CKI_PYTEST_ARGS |
Additional arguments for pytest, e.g. custom selection of tests |
CKI_PYTEST_IGNORELIST |
Directories to ignore for test discovery |
Running Python tests and linting
To run both Python tests and linting, execute the following command:
tox
To run only the testing use the test
tox environment:
tox -e test
Selecting which tests to run
To run only a custom selection of the tests, use:
TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS \
CKI_PYTEST_ARGS="tests/file1.py[::classname[::test_name]] [tests/file2.py...]" \
tox -e test
For example, the following command runs test_get_pipeline
in
tests/test_gitlab.py
and all the tests in tests/test_cronjob.py
:
TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS \
CKI_PYTEST_ARGS="tests/test_gitlab.py::TestGitLabParseURL::test_get_pipeline tests/test_cronjob.py" \
tox -e test
Usually test cases are unique, so you can often select tests by substring with -k
, for example:
TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS \
CKI_PYTEST_ARGS="-k test_get_pipeline" \
tox -e test
Finally, another useful set of filters is:
--lf
,--last-failed
: Rerun only the tests that failed at the last run (or all if none failed)--ff
,--failed-first
: Run all tests, but run the last failures first.
Printing captured logs and summary
Pytest will capture any output generated during execution by default.
If you want to read stdout, stderr, or even specifically logs
,
you’ll need to combine a few options:
--capture=method
Per-test capturing method: one offd|sys|no|tee-sys
.-s
is a shortcut for--capture=no
, which makes any output to be printed when it happens, mixed with the standard pytest logs; useful when you want toprint()
something in the testing code. Default: fd.--show-capture={no,stdout,stderr,log,all}
Controls how captured stdout/stderr/log is shown on failed tests. Note that only the root logger is captured, so make sure other loggers are propagating during tests. Default: all.-r chars
Show extra test summary info (including the captured text) as specified by chars: (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed, (p)assed, (P)assed with output, (a)ll except passed, or (A)ll. (w)arnings are enabled by default (see –disable-warnings), ‘N’ can be used to reset the list. Default: ‘fE’;cki_lint.sh
default: ’s’.--log-cli-level=DEBUG
to see colored logs under the live log call section. Differently from the--show-capture=log
, which shows all the captured logs in the summary at the end of the run according to the-r
filter,--log-cli-level
will shows logs as they happen, between each test. Useful to human-check which test is generatingERROR
logs. Beware that it will only capture logs greater or equal toCKI_LOGGING_LEVEL
.
Subtest caveats: if all failed assertions reside under a subtest context,
pytest will count the test as passed even if we’re using pytest-subtests.
You will see the subtests failures as “SUBFAIL” in the summary, but their logs won’t be included
if you are using -rf
. In that case, if you want to read logs from failed subtest in the summary,
you will need to use -rA
instead.
Note also that tests doing self.assertLogs
will override the logging handlers,
which means they won’t be captured by pytest, and therefore won’t be reported.
Example, showing warning and errors logs as they happen (which ideally shouldn’t exist and instead
be caught by assertLogs
), reporting all logs in the summary, this time including debug and info,
about everything but passed tests:
TOX_OVERRIDE="testenv.passenv+=CKI_PYTEST_ARGS,CKI_LOGGING_LEVEL" \
CKI_LOGGING_LEVEL="DEBUG" \
CKI_PYTEST_ARGS="--log-cli-level=WARNING --show-capture=log -ra" \
tox -e test
Disable coverage report
An additional useful environment variable is CKI_COVERAGE_ENABLED
,
which can be used to disable the coverage report, leaving a cleaner output from tests.
TOX_OVERRIDE=testenv.passenv+=CKI_COVERAGE_ENABLED \
CKI_COVERAGE_ENABLED="false" \
tox -e test