cki_lint.sh

Runs flake8, pydocstyle, isort --check and pytest under coverage. The script takes care to install the needed dependencies via pip beforehand. For pylint and pytest, only the packages specified as parameters will be checked.

Performs linting and link checking for all README.*.md files. It uses the markdownlint tool to check for markdown errors and a cki_check_links.sh script to check for broken links.

The experimental --fix parameter can be used to automatically fix some of the detected issues. At the moment, this will invoke autopep8 and isort without --check.

For correct editor integration of linters/fixers, the following parameters must be set:

• flake8: --max-line-length 100
• autopep8: --max-line-length 100
• isort: --line-length 100 --force-single-line-imports --force-sort-within-sections

When run in a GitLab CI/CD pipeline for a merge request, the script tries to determine the previous code coverage and will fail if code coverage decreased. This can be skipped by adding [skip coverage check] with an explanation to the merge request description.

Environment variables

Running Python tests and linting

To run both Python tests and linting, execute the following command:

tox

Selecting which tests to run

To run only a custom selection of the tests, use:

TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS \
    CKI_PYTEST_ARGS="tests/file1.py[::classname[::test_name]] [tests/file2.py...]" \
    tox

For example, the following command runs test_get_pipeline in tests/test_gitlab.py and all the tests in tests/test_cronjob.py:

TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS \
    CKI_PYTEST_ARGS="tests/test_gitlab.py::TestGitLabParseURL::test_get_pipeline tests/test_cronjob.py" \
    tox

Usually test cases are unique, so you can often select tests by substring with -k, for example:

TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS \
    CKI_PYTEST_ARGS="-k test_get_pipeline" \
    tox

Finally, another useful set of filters is:

• --lf, --last-failed: Rerun only the tests that failed at the last run (or all if none failed)
• --ff, --failed-first: Run all tests, but run the last failures first.

Printing captured logs and summary

Pytest will capture any output generated during execution by default. If you want to read stdout, stderr, or even specifically logs, you’ll need to combine a few options:

• --capture=method Per-test capturing method: one of fd|sys|no|tee-sys. -s is a shortcut for --capture=no, which makes any output to be printed when it happens, mixed with the standard pytest logs; useful when you want to print() something in the testing code. Default: fd.
• --show-capture={no,stdout,stderr,log,all} Controls how captured stdout/stderr/log is shown on failed tests. Note that only the root logger is captured, so make sure other loggers are propagating during tests. Default: all.
• -r chars Show extra test summary info (including the captured text) as specified by chars: (f)ailed, (E)rror, (s)kipped, (x)failed, (X)passed, (p)assed, (P)assed with output, (a)ll except passed, or (A)ll. (w)arnings are enabled by default (see –disable-warnings), ‘N’ can be used to reset the list. Default: ‘fE’; cki_lint.sh default: ’s’.
• --log-cli-level=DEBUG to see colored logs under the live log call section. Differently from the --show-capture=log, which shows all the captured logs in the summary at the end of the run according to the -r filter, --log-cli-level will shows logs as they happen, between each test. Useful to human-check which test is generating ERROR logs. Beware that it will only capture logs greater or equal to CKI_LOGGING_LEVEL.

Subtest caveats: if all failed assertions reside under a subtest context, pytest will count the test as passed even if we’re using pytest-subtests. You will see the subtests failures as “SUBFAIL” in the summary, but their logs won’t be included if you are using -rf. In that case, if you want to read logs from failed subtest in the summary, you will need to use -rA instead.

Note also that tests doing self.assertLogs will override the logging handlers, which means they won’t be captured by pytest, and therefore won’t be reported.

Example, showing warning and errors logs as they happen (which ideally shouldn’t exist and instead be caught by assertLogs), reporting all logs in the summary, this time including debug and info, about everything but passed tests:

TOX_OVERRIDE="testenv.passenv+=CKI_PYTEST_ARGS,CKI_LOGGING_LEVEL" \
    CKI_LOGGING_LEVEL="DEBUG" \
    CKI_PYTEST_ARGS="--log-cli-level=WARNING --show-capture=log -ra" \
    tox

Disable coverage report

An additional useful environment variable is CKI_COVERAGE_ENABLED, which can be used to disable the coverage report, leaving a cleaner output from tests.

TOX_OVERRIDE=testenv.passenv+=CKI_COVERAGE_ENABLED \
    CKI_COVERAGE_ENABLED="false" \
    tox

Disabling linters

To skip certain linters, specify them in the CKI_DISABLED_LINTERS variable as a space-separated list. For example:

TOX_OVERRIDE=testenv.passenv+=CKI_DISABLED_LINTERS \
    CKI_DISABLED_LINTERS="markdownlint pylint" \
    tox

The above command skips the markdownlint and pylint linters when running tox.

To skip all linters, set CKI_DISABLED_LINTERS to “all”:

TOX_OVERRIDE=testenv.passenv+=CKI_DISABLED_LINTERS \
    CKI_DISABLED_LINTERS="all" \
    tox

You can also provide both CKI_PYTEST_ARGS and CKI_DISABLED_LINTERS if you wish to skip linters and run specific tests. For example:

TOX_OVERRIDE=testenv.passenv+=CKI_PYTEST_ARGS,CKI_DISABLED_LINTERS \
    CKI_PYTEST_ARGS="tests/test_gitlab.py::TestGitLabParseURL::test_get_pipeline tests/test_cronjob.py" \
    CKI_DISABLED_LINTERS="all" \
    tox

The above command skips all linters and runs the test_get_pipeline test in tests/test_gitlab.py, as well as all tests in tests/test_cronjob.py.