Python testing¶

Writing tests¶

We have multiple resources for writing new unit tests for Django, Wagtail, and Python code:

• CFPB Django and Wagtail unit testing documentation
• The Django testing documentation
• The Wagtail testing documentation
• Real Python's "Testing in Django"

Testing Elasticsearch¶

When writing tests that rely on a running Elasticsearch service, consider using the search.elasticsearch_helpers.ElasticsearchTestsMixin mixin:

from django.test import TestCase

from search.elasticsearch_helpers import ElasticsearchTestsMixin


class MyTests(ElasticsearchTestsMixin, TestCase):
    def test_something(self):
        self.rebuild_elasticsearch_index()

        # test something that relies on the Elasticsearch index

Refer to the mixin's source code for additional details on its functionality.

Prerequisites¶

If you have set up a standalone installation of consumerfinance.gov, you'll need to activate your virtual environment before running the tests:

workon consumerfinance.gov

If you have not set up the standalone installation of consumerfinance.gov, you can still run the tests if you install Tox in your local installation of Python:

pip install tox

If you have set up a Docker-based installation of consumerfinance.gov, you can run the tests there by accessing the Python container's shell:

docker-compose exec python sh

Running tests¶

Our test suite can either be run in a local virtualenv or in Docker. Please note, the tests run quite slow in Docker.

To run the the full suite of Python tests using Tox, make sure you are in the consumerfinance.gov root and then run:

tox

Tox runs different isolated Python environments with different versions of dependencies. We use it to format and lint our Python files, check out import sorting, and run unit tests in Python 3.8.

To recreate the virtualenv for testing, run

tox -r

For additional information on tox arguments, refer to the tox documentation

You can select specific environments using -e.

Running tox by itself is the same as running:

tox -e lint -e unittest

These default environments are:

• lint, which runs our linters. We require this environment to pass in CI.
• validate-migrations, which checks for any missing Django migrations. We require this environment to pass in CI.
• unittest, which runs unit tests against the current production versions of Python, Django, and Wagtail. We require this environment to pass in CI.

Tests will run against the default Django database.

If you would like to run only a specific test, or the tests for a specific app, you can provide a dotted path to the test as the final argument to any of the above calls to tox:

tox -e unittest -- regulations3k.tests.test_regdown

If you would like to skip running Django migrations when testing, set the SKIP_DJANGO_MIGRATIONS environment variable to any value before running tox.

Formatting¶

We use ruff to do black-compatible formatting of our Python code. ruff is invoked by Tox using the lint environment (this will also run bandit to do Python-specific static security analysis).

tox -e lint

This will run ruff in check-only mode. To automatically fix any formatting issues it flags, run:

ruff format

Linting¶

We also use ruff to ensure compliance with PEP8 style guide, Django coding style guidelines, and the CFPB Python style guide.

We also use Bandit to find any common security issues in our Python code.

ruff and bandit can all be run using the Tox lint environment:

tox -e lint

This will run ruff in check-only mode. To automatically (try to) fix any linting issues it flags, run:

ruff check --fix

From the root of consumerfinance.gov.

Coverage¶

To see Python code coverage information immediately following a test run, you can add the coverage env to the list of envs for tox to run:

tox -e lint -e unittest -e coverage

You can also run coverage directly to see coverage information from a previous test run:

coverage report -m

To see coverage for a limited number of files, use the --include argument to coverage and provide a path to the files you wish to see:

coverage report -m --include=./cfgov/regulations3k/*

Test output¶

Python tests should avoid writing to stdout as part of their normal execution. To enforce this convention, the tests can be run using a custom Django test runner that fails if anything is written to stdout. This test runner is at cfgov.test.StdoutCapturingTestRunner and can be enabled with the TEST_RUNNER environment variable:

TEST_RUNNER=core.testutils.runners.StdoutCapturingTestRunner tox -e unittest

This test runner is enabled when tests are run automatically on GitHub Actions, but is not used by default when running tests locally.

GovDelivery¶

If you write Python code that interacts with the GovDelivery subscription API, you can use the functionality provided in core.govdelivery.MockGovDelivery as a mock interface to avoid the use of patch in unit tests.

This object behaves similarly to the real govdelivery.api.GovDelivery class in that it handles all requests and returns a valid (200) requests.Response instance.

Conveniently for unit testing, all calls are stored in a class-level list that can be retrieved at MockGovDelivery.calls. This allows for testing of code that interacts with GovDelivery by checking the contents of this list to ensure that the right methods were called.

This pattern is modeled after Django's django.core.mail.outbox which provides similar functionality for testing sending of emails.

The related classes ExceptionMockGovDelivery and ServerErrorMockGovDelivery can similarly be used in unit tests to test for cases where a call to the GovDelivery API raises an exception and returns an HTTP status code of 500, respectively.