Testing and code quality
This document describes the current testing strategies and general measures to keep code quality up and its formatting consistent.
Test setup
Problems with standard component testing
FastAPI does provide good tooling for testing the API in a more classical component test setup with pytest. For this application a setup like that would come with a significant amount of work and some serious shortcomings: The backend relies on many external services for its operations: A PostgreSQL database, a Redis database, a SMTP server, one or more runners and optionally LDAP and OIDC providers. These services wouldn’t be available in a component test setup. The solution would be to replace all code that communicates with such services with dummy code during testing time. This could be done either by supplying a different class for database/caching operations that implements the same abstract parent class during testing time (both the PostgreSQL and Redis adapters inherit from an abstract parent class, it would ‘just’ require a dummy implementation of these classes) or my monkey patching. This has two major disadvantages:
for so many services it would be a lot of work
most of the backend’s behavior and logic is actually implemented in these services. If all these services would be replaced by dummy implementations, there wouldn’t remain much else to test for the component tests.
these services, their behavior and correct interfacing by the backend is very crucial and should also be tested
Because of these considerations I chose to prioritize the system and Integration tests. For the time being there are no component tests of Project-W code.
System and Integration testing
The goal of this test setup is to deploy a fully working version of Project-W including PostgreSQL, Redis, runners and more in a way that simulates a real world deployment as much as possible, and then use pytest and httpx to make http requests to the backend, effectively simulating a client. This test setup required the following to work:
GitHub actions as our CI system
Github actions and Github runners are powerful enough to setup multiple docker containers, to build the Project-W container and the Project-W-runner container in it’s dummy version (because the runner doesn’t have enough storage for the full WhisperX version) and then run the pytest test suite.
This test is implemented as matrix test, meaning the same thing is done for each combination of Python version, PostgreSQL version and Redis version that I defined in the action (currently for each the oldest and newest supported by the backend).
In addition to that our CI system also runs pre-commit.ci for code formatting.
pytest fixtures
The docker containers of the backend and runners are started and stopped using pytest fixtures. They also create their config files and flush the PostgreSQL and Redis databases afterwards. This ensures that each test case starts of with a freshly started backend and runner, and clean databases.
Test cases have some control over these fixtures over pytest’s indirect parametrization feature (e.g. some of the contents of config files are handled this way).
Code style and formatting
We aim to have a consistent code styling to increase readability, maintainability and to improve git diffs. To ensure this we use pre-commit hooks. For this we provide a .pre-commit-config.yaml file in the repository root that installs hooks for all sub-projects of Project-W. Just install pre-commit on your system and run pre-commit install while inside the repository.
Python
For python I use various hooks from pre-commit itself like check-python, check-builtin-literals, check-docstring-first as well as ruff for both linting and formatting and codespell for spotting spelling mistakes.
ruff is the most important one of these, it is both a code linter and formatter and acts as a drop-in replacement for Flake8, isort and black. I set it up with the --line-length 100 option to increase the allowed line length as I found the 79 character limit defined by PEP8 to be too restrictive for this project. The point of this is to increase code readability, and having simple and straight forward if statements or prints to be stretched over multiple lines hurts this cause. I found 100 characters per line to be a good sweet spot. Other than that our code should be PEP8 compliant.
Javascript/Typescript
For the Javascript/Typescript code of the frontend I use biome to format the code. Biome is also configured to also check the script sections of all .svelte files to a certain degree however it will ignore any html/css. Most notably explicit usage of the any keyword is banned from the codebase among other things.