4.2. Testing

What are software tests?

Software tests are automated procedures designed to verify that a piece of software behaves as expected. They are fundamental to ensuring reliability, functionality, and preventing unintended changes, known as regressions.

Tests are typically categorized by their scope:

Unit Tests: Focus on the smallest testable parts of an application, such as individual functions or methods, in isolation. For example, a unit test might verify that a data normalization function correctly scales a feature to a [0, 1] range.
Regression Tests: Ensure that recent code changes have not adversely affected existing features. These tests are run frequently to catch bugs introduced during development.
End-to-End (E2E) Tests: Simulate a complete user workflow from start to finish. In an MLOps context, an E2E test could involve running an entire prediction pipeline, from raw data input to model prediction output, to ensure all components integrate correctly.

Why is testing critical in MLOps projects?

Testing is indispensable in any professional software project, but it carries unique importance in MLOps:

Quality Assurance: Confirms that data pipelines, models, and APIs meet their specified requirements.
Regression Prevention: In MLOps, regressions can be subtle, such as a slight drop in model accuracy. A robust test suite helps catch these issues before they impact users.
Confidence in Refactoring: Allows developers to improve and optimize code with confidence, knowing that a suite of tests will validate the continued correctness of their logic.
Living Documentation: Tests serve as executable examples of how the code is intended to be used, which is often clearer than static documentation.

While print() statements are useful for immediate debugging, they provide no lasting guarantees. Automated tests, on the other hand, continuously validate behavior with every code change. This is especially vital in a dynamic language like Python, where the compiler doesn't perform static type checking at compile time.

Which tool should you use for Python testing?

While Python has a built-in unittest module, its syntax can be verbose. We strongly recommend pytest, a modern, powerful framework that simplifies test creation and scales from simple functions to complex applications.

A pytest test is just a function with an assert statement:

# content of tests/test_sample.py
def inc(x: int) -> int:
    return x + 1

def test_answer():
    # Assert that the function's output matches the expected behavior
    assert inc(3) == 4

To execute pytest across your project:

# Install pytest into your "dev" dependency group
uv add --group dev pytest

# Run pytest on the `tests` directory
uv run pytest tests/

You can extend pytest's functionality with plugins:

pytest-cov: Generates code coverage reports to identify which parts of your codebase are not being tested.

# Generate a coverage report for the `src` directory
uv run pytest --cov=src/ tests/

pytest-mock: Exposes a mocker fixture that patches objects for the duration of one test and restores them afterwards.
pytest-xdist: Executes tests in parallel, significantly reducing runtime by utilizing all available CPU cores.

# Run tests in parallel using all available cores
uv run pytest -n auto tests/

How do you enforce a minimum coverage?

Producing a coverage report is not the same as enforcing one. A report is a number a human may or may not read; a gate is a threshold that fails the build. pytest-cov provides the gate with --cov-fail-under=<percent>, and both reference repositories ship one, at deliberately different values:

# MLOps Python Package — the suite genuinely covers every line and branch
[tool.pytest.ini_options]
addopts = [
  "--cov=src",
  "--cov-report=term-missing",
  # The suite actually covers every line and branch, so the gate says so: a drop is a
  # regression to fix, not a threshold to lower.
  "--cov-fail-under=100",
]

# Cookiecutter MLOps Package — a scaffold must not fail on the owner's first commit
[tool.pytest.ini_options]
addopts = [
  "--cov=src",
  "--cov-report=term-missing",
  "--cov-fail-under=80",
]

That difference is the whole lesson about choosing a threshold:

Set it to what your suite achieves today, not to an aspiration. A gate above your real number fails immediately and gets removed; a gate far below it protects nothing. Measure first, then write the number down.
A generated project starts at 80. A freshly scaffolded package contains code the owner has not written tests for yet. Demanding 100 from a template means every new project is born red.
A mature package can afford 100. Once a suite really covers every line and branch, the gate turns "coverage dropped" into a build failure instead of a slow, unnoticed erosion. The rule that makes it work: when the gate fails, you add the missing test, you do not lower the number.

Ratchet the value up as your suite improves, and treat lowering it as a change that needs a justification in the commit message.

Can you run tests in parallel and measure coverage at the same time?

Not always, and the reference package is honest about it. It declares pytest-xdist and exposes parallel execution as a separate task, not as part of the gate:

# mise.toml
[tasks.test]
alias = "t"
description = "Run the test suite with coverage (pytest)"
run = "uv run pytest"

[tasks."test:parallel"]
description = "Run the test suite across CPU cores, without the coverage gate (pytest-xdist)"
# Fast local feedback only. pytest-cov and pytest-xdist deadlock on this suite, so the
# coverage gate stays on the serial `test` task that hooks and CI run.
run = "uv run pytest -n auto --no-cov"

pytest-cov and pytest-xdist combine badly on this suite: the two plugins deadlock, so mise run test:parallel explicitly disables coverage with --no-cov. The split that results is a good pattern in general:

mise run test is the authority. It runs serially, with coverage and the --cov-fail-under gate. This is what the pre-push git hook runs and what CI runs, so the number that blocks a merge is always measured the same way.
mise run test:parallel is the fast local loop. You use it while iterating, when you want the answer "did anything break" in a fraction of the time and do not care about coverage yet.

Do not try to make the fast task the authoritative one. A gate you cannot reproduce deterministically is worse than no gate.

How should you configure your project for testing?

First, prevent pytest cache files from being committed to Git by adding .pytest_cache/ to your .gitignore file.

Next, enable pytest support in VS Code by adding the following to your .code-workspace file:

{
    "settings": {
        "python.testing.pytestEnabled": true,
        "python.testing.pytestArgs": [
            "tests"
        ]
    }
}

Finally, define global pytest configurations in your pyproject.toml file to ensure consistency:

[tool.pytest.ini_options]
addopts = [
  "-ra", # summarize every non-passing outcome at the end of the run
  "--strict-config", # a typo in this very section is an error, not a silent no-op
  "--strict-markers", # an unregistered @pytest.mark is an error, not a silent skip
  "--cov=src", # measure coverage of the source tree
  "--cov-report=term-missing", # list the uncovered lines in the terminal
  "--cov-fail-under=100", # fail the run below the threshold (see the section above)
]
# Add the `src` directory to the Python path for imports
pythonpath = ["src"]
# Only collect from the tests directory
testpaths = ["tests"]
# An `xfail` test that unexpectedly passes is a failure, not a pass
xfail_strict = true

[tool.coverage.run]
# Measure branch coverage to check if `if/else` statements are tested
branch = true
source = ["src"]
omit = ["__main__.py"] # Exclude non-testable files

[tool.coverage.report]
show_missing = true
skip_covered = true # keep the report focused on what still needs tests

How should you structure your tests?

Organize tests in a dedicated tests/ directory that mirrors your source code structure. For a module like src/bikes/models.py, the corresponding test file should be tests/test_models.py. Test function names must be prefixed with test_.

src/
    bikes/
        models.py
        metrics.py
        datasets.py
tests/
    test_models.py
    test_metrics.py
    test_datasets.py

This structure keeps your test suite organized and easy to navigate. A popular pattern for structuring individual tests is Given-When-Then, which clearly separates setup, execution, and validation.

def test_inputs_schema_is_valid(inputs_reader: datasets.Reader) -> None:
    # Given: A predefined data schema and an inputs reader
    schema = schemas.InputsSchema

    # When: The input data is read
    data = inputs_reader.read()

    # Then: The data conforms to the schema
    assert schema.check(data) is not None, "Input data validation failed!"

How can you define reusable test components?

pytest fixtures are functions that provide a fixed baseline for tests to build upon. They are ideal for setting up reusable objects like datasets, models, or temporary file paths, eliminating redundant code.

Fixtures can be defined in individual test files or in a central tests/conftest.py file to be shared across the entire test suite.

# in tests/conftest.py
import pytest
import os

@pytest.fixture(scope="session")
def tests_path() -> str:
    """Return the path of the tests folder."""
    file_path = os.path.abspath(__file__)
    parent_directory = os.path.dirname(file_path)
    return parent_directory

@pytest.fixture(scope="function")
def tmp_outputs_path(tmp_path: str) -> str:
    """Return a tmp path for the outputs dataset."""
    return os.path.join(tmp_path, "outputs.parquet")

The scope parameter controls the fixture's lifecycle. A session-scoped fixture is created once for the entire test run, while a function-scoped fixture is recreated for every test.

How do you keep expensive setup from dominating the run?

Choosing a scope is usually a trade-off between isolation (a fresh object per test) and speed (one object for the whole session). Sometimes you can have both, by paying the expensive part once and cheaply cloning it.

The MLOps Python Package hit exactly this case when its MLflow tracking store moved from a directory of files to a SQLite database (see 5.5. AI/ML Experiments). Creating an MLflow SQLite store runs its Alembic schema migrations, which costs several seconds. Doing that per test took the suite from 34 seconds to 339 seconds.

The fix keeps per-test isolation and pays the migration once: a session-scoped fixture builds one migrated, empty database, and a function-scoped fixture copies that file for each test.

@pytest.fixture(scope="session")
def mlflow_db_template(tmp_path_factory: pytest.TempPathFactory) -> str:
    """Return a migrated but empty MLflow database used as a template by every test.

    Creating an MLflow SQLite store runs its Alembic migrations, which costs seconds.
    Paying that once per session and copying the file per test keeps the isolation of a
    fresh database at the cost of a file copy.
    """
    path = tmp_path_factory.mktemp("mlflow") / "template.db"
    services.MlflowService(
        tracking_uri=f"sqlite:///{path}",
        registry_uri=f"sqlite:///{path}",
        experiment_name="Experiment-Template",
        registry_name="Registry-Template",
    ).start()
    return str(path)


@pytest.fixture(scope="function", autouse=True)
def mlflow_service(tmp_path: str, mlflow_db_template: str) -> T.Generator[services.MlflowService]:
    """Return and start the mlflow service."""
    # Each test gets its own SQLite file under tmp_path, so runs stay isolated.
    database = os.path.join(tmp_path, "mlflow.db")
    shutil.copyfile(mlflow_db_template, database)
    service = services.MlflowService(
        tracking_uri=f"sqlite:///{database}",
        registry_uri=f"sqlite:///{database}",
        experiment_name="Experiment-Testing",
        registry_name="Registry-Testing",
    )
    service.start()
    yield service
    service.stop()

Copying the file costs about 0.07 seconds instead of seconds of migrations, bringing the suite back to 63 seconds with the same isolation guarantee. The generalizable rule: when setup is expensive but its result is a cheap, copyable value (a database file, a fitted model, a rendered dataset), build it once at session scope and hand out copies at function scope.

How can you avoid repetitive test scenarios?

To test a function against multiple input scenarios without writing duplicate code, use the @pytest.mark.parametrize decorator. This feature allows you to run the same test function with different argument sets.

import pytest

@pytest.mark.parametrize(
    "name, interval, greater_is_better",
    [
        ("mean_squared_error", [0, float("inf")], False),
        ("mean_absolute_error", [0, float("inf")], False),
        ("r2_score", [float("-inf"), 1.0], True),
    ],
)
def test_sklearn_metric_properties(
    name: str, interval: list, greater_is_better: bool
) -> None:
    # This test will run three times, once for each set of parameters.
    assert isinstance(name, str)
    assert isinstance(interval, list)
    assert isinstance(greater_is_better, bool)

How do you validate program outputs and exceptions?

pytest provides built-in fixtures for capturing output and testing for exceptions.

capsys: Captures anything written to standard output (stdout) and standard error (stderr).
pytest.raises: A context manager that asserts a specific exception is raised.

import json
import pytest

def test_json_output_is_valid(capsys) -> None:
    # Given: A program that prints a JSON object
    print(json.dumps({"key": "value"}))

    # When: The output is captured
    captured = capsys.readouterr()

    # Then: The output is a valid JSON with no errors
    assert captured.err == "", "Stderr should be empty"
    assert json.loads(captured.out), "Stdout should be a valid JSON"

def test_main_raises_error_on_no_configs() -> None:
    # Given: A function that requires configuration
    def run_main(argv: list):
        if not argv:
            raise RuntimeError("No configs provided.")

    # When/Then: Calling the function with no arguments raises a RuntimeError
    with pytest.raises(RuntimeError) as error:
        run_main([])

    assert "No configs provided." in str(error.value)

How do you test code with randomness?

Machine learning code often involves randomness (e.g., train/test splits, model initialization). To make tests deterministic and reproducible, always set a fixed random seed at the beginning of your test.

import numpy as np
from sklearn.model_selection import train_test_split

def test_data_split_is_reproducible():
    # Given: A dataset and a fixed random state
    X = np.arange(100).reshape(50, 2)
    y = np.arange(50)

    # When: The data is split with a fixed random_state
    X_train_1, _, y_train_1, _ = train_test_split(X, y, random_state=42)
    X_train_2, _, y_train_2, _ = train_test_split(X, y, random_state=42)

    # Then: The splits are identical
    np.testing.assert_array_equal(X_train_1, X_train_2)

What are best practices for writing unit tests?

Write Clear, Readable Tests: A test should be easy to understand. Use descriptive names and the Given-When-Then structure to clarify intent.
Keep Tests Independent and Isolated: Each test should be able to run on its own, without depending on the state left by other tests.
Use Fixtures for Setup and Teardown: Leverage fixtures to manage setup and cleanup logic, keeping tests clean and focused.
Test for Edge Cases: Go beyond the "happy path." Test for invalid inputs, empty data, and other edge conditions that could cause failures.
Enforce Coverage, Don't Just Measure It: Start at 80% with --cov-fail-under=80, then ratchet the threshold up as the suite improves. A number nobody enforces erodes silently.
Keep Tests Fast: Slow tests slow down development. Move expensive, reusable setup to a session-scoped fixture, and keep a fast parallel task (mise run test:parallel) for the local loop.
Run Tests Automatically: Integrate your test suite into a CI/CD workflow to catch issues early and often.
Review and Refactor Tests: Just like production code, tests should be reviewed and updated as the codebase evolves.