FlakeFighters

A test is flaky when it can pass and fail on the same code. When a flaky test fails, it slows everyone down: CI pipelines go red for no reason, developers rerun jobs until they turn green, and real failures start to get ignored.

FlakeFighters is a pytest plugin, built as part of the TestFLARE project, that tries to answer the question: is this a genuine failure, or is it flaky?

What it does

Most existing pytest plugins deal with flakiness by rerunning failed tests until they pass. FlakeFighters instead brings together several flaky-test detection techniques from research, which it calls “flakefighters”, and uses them to classify each failure as genuine or flaky.

Flaky failures are reported separately and can optionally be rerun or suppressed, so they don’t block a CI/CD pipeline. Results can be written to JSON, HTML or JUnit XML, and test history can be kept in a local or remote database.

uv add --dev pytest-flakefighters
uv run pytest --flakefighters

My contribution

My role on FlakeFighters has been supporting the work and reviewing code rather than leading development.

• Code review: I review and merge pull requests from the team, checking that changes are correct, consistent and clearly documented.
• Supporting the project: I help with the wider work around the tool, including contributing to project decision meetings, raising issues and creating material, such as the poster below, to share the work with others in the field.

Poster