Tools: Your Tests Pass. But Would They Catch This Bug?

Why Mutation Testing Has Been Impractical ## pytest-gremlins Architecture ## Benchmark: pytest-gremlins vs mutmut ## Installation and Usage ## Configuration ## Try It On Your Code You have 90% code coverage, green CI, and you ship. A user reports that >= should have been >. Your tests executed that line but never verified the boundary mattered. Code coverage counts executed lines. Mutation testing injects small bugs and checks whether your tests detect them. If tests still pass after changing >= to >, you found a gap. Traditional tools (mutmut, cosmic-ray) rewrite source files, reload modules, and run the full test suite per mutation. A codebase with 100 mutations and a 10-second test suite takes 17+ minutes. That runtime kills feedback loops. pytest-gremlins achieves 13.8x speedup through three mechanisms: Mutation Switching: All mutations are embedded during a single instrumentation pass. Switching between mutations requires only an environment variable change, eliminating per-mutation file I/O and module reloads. Coverage-Guided Test Selection: The plugin tracks which tests cover each line. When testing a mutation on line 42, it runs only the 3 tests that touch line 42 instead of all 200 tests. Incremental Caching: Results are keyed by content hash of source and test files. Unchanged code skips mutation testing entirely on subsequent runs. Measured on Python 3.12 in Docker: Sequential mode is slower because pytest-gremlins runs additional mutation operators. Parallel mode, safe due to mutation switching (no shared mutable state), delivers the speedup. Cached runs approach instant for unchanged code. Output identifies specific gaps: Each survivor is a line number, the mutation applied, and the gap it reveals. Line 42 has a boundary condition no test verifies. Add to pyproject.toml: Target specific files with --gremlin-targets=src/auth.py. Run this on your highest-coverage module: Survivors show exactly where your tests verify execution but not correctness. Fix one, run again in under 2 seconds with caching. Links: PyPI | GitHub | Docs Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse COMMAND_BLOCK: pip install pytest-gremlins pytest --gremlins --gremlin-parallel --gremlin-cache Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: pip install pytest-gremlins pytest --gremlins --gremlin-parallel --gremlin-cache COMMAND_BLOCK: pip install pytest-gremlins pytest --gremlins --gremlin-parallel --gremlin-cache COMMAND_BLOCK: ================== pytest-gremlins mutation report ================== Zapped: 142 gremlins (89%) Survived: 18 gremlins (11%) Top surviving gremlins: src/auth.py:42 >= → > (boundary not tested) src/utils.py:17 + → - (arithmetic not verified) src/api.py:88 True → False (return value unchecked) ===================================================================== Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: ================== pytest-gremlins mutation report ================== Zapped: 142 gremlins (89%) Survived: 18 gremlins (11%) Top surviving gremlins: src/auth.py:42 >= → > (boundary not tested) src/utils.py:17 + → - (arithmetic not verified) src/api.py:88 True → False (return value unchecked) ===================================================================== COMMAND_BLOCK: ================== pytest-gremlins mutation report ================== Zapped: 142 gremlins (89%) Survived: 18 gremlins (11%) Top surviving gremlins: src/auth.py:42 >= → > (boundary not tested) src/utils.py:17 + → - (arithmetic not verified) src/api.py:88 True → False (return value unchecked) ===================================================================== CODE_BLOCK: [tool.pytest-gremlins] operators = ["comparison", "arithmetic", "boolean"] paths = ["src"] exclude = ["**/migrations/*"] min_score = 80 Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: [tool.pytest-gremlins] operators = ["comparison", "arithmetic", "boolean"] paths = ["src"] exclude = ["**/migrations/*"] min_score = 80 CODE_BLOCK: [tool.pytest-gremlins] operators = ["comparison", "arithmetic", "boolean"] paths = ["src"] exclude = ["**/migrations/*"] min_score = 80 COMMAND_BLOCK: pip install pytest-gremlins pytest --gremlins --gremlin-parallel --gremlin-targets=src/your_critical_module.py Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: pip install pytest-gremlins pytest --gremlins --gremlin-parallel --gremlin-targets=src/your_critical_module.py COMMAND_BLOCK: pip install pytest-gremlins pytest --gremlins --gremlin-parallel --gremlin-targets=src/your_critical_module.py