Deleting code statements to see if the test suite notices!
I built PseudoSweep, a tool that automatically identifies pseudo-tested methods and statements in Java projects. It instruments the code, runs the test suite, and reports which covered code the tests would not notice going missing. There’s a demo project and a video walkthrough if you’d like to try it [1].
Where tests fall short
Pseudo-tested statements are one kind of oracle gap: covered code that isn’t checked by an assertion. In my ESEM 2025 paper, I compared three ways of finding oracle gaps using checked coverage using a dynamic slicer, checked coverage using an observational slicer, and pseudo-tested statement identification.
Pseudo-tested statement identification found the gaps that most need attention (they had the lowest median mutation score) and was by far the quickest, with a median of about 20 seconds compared with minutes, hours or days for the slicing-based approaches [2].
References
2025
Where tests fall short: Empirically analyzing oracle gaps in covered code
Background: Developers often rely on statement coverage to assess test suite quality. However, statement coverage alone may only lead to 10% fault detection, necessitating more rigorous approaches. While mutation testing is effective, its execution and human analysis costs remain high. Identifying covered statements that are not checked by oracles (e.g., assertions) offers a cost-effective alternative; however, the lack of empirical evidence for selecting the appropriate Oracle Gap Calculation Approach (OGCA) prevents developers from making informed choices. Aims: This knowledge-seeking study compares oracle gap characteristics determined by different OGCAs to assist developers in choosing the most valuable approach for their use cases. Method: Using mixed-method empirical analysis, we conduct an in-depth evaluation of the oracle gaps produced using three OGCAs: Checked Coverage using a Dynamic Slicer (CCDS), Checked Coverage using an Observational Slicer (CCOS), and Pseudo-Tested Statement Identification (PTSI). Across 30 Java classes from six open-source projects, we report on a quantitative evaluation of gap prominence, distribution, fault detection correlation and execution times, as well as results from a qualitative manual inspection of the statement types found in the oracle gaps. Results: The qualitative analysis showed data-loading statements, iteration statements and output updates to be most prominent in the oracle gaps. PTSI identified the oracle gaps with the lowest median mutation score (0.32), highlighting areas requiring more fault detection improvement compared to CCDS (0.76) and CCOS (0.50). PTSI also had the shortest median execution time (19.9 seconds), far quicker than both CCDS (273.2 seconds) and CCOS (5957.1 seconds). Conclusions: PTSI quickly reveals the priority testing areas for improved fault detection, making it an effective OGCA for developers to identify where tests fall short
Software testing remains a crucial practice for ensuring and maintaining code quality. Yet, a critical issue remains: the existence of pseudo-tested statements. Tests cover these statements, but removing them does not trigger test failures. Since no established tools address this challenge, this paper introduces PseudoSweep, a novel tool that automatically identifies pseudotested methods and statements in Java projects. PseudoSweep combines method and statement deletion techniques to reveal these maintenance problems. In addition to explaining the approach used by PseudoSweep, this paper details use-cases and overviews results from experiments with PseudoSweep. The tool is available (including set-up instructions and examples) at https://github.com/PseudoTested/PseudoSweep and there is a video demonstration at https://youtu.be/5QCsu7MbiXI.
