Quantifying the Effectiveness of Testing via Efficient Residual Path Profiling

Trishul Chilimbi, Aditya Nori, Kapil Vaswani

MSR-TR-2007-62 |

Writing correct programs is hard. Proving that they are correct is even harder. Consequently, testing is extensively used for uncovering bugs in large, complex software. Since testing software exhaustively is infeasible, well designed regression test suites aim to anticipate all reasonable software usage scenarios and generate test cases that exercise those behaviors. Unfortunately, testers today have no way of knowing how much of real-world software usage was untested by their regression suite. While collecting path profiles of deployed software would provide this information, profiling overheads preclude this. This often results in released software shipping with bugs that could have been detected with a better test suite. Recent advances in low-overhead path profiling provide the opportunity to rectify this deficiency and perform residual path profiling on deployed software. Residual path profiling identifies all paths executed by deployed software that were untested during software development. We extend prior research to perform low-overhead interprocedural path profiling. We demonstrate experimentally that low-overhead path profiling, both intraprocedural and interprocedural, provides valuable quantitative information on testing effectiveness. We also show that residual edge profiling is inadequate as a significant number of untested paths include no new untested edges.