GTAC 2015: Coverage is Not Strongly Correlated with Test Suite Effectiveness

http://g.co/gtac Slides: https://docs.google.com/presentation/d/1ttVaFhAfnbO0027Hd_Tz47P5xPHtLf4mLr12xFRMKig/pub Laura Inozemtseva (University of Waterloo) The coverage of a test suite is often used as a proxy for its ability to detect faults. However, previous studies that investigated the correlation between code coverage and test suite effectiveness have failed to reach a consensus about the nature and strength of the relationship between these test suite characteristics. Moreover, many of the studies were done with small or synthetic programs, making it unclear whether their results generalize to larger programs, and some of the studies did not account for the confounding influence of test suite size. We have extended these studies by evaluating the relationship between test suite size, coverage, and effectiveness for realistic Java programs; our study is the largest to date in the literature. We measured the statement coverage, decision coverage, and modified condition coverage of these suites and used mutation testing to evaluate their fault detection effectiveness. We found that there is a low to moderate correlation between coverage and effectiveness when the number of test cases in the suite is controlled for. In addition, we found that stronger forms of coverage do not provide greater insight into the effectiveness the suite.