On Use of Coverage Metrics in Assessing Effectiveness of Combinatorial Test Designs

Combinatorial test suite design is a test generation technique, popular in part due to its ability to achieve coverage and defect finding power approximating that of exhaustive testing while keeping test suite sizes constrained. In recent years, there have been numerous advances in combinatorial test design techniques, in terms of efficiency and usability of methods used to create them as well as in understanding of their benefits and limitations when applied to real world software. Numerous case studies have appeared presenting practical applications of the combinatorial test suite design techniques, often comparing them with manually-created, random, or exhaustive suites. These comparisons are done either in terms of defects found or by applying some code coverage metric. Since many different and valid combinatorial test suites of strength t can be created for a given test domain, the question whether they all have the same coverage properties is a pertinent one.