Applying Higher Strength Combinatorial Criteria to Test Case Prioritization: A Case Study

Abstract

Faults in software systems often occur due to interactions between parameters. Several studies show that faults are caused by 2-way through 6-way interactions of parameters. In the context of test suite prioritization, we have studied prioritization by 2-way inter-window interaction coverage and found that this criterion is effective at finding faults quickly in the test execution cycle. However, since faults may be caused by interactions between more than 2 parameters, in this paper, we provide a greedy algorithm for test suite prioritization by $n$-way combinatorial coverage of inter-window interactions. While greedy algorithms that generate Combinatorial Interaction Test suites enumerate and track the coverage of all possible $t$-tuples and constraints, we have noticed that our user-session-based test suites often do not contain every possible $t$-tuple, and we can take advantage of this in our algorithm by only storing $t$-tuples that appear in the test suite. Our empirical study shows both time and memory usage associated with our algorithm for 3-way inter-window parameter-value interaction coverage. Further, we conduct an empirical study where we compare 2-way and 3-way combinatorial coverage of inter-window parameter interactions in terms of the rate of fault detection for a web application called Schoolmate and a user-session-based test suite. Our results show that the rate of fault detection for 2-way and 3-way prioritization are within $1\mathrm{\%}$ of each other, but 2-way provides a slightly better result. A closer look at the characteristics of the web application, test cases, and faults reveals that most faults are triggered by 2-way interactions. We motivate the need for future work to examine a larger set of empirical studies to identify characteristics of web applications that benefit from prioritization with higher strength inter-window event interaction coverage.