Posted on Leave a comment

QA Bite ::: Use Combinatorial Testing to Reduce the Number of Test Cases

In practice, it is usually impossible to test all parameter combinations for real life applications. Say, you have 10 different parameters of a tax advice software where each parameter is true of false. Enumerating all parameter combinations leads to 3,628,800 test cases already. So, testing can always check a few combinations, only.

But, when the software finally goes public, I often see that rare usage patterns occur which have never been tested. There is a neat trick by a research group at NIST: Automated Combinatorial Testing for Software. Richard Kuhn, Raghu Kacker and Yu Lei observed that most errors occur by a change of one or two parameters, only. A two way test with 10 parameters can be done in 10 test cases. Even a 4-way test, where a specific set of 4 parameters is required to trigger a bug, is easy to do with 42 test cases.