Combinatorial testing is a difficult problem. Having to test a small number of inputs can result in a combinatorial explosion of possible permutations. In Cucumber we see this problem in Scenario Outlines where we can have a large number of rows for the Examples table.
We want to reduce the combinations to a more manageable size while still providing effective fault detection.
Pairwise testing provides one method of achieving this. However providing effective fault detection is dependent on the suitability of Pairwise to the data/system, it is not guaranteed.
What is Pairwise testing?
Pairwise testing (also called All-pairs testing or 2-way testing) is a way of generating a test suite which covers all input combinations of two and is therefore much smaller than exhaustive suites.
To put this in perspective a system of 72 binary inputs with Pairwise testing would require 28 combinations. Exhaustive testing would require 37,778,931,862,957,161,709,568 combinations!
Sounds great, but how does the method preserve good defect detection?
Pairwise relies on a simple principle:
“most faults are caused by interactions of at most two factors” – The Combinatorial Design Approach to Automatic Test Generation
Pairwise focuses on the minimal set of inputs (1 or 2 interactions) that cover the most likely causes of faults.
So whats the catch?
Pairwise does not guarantee coverage of all faults of at most two factors. It only provides coverage of faults reachable from the inputs and values you select when generating the Pairwise set. Also the general principle that faults are caused by at most two factors does not guarantee that your data sets faults will follow that distribution. And hence Pairwise testing might miss a proportion of faults.
Pairwise Cucumber example
Here is a scenario taken from a web based system which deals with sports events.
Scenario Outline: Visiting events with another events media Given I have a <event without media> And I have a <media item> attached to <event with media> When I go to the <media item> page for the <event without media> Then I should be redirected And I should see the <media item> in the <event with media> Examples: |media item|event without media|event with an media| ...
The permutations for the Examples table cells are:
media item: [Image, Video, Music] event with media: [Football, Basketball, Soccer] event without media: [Football, Basketball, Soccer]
There are a total of 27 possible permutations. So lets see what permutations Pairwise testing would suggest.
This tool outputs a table ready to be used in a Cucumber feature:
| media item | event without media | event with media | | Image | Football | Football | | Image | Basketball | Basketball | | Image | Soccer | Soccer | | Video | Football | Soccer | | Video | Basketball | Football | | Video | Soccer | Basketball | | Music | Football | Basketball | | Music | Basketball | Soccer | | Music | Soccer | Football |
That’s 11 permutations covering all possible input pairs.
Giving the final Scenario:
Scenario Outline: Visiting events with another events media Given I have a <event without media> And I have a <media item> attached to <event with media> When I go to the <media item> page for the <event without media> Then I should be redirected And I should see the <media item> in the <event with media> Examples: | media item | event without media | event with media | | Image | Football | Football | | Image | Basketball | Basketball | | Image | Soccer | Soccer | | Video | Football | Soccer | | Video | Basketball | Football | | Video | Soccer | Basketball | | Music | Football | Basketball | | Music | Basketball | Soccer | | Music | Soccer | Football |
Selecting the right values to test with
In Cucumber we encounter plain text steps which use high level language to hide low level details that are not important for the scenario. For example:
Given a default configuration
By abstracting and pushing the data out of the plain text we are also hiding the values from any Pairwise generation. It is therefore important to realise there will be sets of uncovered pairs and hence potential faults missed.
In deciding what is exposed in the Scenario Outline (as columns in the Examples table) you are selecting what inputs your want to focus the Pairwise generation around. And in turn the input space you want to limit the search for faults to.
When Pairwise testing fails
Pairwise testing is not suitable for all data sets. With any technique that throws away data examples based on a pattern you loose some degree of resolution and you may misfit your data missing important faults.
Here are some cases where Pairwise testing may not be effective.
You have probable combinations which you want to focus on.
If you have certain combinations of inputs that have far more commonality or importance you would want to focus your testing data around these. Pairwise testing ignores any probable importance.
You don’t know how the input variables interact
Pairwise testing assumes each input value carries the same significance on the outcome. If certain inputs have greater influence over the outcome you would want to focus your test data around these.
Your data requires higher order test combinations
Not all data sets follow the distribution of most faults being caused by at most 2 factors. You may have a larger number of faults outside the 2 parameter range. Using higher order test data combinations (Such as 3-wise, 4-wise etc) or completely different test methods may be required to reduce the risk of missing faults.
The real difficulty here is knowing beforehand the distribution of faults.
You want a best practice
There are no testing “best practices” that you can simply “follow” in order to achieve success.
To quote James Marcus Bach:
“blindly applying Pairwise testing to combinatorial testing problems may increase the risk of delivering faulty software.” – http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
Your faults mean people are going to die
In realtime or safety critical systems any faults irrelevant of the number of interacting parameters wants to be detected.
Final words on Pairwise
Shortcuts to Combinatorial problems in testing carry their pitfalls. No matter how you reduce the input permutations you should spend time understanding your combinatorial data set. This is especially the case with Pairwise testing which does not consider the relationships the inputs have on the outputs. Its up to you to examine any interactions or strong influences in order to evaluate Pairwise’s suitability. If Pairwise testing is used thoughtfully with due considerations to its limitations and pitfalls it can provide a powerful tool to testers and Cucumberists alike.
Ruby based tool which supports usage through Ruby or on the command line: http://wiki.github.com/josephwilk/pairwise
A free web-based tool from Testers Desk that provides generation of Pairwise data sets: http://www.testersdesk.com/TestersDesk/
Pairwise – http://www.pairwise.org/
Combinatorial Testing – http://www.combinatorialtesting.com/
Pairwise Testing: A Best Practice That Isn’t – http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
A Test Generation Strategy for Pairwise Testing – [http://ranger.uta.edu/~ylei/paper/ipo-tse.pdf]