Joseph Wilk

Joseph Wilk

Programming bits and bobs

Pairwise Testing With Cucumber

Combinatorial testing is a difficult problem. Having to test a small number of inputs  can result in a combinatorial explosion of possible permutations. In Cucumber we see this problem in Scenario Outlines where we can have a large number of rows for the Examples table.

We want to reduce the combinations to a more manageable size while still providing effective fault detection.

Pairwise testing provides one method of achieving this. However providing effective fault detection is dependent on the suitability of Pairwise to the data/system, it is not guaranteed.

What is Pairwise testing?

Pairwise testing (also called All-pairs testing or 2-way testing) is a way of generating a test suite which covers all input combinations of two and is therefore much smaller than exhaustive suites.

To put this in perspective a system of 72 binary inputs with Pairwise testing would require 28 combinations. Exhaustive testing would require 37,778,931,862,957,161,709,568 combinations!

Sounds great, but how does the method preserve good defect detection?

Pairwise relies on a simple principle:

“most faults are caused by interactions of at most two factors”The Combinatorial Design Approach to Automatic Test Generation

Pairwise focuses on the minimal set of inputs (1 or 2 interactions) that cover the most likely causes of faults.

So whats the catch?

Pairwise does not guarantee coverage of all faults of at most two factors. It only provides coverage of faults reachable from the inputs and values you select when generating the Pairwise set. Also the general principle that faults are caused by at most two factors does not guarantee that your data sets faults will follow that distribution. And hence Pairwise testing might miss a proportion of faults.

Pairwise Cucumber example

Here is a scenario taken from a web based system which deals with sports events.

Scenario Outline: Visiting events with another events media
  Given I have a <event without media>
  And I have a <media item> attached to <event with media>
  When I go to the <media item> page for the <event without media>
  Then I should be redirected
  And I should see the <media item> in the <event with media>
  Examples:
    |media item|event without media|event with an media|
    ...

The permutations for the Examples table cells are:

media item: [Image, Video, Music]
event with media: [Football, Basketball, Soccer]
event without media: [Football, Basketball, Soccer]

There are a total of 27 possible permutations. So lets see what permutations Pairwise testing would suggest.

To generate the combinations I used a Ruby based tool I’ve written called Pairwise which uses the in-parameter-order Pairwise generation strategy (http://ranger.uta.edu/~ylei/paper/ipo-tse.pd).

This tool outputs a table ready to be used in a Cucumber feature:

 | media item | event without media | event with media |
 | Image      | Football            | Football         |
 | Image      | Basketball          | Basketball       |
 | Image      | Soccer              | Soccer           |
 | Video      | Football            | Soccer           |
 | Video      | Basketball          | Football         |
 | Video      | Soccer              | Basketball       |
 | Music      | Football            | Basketball       |
 | Music      | Basketball          | Soccer           |
 | Music      | Soccer              | Football         |

That’s 11 permutations covering all possible input pairs.

Giving the final Scenario:

Scenario Outline: Visiting events with another events media
  Given I have a <event without media>
  And I have a <media item> attached to <event with media>
  When I go to the <media item> page for the <event without media>
  Then I should be redirected
  And I should see the <media item> in the <event with media>
Examples:
  | media item | event without media | event with media |
  | Image      | Football            | Football         |
  | Image      | Basketball          | Basketball       |
  | Image      | Soccer              | Soccer           |
  | Video      | Football            | Soccer           |
  | Video      | Basketball          | Football         |
  | Video      | Soccer              | Basketball       |
  | Music      | Football            | Basketball       |
  | Music      | Basketball          | Soccer           |
  | Music      | Soccer              | Football         |

Selecting the right values to test with

In Cucumber we encounter plain text steps which use high level language to hide low level details that are not important for the scenario. For example:

Given a default configuration

By abstracting and pushing the data out of the plain text we are also hiding the values from any Pairwise generation. It is therefore important to realise there will be sets of uncovered pairs and hence potential faults missed.

In deciding what is exposed in the Scenario Outline (as columns in the Examples table) you are selecting what inputs your want to focus the Pairwise generation around. And in turn the input space you want to limit the search for faults to.

When Pairwise testing fails

Pairwise testing is not suitable for all data sets. With any technique that throws away data examples based on a pattern you loose some degree of resolution and you may misfit your data missing important faults.

Here are some cases where Pairwise testing may not be effective.

You have probable combinations which you want to focus on.

If you have certain combinations of inputs that have far more commonality or importance you would want to focus your testing data around these. Pairwise testing ignores any probable importance.

You don’t know how the input variables interact

Pairwise testing assumes each input value carries the same significance on the outcome. If certain inputs have greater influence over the outcome you would want to focus your test data around these.

Your data requires higher order test combinations

Not all data sets follow the distribution of most faults being caused by at most 2 factors. You may have a larger number of faults outside the 2 parameter range. Using higher order test data combinations (Such as 3-wise, 4-wise etc) or completely different test methods may be required to reduce the risk of missing faults.

The real difficulty here is knowing beforehand the distribution of faults.

You want a best practice

There are no testing “best practices” that you can simply “follow” in order to achieve success.

To quote James Marcus Bach:

“blindly applying Pairwise testing to combinatorial testing problems may increase the risk of delivering faulty software.”http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf

Your faults mean people are going to die

In realtime or safety critical systems any faults irrelevant of the number of interacting parameters wants to be detected.

Final words on Pairwise

Shortcuts to Combinatorial problems in testing carry their pitfalls. No matter how you reduce the input permutations you should spend time understanding your combinatorial data set. This is especially the case with Pairwise testing which does not consider the relationships the inputs have on the outputs. Its up to you to examine any interactions or strong influences in order to evaluate Pairwise’s suitability. If Pairwise testing is used thoughtfully with due considerations to its limitations and pitfalls it can provide a powerful tool to testers and Cucumberists alike.

Pairwise tools

Further Reading

Comments