Joseph Wilk

There has been lots of activity around the value of metrics for source code and tests. In the Ruby world tools like metric_fu provide a wealth of analysis.

While working on my Cucumber talk for Rails Underground I started investigating how we could apply metrics to the customer focused plain text of Cucumber. For those not familiar with Cucumber it’s an acceptance testing framework which allows non-technical people to write plain-text describing the behaviors of their system. The developers/testers map the plain-text to tests.

Having spent time teaching people about the plain-text side of Cucumber I often found myself recommending the same guidelines and plain-text anti-patterns. This lead me to think about providing metrics scoring the customers plain-text.

Why would we want plain-text acceptance test metrics?

• Help plain-text beginners avoid bad practices early on.
• Help improve the quality of plain-text
• Help quality review with a large frequency of incoming features

Why does the quality of the plain-text matter?

Why focus on quality, the plain-texts primarily goal is to be easy for customer to use?

• The developer builds the Domain specific language via mapping plain text to ruby. Higher quality plain-text could make it easier to manage these mappings without any major impact to readability.
• Higher quality text is easier to read, edit and understand.

Who would find it useful?

Initially Developers.

• In some scenarios the developers write the features from discussions and give to the customer to review.
• Developers may tweak/review customer written changes/features.
• Developers often edit/tweak plain-text from the customer to enable reuse of existing test code .
• In open source projects often developers write Cucumber features. Metrics are something they are comfortable with.

Can you measure quality in plain-text?

First its important to distinguish acceptance tests from pure plain-text. Within acceptance tests we have some degree of structure, for example using Given/When/Then to describe scenarios.

Cucumber Example:

Scenario: Eating all cucumbers
  Given there are 5 cucumbers
  When I eat 5 cucumbers
  Then I should have 0 cucumbers

This structure reduces the complexity of analysing the quality of the text. It provides us with different structural elements which have different rules/guidelines on what their content should be.

The problem with measuring the quality of text is that it is far more subjective in than in code. So while we cannot be absolute in our assessment of quality we can try and codify smells that *could* indicate areas in the text that *could* be improved. This is pretty much true for all metrics, they are guidelines not absolutes (Dan Norths highlights the dangers of absolute metrics in the Parable of Metrics)

So what useful metrics could we look at?

Plain text Metrics

From my experience with Cucumber I would suggest examining:

• Lack of Feature/Story Narrative
  • We don’t know why this feature is useful and who it’s useful to!
• Scenario length
  • Very long scenarios may reflect more abstraction should be used to help make the scenarios easier to read (See declarative vs imperative scenarios: http://www.benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories/)
• Size of Background (like  a before/setup)
  • Increases conceptual overhead for each scenario. More reasons here: http://wiki.github.com/aslakhellesoy/cucumber/background
• Semantically close words across features
  • Can indicate ambiguity around domain terminology
• Frequency of noise words (‘basically’, ‘be able to’, etc)
  • Adds no value and drives me crazy.

Feedback

What do you think of the idea?

Can you think of any other useful plain-text metrics?

“Cucumber needs you to experiment with your favourite Java Virtual Machine based language and connect to Cucumber via JRuby.”

What’s this Cucumber you speak of? Checkout: http://cukes.info/

Wait that’s Ruby, how do I use a JVM based language to play with it?

Cuke4Duke (http://wiki.github.com/aslakhellesoy/cuke4duke) allows writing Cucumber step definitions in Java. This means Java developers can use the Cucumber tool without having to write any Ruby.

Ruby step definitions

Given /I have (\d+) cukes in my belly/ do |n|
  @belly ||= []
  n.to_i.times {|i| @belly << "cuke"}
end

Equivalent Java step definitions

package cukes;

import cuke4duke.Given;
import cuke4duke.Steps;
import java.util.List;
import java.util.ArrayList;

@Steps
public class BellySteps {
    private List<String> belly = new ArrayList<String>();

    @Given("I have (\\d+) cukes in my belly")
    public void bellyCukes(int cukes) {
        for(int i = 0; i < cukes; i++) {
            belly.add("cukes");
        }
    }
}

This works by connecting Java to Cucumber via JRuby (http://jruby.codehaus.org/).

So in theory if your language runs on the JVM you can use JRuby, and hence use Cucumber and its wonderful Gherkin language.

So what are you waiting for! Pick up your favourite JVM language and arm yourself with Cucumber!

Some Example JVM languages:

• Groovy
• Clojure
• Scala
• Ioke

Good luck and safe Cuking

FutureRuby was an exceptional conference and I was excited to be a part of such a creative group of people. I talked about Cucumber, looking at what it is, how to use it, and why to use it. Useful links for Cucumber:

• Wiki http://wiki.github.com/aslakhellesoy/cucumber
• Source code http://github.com/aslakhellesoy/cucumber.

I demonstrated using Cucumber to test a simple IPhone application. To make the IPhone testing a little more palatable I used a little gem called IRobat. Its very rough around the edges and in no way complete but you can take a look at the IRobat code on Github.

No Cucumbers were harmed (just mildly shaken up) in the making of this presentation.

What people were saying about the talk:

I recently wrote an article for ‘The European Software Tester’ magazine which looked at ways Cucumber, Acceptance tests in general and practices around Behaviour Driven Development can help avoid the gap between what the customer wants and what they get. I’m happy to say the article has been published and is also available digitally:

European Software Tester – 09 March Issue - Page 28

We want to be able to commit our code frequently to prevent merge headaches.

“the longer you wait, the more your code will diverge from your teammates. If you don’t commit often you rob them of the opportunity to reduce merge hell.” Aslak Hellesøy

When dealing with Cucumber and Features/Scenarios we may find we want to commit part way through a scenario but we won’t because:

1. We don’t want to break the build
2. We don’t want to pollute the build with lots of pending steps.

A common solution to this problem is to create two streams for running the features:

1. In-progress
   • If an in-progress scenario fails then the build carries on.
   • If an in-progress scenario passes then the build fails (This is very similar to how Rspec works with pending)
2. Finished
• If a completed scenario fails it causes the build to fail.

We can implement this model using Cucumber’s new Tag feature. We can tag Scenarios and Features with @in-progress and use this tag to help exclude in-progress Features/Scenarios from the finished build.

@in-progress
Feature:
  In order to avoid merge headaches
  As a developer
  I want to tag my features and scenarios with a in-progress tag

  @in-progress
  Scenario: I'm not finished yet
    Given ...
    When ...
    Then ...

The Rake tasks

Finished features/scenarios task

We prefix tags with ~ to exclude features or scenarios having that tag

  desc "Run finished features"
  Cucumber::Rake::Task.new(:finished) do |t|
    t.cucumber_opts = "--format progress --tags ~in-progress"
  end

In-progress features/scenarios task

  desc "Run in-progress features"
  Cucumber::Rake::Task.new(:in_progress) do |t|
    t.cucumber_opts = "--require formatters/ --format Cucumber::Formatter::InProgress --tags in-progress"
  end

We require a special formatter Cucumber::Formatter::InProgress which is essential for making the task work. This formatter as well as giving helpful output changes the command line exit codes of Cucumber. This is kind of crazy but only within the formatter do we have enough information to decided if we should fail or pass. The formatter only returns a failure exit code if there were any scenarios which passed. So unlike the default exit codes failing steps will not cause a failure exit code.

Full Source Code

Also available at: http://github.com/josephwilk/cucumber_cocktails/tree/master

(more…)

I was speaking in Edinburgh at Scotland on Rails 2009 about Cucumber and Rspec.

You can watch the recorded full talk.

I’ve also posted the slides from the presentation and uploaded the screencasts used in the presentation in both high and low resolutions. They are accessible from links within the presentation.

Here are some of the useful links from the presentation:

• Cuke’s official website – http://cukes.info
• Cucumer’s github wiki – http://wiki.github.com/aslakhellesoy/cucumber
• Rspec’s website – http://rspec.info
• The Rspec book – Pragmatic book store – http://www.pragprog.com/titles/achbd/the-rspec-book
• In-memory browser testing – Webrat – http://wiki.github.com/brynary/webrat
• Browser based testing – Watir – http://wtr.rubyforge.org/
• Browser based testing – Selenium – http://seleniumhq.org/
• Headless Browser testing – Celerity – http://celerity.rubyforge.org/
• Using Celerity from within Ruby – Culerity – http://upstream-berlin.com/2009/01/28/culerity-full-stack-rails-testing-with-cucumber-and-celerity/
• Website for checking regular expressions – Rubular – http://www.rubular.com/

I would like to thank the organisers, everyone who came to listen and speak at the conference. It was a pleasure to be a part of such an enthusiastic group of people in such a beautiful venue.

Cucumber as of version 0.2 has removed the GivenScenario feature. GivenScenario was introduced in the original story runner to allow calling a scenario from another within the same feature.

Scenario: setup
  Given ...

 Scenario: example
   GivenScenario setup
  ...

I initially thought this was a great feature (as mentioned in Rspec stories from the trenches). However when I was using GivenScenario I was thinking as a programmer, which is usually where you start going off the tracks when writing features.

Lets look at an example of GivenScenario
(more…)

I’m really excited to be giving a talk at this years Scotland on Rails conference in Edinburgh.

I’ll be talking about working outside-in with Cucumber and RSpec. Having used Cucumber and as a member of the Cucumber core developer team I hope to share lots of experiences and lessons about getting the most out of the tool.

It’s looking like a great line up with with some really interesting presentations across a broad number of topics. The keynotes speakers are Michael Feathers and Marcel Molina, Jnr.

If you’re going to be in Edinburgh for the conference or have any burning questions about Cucumber, let me know.