Joseph Wilk

The Current State of the Art

In the Ruby world there is a wealth of metrics which can provide insight into our code. Looking at such things as:

• Structural similar code (Flay)
• Complexity (Flog)
• Cyclic complexity (Saikuro)
• Code smells (Reek)
• Design issues (Roodi)
• File change frequencies (Churn)

When it comes to metrics involving our tests we have:

• Code coverage (Rcov)
• Tools to help identify missed edge cases (Heckle).
• Random Testing tools (RushCheck)

Is that it? I think we can do better than that!

What useful metrics are we missing that our tests could provide and what should we be recording?

Recording Test Builds

Your using a Continuous Integration server right? Running all your tests at every checkin in your source control repository. The CI environment represents our pipeline in which all code needs to flow through. It tends to be the place where all of the tests are run before the code flows into the outside world. Hence this is a perfect environment to start capturing detailed metrics about all of our tests. It’s also not the end of the world if we add a little extra time to the test build in-order to capture these metrics.

Mining Metrics from Test Builds

What interesting things can we discover? Here are some suggestions:

• Failure rates
  • Areas of your product which are prone to failure/bugs and tests which might be fragile. Perhaps highlighting area QA’s should focus extra attention to.
• Flickering tests
  • If a test keeps failing and passing frequently.
• Fragile Tests
  • An all or nothing feature where all the tests fail or none fail.
• Never failing tests
  • Tests which have never failed, do we need to run them all the time, are they now redundant?
• Average build failures a day
  • How often is the build broken.
• Discover Shotgun Surgery
  
  • Small code changes broke all the tests!

What other metrics do you think would be useful?

Kent Beck is Smart

Kent Beck has some additional ideas, lets copy him and pretend to look smart.

Intelligent Selection of the Tests to Run

Kent Beck wrote a tool called JUnit Max which is a plugin for Eclipse and JUnit which helps programmers stay focused on coding by running tests intelligently.

“Max fails fast, running the tests most likely to fail first.“

One of the key principles behind this tool is that:

“Tests that failed recently are more likely to fail than tests which have never failed.”

Super Fast Feedback

If we prioritise the tests that failed recently and those which have been recorded as being likely to fail we increase the chance that a failure occurs early on in the test build. The closer the distance between pushing the code and knowing there is a fail the better.

One problem this helps alleviate is when a test fails 99% of the way through the build. To know you’re fix worked you have to sit and wait for the entire build to run.

CukeMax (alpha-1)

CukeMax is a project that aims to:

• Provide a web service to record Cucumber test builds
• Provide a web based interface to uncover juicy metrics about your tests.
• Feed recorded metrics back into the running of tests prioritising those most likely to fail.
• Cool stuff

CukeMax is intended to be used when you run your tests on your CI server. While this initial version just supports Cucumber there is no reason why it cannot be expanded to other test tools such as Rspec. I’m already using this for my own projects and I have a special version working at Songkick.com HQ.

Wanna Play?

You can browser around an example of the web interface at CukeMax - www.cukemax.com

Want to be one of the first Guinea pigs to try out CukeMax? Let me know.

The client tool will be leaked slowly into the world to ensure we can balance server load.

Whats next?

All I can say is there is a lot of activity around this project with some exciting tools in the pipeline

Also Matt Wynne has been working on some similar ideas and we are discussing if we can combine our thoughts.

At Songkick.com we have developed a number of patterns to make it easier to write Cucumber features. I thought I would start sharing some of those patterns here. So here is the first one:

Implicit Reference Pattern

Make use of implicit references to previously discussed topics to produce scenarios which are easier to read and write. Achieving this while avoiding highly coupled steps.

Problem

Storing and relying on state in a step definition can make it hard to reuse. So often state is avoided. This can lead to scenarios like the following:

Given there is a Artist named "XXs"
 And I visit the page for the Artist named "The XXs"

This leaves us with:

• Verbose steps which are not natural to read.
• Extra noise information purely for identity (referencing the name)

Solution

Map implicit references in the language to the objects being discussed.

Accept that we have to store state but do so in an encapsulated way where the feature language is the only thing needed within the step definition to provide a direct mapping to the relevant object from the state.

What we are aiming for is a scenario like this:

Given there is an Artist named "XXs"
 When I visit the page for the Artist

Requirements

This pattern is described in the context of ActiveRecord and Factory Girl.

Identification through a single meaningful name

Key to this pattern is mapping a single identifier (that we would happily talk about in our Features) to a model.

We have a method that provides all the mappings of the models to the field that uniquely identifies them.

def map_to_id_attribute(type_of_thing)
  {
    :user => :username,
    :concert => :title,
  }[type_of_thing] || :name
end

Domain model class names are meaningful to everyone

In our features when we talk about something in our domain we refer to its class name and we use the correct capitalization.

Non-technical people still understand what the references mean while allowing us to simplify identifying the model in a snippet of feature text.

This leads to steps such as:

Given the Artist
 Given the AdminUser

Implementation

Somewhere to store stuff

We store all the state in a specialised hash. This has a special find_things method which performs some validation and gives us nice error messages if we try and access incorrect types or non-existent objects.

class StuffContainer < Hash
  def find_thing(opts)
    expected_type = opts[:type]
    name = opts[:name]
    thing = self[name]

    raise("Unable to find any object in stuff[] with the name '#{name}' that you asked for, boss. I could however offer you one of the following: #{self.to_s}") unless thing

    raise("That thing you asked for, it appears to be a #{thing.class.name} when you asked for a #{expected_type.name}") unless thing.is_a?(expected_type)

    thing
  end

  def to_s
    result = ["#{self.length} items in total:"]
    self.each do |key, thing|
      result << "the #{thing.class.name} \"#{key}\""
    end

    result.join("\n")
  end
end

Recording the subjects under discussion

In order to record references to created Models we extend Factory Girl’s ‘Factory’ method which is used for all our model creations.

def Factory(type_of_thing, attributes = {})
  test_id = attributes[map_to_id_attribute(type_of_thing)]
  new_thing = super(type_of_thing, attributes)
  stuff[test_id] = new_thing if test_id
  new_thing
end

We will record created models in steps like these:

Given /^there is (?:one|an|a|another) ([^ ]+) named "([^"]+)"$/ do |entity_type, name|
  attributes = {:name => name}
  entity = Factory(entity_type.underscore.to_sym, attributes)
end

Resolving Implicit references

Starting with the step definition:

When /I (?:view|visit|go to) the page for (#{IDRE})$/ do |entity|
  visit model_path(identified_model(entity))
end 

IDRE represents the ID regexp which provides a way of identifying a model and is reused in many steps.

IDRE = /(?:the(?: first | last | )(?:[^ ]+)|the (?:[^ ]+) "(?:[^"]+)"|"(?:[^"]+)")/

The identified_model method turns an English string into a Model. It provides a number of ways of referencing a model.

the Artist
 the first Artist
 the last Artist
 the Artist "Jude"

The identified_model method:
(The key case we are focusing on in this example is where we have no identify just the type of the model – line 11 and 12)

def identified_model(str)
  case str
  when /^the (first|last) ([^ ]+)$/
    klass = safe_constantize($2)
    return klass.__send__($1.to_sym)
  when /^the ([^ ]+) "(.+)"$/
    klass = safe_constantize($1)
    instance = stuff.find_thing(:type => klass, :name => $2)
    instance.reload
    return instance
  when /^the ([^ ]+)$/
    return implicit_model($1)
  when /^"(.+)"$/
    instance = stuff[$1]
    instance.reload if instance
    return instance
  end
  raise "No such instance: '#{str}'.\n Current stuff: #{stuff.to_s}"
end

The implicit_model method

def implicit_model(str)
   klass = safe_constantize(str)
   raise "expected only one #{klass.name}" if klass.count > 1
   raise "expected one #{klass.name} to exist" if klass.count == 0
   klass.first
end

Notice to avoid ambiguity we restrict that only one model of the specified type must exist.

Full source

module StuffManagment
  def map_to_id_attribute(type_of_thing)
    {
      :user   => :username,
      :concert => :title,
    }[type_of_thing] || :name
  end

  class StuffContainer < Hash
    def find_thing(opts)
      expected_type = opts[:type]
      name = opts[:name]
      thing = self[name]
      raise("Unable to find any object in stuff[] with the name '#{name}' that you asked for, boss. I could however offer you one of the following: #{self.to_s}") unless thing
      raise("That thing you asked for, it appears to be a #{thing.class.name} when you asked for a #{expected_type.name}") unless thing.is_a?(expected_type)
      thing
    end

    def to_s
      result = ["#{self.length} items in total:"]
      self.each do |key, thing|
        result << "the #{thing.class.name} \"#{key}\""
      end
      result.join("\n")
    end
  end

  def clear_stuff
    @stuff = StuffContainer.new
  end

  def stuff
    return @stuff if @stuff
    clear_stuff
  end

  SEARCH_MODULES = ['']

  def identified_model(str)
    case str
    when /^the (first|last) ([^ ]+)$/
      klass = safe_constantize($2)
      return klass.__send__($1.to_sym)
    when /^the ([^ ]+) "(.+)"$/
      klass = safe_constantize($1)
      instance = stuff.find_thing(:type => klass, :name => $2)
      instance.reload
      return instance
    when /^the ([^ ]+)$/
      return implicit_model($1)
    when /^"(.+)"$/
      instance = stuff[$1]
      instance.reload if instance
      return instance
    end
    raise "No such instance: '#{str}'.\n Current stuff: #{stuff.to_s}"
  end

  def implicit_model(str)
    klass = safe_constantize(str)
    raise "expected only one #{klass.name}" if klass.count > 1
    raise "expected one #{klass.name} to exist" if klass.count == 0
    klass.first
  end

  def safe_constantize(str)
    begin
      recorded_exception = nil
      SEARCH_MODULES.each do |mod|
        begin
          return "#{mod}::#{str}".constantize
        rescue NameError => e
          recorded_exception = e
        end
      end
      error_message = "\"#{str}\" does not appear to be a valid object in the domain. Did you mean \"#{str.classify}\"?"\
                + "\nDetailed error message:\n#{recorded_exception.message}"
      raise NameError.new(error_message)
    end
  end
end

World(StuffManagment)

Here are the slides from my talk at Railsconf 2010 on Rocket Fuelled Cucumbers. This talk looked at a number of issues in scaling Cucumber tests. I’ve also been speaking about this at Frozen Rails and The Scottish Ruby conference.

And heres what people have been saying about the talk.

I recently created an adapter in Cucumber which provides support for writing step definitions in Javascript. So as a Javascript programmer you can test your code with Cucumber without having to write any Ruby.

It does this through TheRubyRacer (http://github.com/cowboyd/therubyracer written by Charles Lowell) which provides Ruby bindings to V8 (http://code.google.com/p/v8/).

Lets look at an example:

The Feature

Feature: Fibonacci
  In order to calculate super fast fibonacci series
  As a Javascriptist
  I want to use Javascript for that

  @fibonacci
  Scenario Outline: Series
    When I ask Javascript to calculate fibonacci up to <n>
    Then it should give me <series>

    Examples:
      | n   | series                                 |
      | 1   | []                                     |
      | 2   | [1, 1]                                 |
      | 3   | [1, 1, 2]                              |
      | 4   | [1, 1, 2, 3]                           |
      | 6   | [1, 1, 2, 3, 5]                        |
      | 9   | [1, 1, 2, 3, 5, 8]                     |
      | 100 | [1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] |

The Step Definitions

Before(['@fibonacci'], function(){
  fibResult = 0;
});

When(/^I ask Javascript to calculate fibonacci up to (\d+)$/, function(n){
  assertEqual(0, fibResult)
  fibResult = fibonacciSeries(n);
});

Then(/^it should give me (\[.*\])$/, function(expectedResult){
  assertEqual(expectedResult, fibResult)
});

The Cucumber Javascript API

The best reference for the Javascript Api is the examples within the Cucumber source:
http://github.com/aslakhellesoy/cucumber/tree/master/examples/javascript/features/

I have tried to make the Javascript Api as close to the Cucumber Ruby Api as possible. However it currently does not support a couple of things the Ruby version is capable of: Calling step definitions from within step definitions with multiline arguments and giving line reporting on step definitions.

Loading your Javascript code into the World

The most important difference to take note of in the Javascript Api compared with the Ruby one is how we load code into the World so it is in scope within the step definitions.

Given the following folder structure:

my_js_project/lib/code_lives_here.js
my_js_project/features/support/env.js
my_js_project/features/my_feature.feature


Within our features/support/env.js setup file we would have:

//Cucumber resolves the files relative to the folder that contains the features folder.
World(['lib/code_lives_here.js'])

The code within code_lives_here.js would be availabe in the step definitions.

Full Javascript Example

You can see the full example in the Cucumber project:

http://github.com/aslakhellesoy/cucumber/tree/master/examples/javascript/features/

Feedback

This Javascript Cucumber adapter represents an experiment to see how well we can use Cucumber through Javascript and V8. I would love to hear ideas and feedback on the Javascript Api.

Testing your tests is kind of crazy. However when writing a library of Cucumber step definitions which will be used in many projects it started to make sense to test my tests.

• The step definitions are the code.
• It helps reduce fear of breaking lots of projects which use the steps.
• The tests/specs show examples of how to use the step definitions.

It is important to note that I’m not imply TDD/BDDing these step definitions. My use-case is adding tests afterwards when it comes time to extract them to a library.

How to test step definitions

Exercise the full step (with Rspec)

The common way of testing complex steps is to extract all the ruby from the step definitions and then just test that. But this way of testing does not exercise the step definitions from the outside, getting as close as possible to how they will be used. It also does not provide examples of how to use the step definitions.

So I sat down with Matt Wynne, who started this discussion and we thrashed out some Rspec macros for testing whole step definitions.

If we were testing this step definition (icalendar_steps.rb):

require 'icalendar'

module Cucumber
  module Stepdefs
    module Icalendar
      def response_calendars
        ::Icalendar.parse(response.body)
      end

      def response_events
        response_calendars.length.should == 1
        response_calendars.first.events
      end
    end
  end
end

Before('@ical') do
  extend Cucumber::Stepdefs::Icalendar
end

Then /^the iCalendar should have exactly (\d+) events?$/ do |number_of_events|
  response_events.length.should == number_of_events.to_i
end

Our spec would look like this (Note: we test the Before hook as well as the step definition):

describe 'icalendar_steps' do
  step_file File.dirname(__FILE__) + '/../../../lib/cucumber/stepdefs/icalendar_steps'

  # Test that the Before hook is not called when there is no tag
  without_tags do
    it "should not mix in any calendar related methods" do
      world_methods.should_not include('response_calendars')
      world_methods.should_not include('response_events')
    end
  end

  # Test the Before hook mixes in the right methods when tagged with @ical
  with_tag '@ical' do
    ['response_calendars', 'response_events'].each do |method|
      it "should add the #{method} to world" do
        world_methods.should include(method)
      end
    end

    the_step "the iCalendar should have exactly 1 event" do
      describe "when 1 calendar with 0 events is in the response body" do
        before(:each) do
          world.stub!(:response).and_return(mock_response_with_0_events)
        end

        it_should_fail_with(Spec::Expectations::ExpectationNotMetError)
      end

      describe "when 1 calendar with 1 event is in the response body" do
        before(:each) do
          world.stub!(:response).and_return(mock_response_with_1_event)
        end

        it_should_pass
      end
    end
  end
end

Experiment’s Source code

You can see the source on Github:

git clone git://github.com/mattwynne/cucumber-step_definitions.git

If this experiment proves successful these macros will make their way to a nice gem.

Combinatorial testing is a difficult problem. Having to test a small number of inputs  can result in a combinatorial explosion of possible permutations. In Cucumber we see this problem in Scenario Outlines where we can have a large number of rows for the Examples table.

We want to reduce the combinations to a more manageable size while still providing effective fault detection.

Pairwise testing provides one method of achieving this. However providing effective fault detection is dependent on the suitability of Pairwise to the data/system, it is not guaranteed.

What is Pairwise testing?

Pairwise testing (also called All-pairs testing or 2-way testing) is a way of generating a test suite which covers all input combinations of two and is therefore much smaller than exhaustive suites.

To put this in perspective a system of 72 binary inputs with Pairwise testing would require 28 combinations. Exhaustive testing would require 37,778,931,862,957,161,709,568 combinations!

Sounds great, but how does the method preserve good defect detection?

Pairwise relies on a simple principle:

“most faults are caused by interactions of at most two factors”
- The Combinatorial Design Approach to Automatic Test Generation

Pairwise focuses on the minimal set of inputs (1 or 2 interactions) that cover the most likely causes of faults.

So whats the catch?

Pairwise does not guarantee coverage of all faults of at most two factors. It only provides coverage of faults reachable from the inputs and values you select when generating the Pairwise set. Also the general principle that faults are caused by at most two factors does not guarantee that your data sets faults will follow that distribution. And hence Pairwise testing might miss a proportion of faults.

Pairwise Cucumber example

Here is a scenario taken from a web based system which deals with sports events.

Scenario Outline: Visiting events with another events media
  Given I have a <event without media>
  And I have a <media item> attached to <event with media>
  When I go to the <media item> page for the <event without media>
  Then I should be redirected
  And I should see the <media item> in the <event with media>
  Examples:
    |media item|event without media|event with an media|
    ...

The permutations for the Examples table cells are:

media item: [Image, Video, Music]
event with media: [Football, Basketball, Soccer]
event without media: [Football, Basketball, Soccer]

There are a total of 27 possible permutations. So lets see what permutations Pairwise testing would suggest.

To generate the combinations I used a Ruby based tool I’ve written called Pairwise which uses the in-parameter-order Pairwise generation strategy (http://ranger.uta.edu/~ylei/paper/ipo-tse.pd).

This tool outputs a table ready to be used in a Cucumber feature:

| media item | event without media | event with media |
 | Image      | Football            | Football         |
 | Image      | Basketball          | Basketball       |
 | Image      | Soccer              | Soccer           |
 | Video      | Football            | Soccer           |
 | Video      | Basketball          | Football         |
 | Video      | Soccer              | Basketball       |
 | Music      | Football            | Basketball       |
 | Music      | Basketball          | Soccer           |
 | Music      | Soccer              | Football         |

That’s 11 permutations covering all possible input pairs.

Giving the final Scenario:

Scenario Outline: Visiting events with another events media
  Given I have a <event without media>
  And I have a <media item> attached to <event with media>
  When I go to the <media item> page for the <event without media>
  Then I should be redirected
  And I should see the <media item> in the <event with media>
Examples:
  | media item | event without media | event with media |
  | Image      | Football            | Football         |
  | Image      | Basketball          | Basketball       |
  | Image      | Soccer              | Soccer           |
  | Video      | Football            | Soccer           |
  | Video      | Basketball          | Football         |
  | Video      | Soccer              | Basketball       |
  | Music      | Football            | Basketball       |
  | Music      | Basketball          | Soccer           |
  | Music      | Soccer              | Football         |

Selecting the right values to test with

In Cucumber we encounter plain text steps which use high level language to hide low level details that are not important for the scenario. For example:

Given a default configuration

By abstracting and pushing the data out of the plain text we are also hiding the values from any Pairwise generation. It is therefore important to realise there will be sets of uncovered pairs and hence potential faults missed.

In deciding what is exposed in the Scenario Outline (as columns in the Examples table) you are selecting what inputs your want to focus the Pairwise generation around. And in turn the input space you want to limit the search for faults to.

When Pairwise testing fails

Pairwise testing is not suitable for all data sets. With any technique that throws away data examples based on a pattern you loose some degree of resolution and you may misfit your data missing important faults.

Here are some cases where Pairwise testing may not be effective.

You have probable combinations which you want to focus on.

If you have certain combinations of inputs that have far more commonality or importance you would want to focus your testing data around these. Pairwise testing ignores any probable importance.

You don’t know how the input variables interact

Pairwise testing assumes each input value carries the same significance on the outcome. If certain inputs have greater influence over the outcome you would want to focus your test data around these.

Your data requires higher order test combinations

Not all data sets follow the distribution of most faults being caused by at most 2 factors. You may have a larger number of faults outside the 2 parameter range. Using higher order test data combinations (Such as 3-wise, 4-wise etc) or completely different test methods may be required to reduce the risk of missing faults.

The real difficulty here is knowing beforehand the distribution of faults.

You want a best practice

There are no testing “best practices” that you can simply “follow” in order to achieve success.

To quote James Marcus Bach:

“blindly applying Pairwise testing to combinatorial testing problems may increase the risk of delivering faulty software.”
- http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf

Your faults mean people are going to die

In realtime or safety critical systems any faults irrelevant of the number of interacting parameters wants to be detected.

Final words on Pairwise

Shortcuts to Combinatorial problems in testing carry their pitfalls. No matter how you reduce the input permutations you should spend time understanding your combinatorial data set. This is especially the case with Pairwise testing which does not consider the relationships the inputs have on the outputs. Its up to you to examine any interactions or strong influences in order to evaluate Pairwise’s suitability. If Pairwise testing is used thoughtfully with due considerations to its limitations and pitfalls it can provide a powerful tool to testers and Cucumberists alike.

Pairwise tools

• Ruby based tool which supports usage through Ruby or on the command line:
  http://wiki.github.com/josephwilk/pairwise
• A free web-based tool from Testers Desk that provides generation of Pairwise data sets:
  http://www.testersdesk.com/TestersDesk/

Further Reading

• Pairwise – http://www.pairwise.org/
• Combinatorial Testing – http://www.combinatorialtesting.com/
• Pairwise Testing: A Best Practice That Isn’t – http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
• A Test Generation Strategy for Pairwise Testing – http://ranger.uta.edu/~ylei/paper/ipo-tse.pdf

There has been lots of activity around the value of metrics for source code and tests. In the Ruby world tools like metric_fu provide a wealth of analysis.

While working on my Cucumber talk for Rails Underground I started investigating how we could apply metrics to the customer focused plain text of Cucumber. For those not familiar with Cucumber it’s an acceptance testing framework which allows non-technical people to write plain-text describing the behaviors of their system. The developers/testers map the plain-text to tests.

Having spent time teaching people about the plain-text side of Cucumber I often found myself recommending the same guidelines and plain-text anti-patterns. This lead me to think about providing metrics scoring the customers plain-text.

Why would we want plain-text acceptance test metrics?

• Help plain-text beginners avoid bad practices early on.
• Help improve the quality of plain-text
• Help quality review with a large frequency of incoming features

Why does the quality of the plain-text matter?

Why focus on quality, the plain-texts primarily goal is to be easy for customer to use?

• The developer builds the Domain specific language via mapping plain text to ruby. Higher quality plain-text could make it easier to manage these mappings without any major impact to readability.
• Higher quality text is easier to read, edit and understand.

Who would find it useful?

Initially Developers.

• In some scenarios the developers write the features from discussions and give to the customer to review.
• Developers may tweak/review customer written changes/features.
• Developers often edit/tweak plain-text from the customer to enable reuse of existing test code .
• In open source projects often developers write Cucumber features. Metrics are something they are comfortable with.

Can you measure quality in plain-text?

First its important to distinguish acceptance tests from pure plain-text. Within acceptance tests we have some degree of structure, for example using Given/When/Then to describe scenarios.

Cucumber Example:

Scenario: Eating all cucumbers
  Given there are 5 cucumbers
  When I eat 5 cucumbers
  Then I should have 0 cucumbers

This structure reduces the complexity of analysing the quality of the text. It provides us with different structural elements which have different rules/guidelines on what their content should be.

The problem with measuring the quality of text is that it is far more subjective in than in code. So while we cannot be absolute in our assessment of quality we can try and codify smells that *could* indicate areas in the text that *could* be improved. This is pretty much true for all metrics, they are guidelines not absolutes (Dan Norths highlights the dangers of absolute metrics in the Parable of Metrics)

So what useful metrics could we look at?

Plain text Metrics

From my experience with Cucumber I would suggest examining:

• Lack of Feature/Story Narrative
  • We don’t know why this feature is useful and who it’s useful to!
• Scenario length
  • Very long scenarios may reflect more abstraction should be used to help make the scenarios easier to read (See declarative vs imperative scenarios: http://www.benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories/)
• Size of Background (like  a before/setup)
  • Increases conceptual overhead for each scenario. More reasons here: http://wiki.github.com/aslakhellesoy/cucumber/background
• Semantically close words across features
  • Can indicate ambiguity around domain terminology
• Frequency of noise words (‘basically’, ‘be able to’, etc)
  • Adds no value and drives me crazy.

Feedback

What do you think of the idea?

Can you think of any other useful plain-text metrics?

I have written a plugin for the continuous integration server Hudson which uses a metric-fu rake task at its core to build and present graphs representing different metrics over successful builds.
It currently supports:

• Flog (Complexity)
• Flay (Duplication)
• Rcov (Code coverage)

The source is available on Github:

http://github.com/josephwilk/rubymetricfu

Installing

Currently all Hudson’s plugins are stored in something called SVN. Being more of a GIT myself you have to manually install the plugin rather than using the automatic Hudon GUI install method.

Install steps:

1. Ensure you have the Ruby and Rake Hudson plugins installed.
2. Follow the metric-fu installation guide (http://metric-fu.rubyforge.org/)
3. Ensure project code has a metrics:all rake task (auto added when you require metric-fu)
4. Download the rubymetricfu.hpi plugin file
5. Copy the file into your plugins folder within you Hudson install. Hudson’s default is ~/.hudson/plugins
6. Restart hudson
7. Go to the ‘configure’ link for a project and select the Ruby Metric-fu report option (see below)
8. (Optionally) Pick which Rake version you want to use.

Future

• Futher metrics to graph:
  • Roodi (design issues)
  • Reek (Code smells)
  • Saikuro (cyclomatic complexity)
• Configure which metrics you want on your project page.
• Create a Crap4R meter using Rcov and Flog (Similar to Crap4J).
• Better integration with the html reports generated by metric-fu.

“Cucumber needs you to experiment with your favourite Java Virtual Machine based language and connect to Cucumber via JRuby.”

What’s this Cucumber you speak of? Checkout: http://cukes.info/

Wait that’s Ruby, how do I use a JVM based language to play with it?

Cuke4Duke (http://wiki.github.com/aslakhellesoy/cuke4duke) allows writing Cucumber step definitions in Java. This means Java developers can use the Cucumber tool without having to write any Ruby.

Ruby step definitions

Given /I have (\d+) cukes in my belly/ do |n|
  @belly ||= []
  n.to_i.times {|i| @belly << "cuke"}
end

Equivalent Java step definitions

package cukes;

import cuke4duke.Given;
import cuke4duke.Steps;
import java.util.List;
import java.util.ArrayList;

@Steps
public class BellySteps {
    private List<String> belly = new ArrayList<String>();

    @Given("I have (\\d+) cukes in my belly")
    public void bellyCukes(int cukes) {
        for(int i = 0; i < cukes; i++) {
            belly.add("cukes");
        }
    }
}

This works by connecting Java to Cucumber via JRuby (http://jruby.codehaus.org/).

So in theory if your language runs on the JVM you can use JRuby, and hence use Cucumber and its wonderful Gherkin language.

So what are you waiting for! Pick up your favourite JVM language and arm yourself with Cucumber!

Some Example JVM languages:

• Groovy
• Clojure
• Scala
• Ioke

Good luck and safe Cuking

FutureRuby was an exceptional conference and I was excited to be a part of such a creative group of people. I talked about Cucumber, looking at what it is, how to use it, and why to use it. Useful links for Cucumber:

• Wiki http://wiki.github.com/aslakhellesoy/cucumber
• Source code http://github.com/aslakhellesoy/cucumber.

I demonstrated using Cucumber to test a simple IPhone application. To make the IPhone testing a little more palatable I used a little gem called IRobat. Its very rough around the edges and in no way complete but you can take a look at the IRobat code on Github.

No Cucumbers were harmed (just mildly shaken up) in the making of this presentation.

What people were saying about the talk: