Joseph Wilk

Failure Rates vs Change Rates

I’ve been spending a lot of  time recently data mining our test suite at Songkick.com. I’ve been using our own internal version of www.limited-red.com which records all our test fails for every build on our continuous integration server.

With the Cucumber features I decided to plot the number of times a feature file has been significantly changed vs the number of times the feature has failed in the build. The theory being that a change to a feature file(the plain text specification of behaviour) most often represents a chance in functionality.

So I wrote a simple script checking the number of commits to each feature file.

features_folder = ARGV[0]
feature_files = Dir["#{features_folder}/**/*.feature"]

feature_files.each do |feature_file|
  change_logs = `git log --oneline #{feature_file} 2>/dev/null`
  change_count = change_logs.split("\n")
  puts "#{feature_file},#{change_count}"
end

Feature changes compared with feature failures

Throwing that into a pretty graph, here is a snapshot of some of the features (I’ve changed the name of the features to hide top secret Songkick business).

Insights

Based on this I identified two possible groups of features:

• Features failing more often than the code around them is changing
• Features which are robust and are not breaking when the code around them is changing.

Further investigation into the suspect features highlighted 3 causes:

• Brittle step definitions
• Brittle features coupled to the UI
• Tests with a high dependency on asynchronous behaviour

Holes in the data – step definitions

We only recorded the change rate of features files in Git. Features could be broken without ever changing the feature file, for example if a common step definition is broken. Next steps are to identify all the step definitions used by a feature and examine how often the step definitions have changed.

First find the change count for all the step definitions.

step_files = Dir["features/**/*_steps.rb"]

step_files.each do |step_file|
  change_logs = `git log --oneline #{step_file} 2>/dev/null`
  change_count = change_logs.split("\n").count
  puts "#{step_file},#{change_count}"
end

Then working out what step definitions a feature uses. We can do this by running cucumber with the json formatter and match up step definitions (and hence step definition files) to feature files:

require 'json'

`cucumber features --dry-run --format json --out .json.out`
features_json = JSON.parse(File.read('.json.out'))

stats = Hash.new{|h,k| h[k] = []}
features_json['features'].each do |feature|
  feature_name = feature['name']
  #The JSON does not have the feature file. Find the file via the feature name. Messy
  feature_file = `egrep -riE "feature:? *#{feature_name}" features/`.split(":")[0]
  feature["elements"].each do |element|
    element["steps"].each do |step|
      file_location = step['match']['location']
      file, _ = file_location.split(":")
      if file =~ /_steps\.rb$/
        stats[feature_file] = (stats[feature_file] + [file]).uniq
      end
    end
  end
end
pp stats

Change rate vs Failure rate with step definition changes

Combining those two bits of data we can now add to our original graph the step definition change rates for a feature.

We also can examine an individual break down of the step definition change rates for a feature:

Holes in using the step definition change rate

The step definition changes from git are at the file level (the *_step.rb file) so a change in git may not touch a step definition used by a feature. Hence we may be counting changes which are not relevant for a feature. Further work would be to examine the git diffs and check if a change touched a step definition used by a feature.

Conclusions

Our tests hide lots of interesting information that can provide evidence of areas we can make improvements. It’s important to realise that like anything in statistics our data mining does not yield facts, just suggestions. At Songkick we are already mining this information with Cucumber and using it to help improve and learn about our tests.

What Is the Page Object Pattern?

The Page model is a pattern that maps a UI page to a class, where for example a page could be a HTML page. The functionality to interact or make assertions about that that page is captured within the Page class. Then these methods may be called by a test. So ultimately we are introducing a gatekeeper to the GUI of a page.

Why use the Page Object Pattern?

• Readable dsl for tests
• Promotes Reuse
• Centralise UI coupling – One place to make changes around the UI.

Implementing the Page Pattern in Cucumber

Within Cucumber there are two main ways we can encapsulate the page UI:

The Page Object Pattern

features/pages/login_page.rb

class LoginPage
  def login(user, password)
    fill_in :user, user
    fill_in :password password
    click 'login'
  end

  def visit
    visit "/login"
  end
end 

features/step_definitions/user_steps.rb

Given /^I login with username "Joseph" and password "cuker"$/ do |username, password|
  login_page = LoginPage.new

  login_page.visit
  login_page.login(username, password)
end 

The Page Step definition Pattern

Cucumber step definitions are all defined at the same scope, but we use folders and files to create logical organisation. We can create folders for UI step definitions and domain step definitions.

• features/domain/step_definitions/*
• features/ui/step_definitions/*

We create a step definition file mapping to a UI page.
features/ui/step_definitions/login_page_steps.rb

Given /^I login with username "Joseph" and password "cuker"$/ do |username, password|
  fill_in :user, user
  fill_in :password password
  click 'login'
end

Whats the right way to encapsulate the UI?

Just using step definitions for organisation within a project can have a number of problems:

Global scope within Cucumbers world
Instance variables are global across all step definitions

Given /^I mess with scope$/ do
  @this_can_be_seen_by_every_other_step = 'uh oh'
end

Managed and run through Cucumber
No easy way to be reused outside of Cucumber or test in isolation. By isolating the test code we can easily provide adapters for reuse in different test frameworks (for example similar to what email-spec does).

The Page Object pattern (and adding another layer of abstraction) has a couple of nice properties:

• Bounded scope (if you use your classes/objects nicely)
• Isolated units that can be invoked and controlled independently of overarching testing framework

Should I be using the Page Object Pattern?

Yes, No, Maybe.

Extra layers of abstraction introduce complexity and so the Page Object Pattern should be used carefully when there is a sufficiently high burden of maintenance (which usually means lots of step definitions).

Its important outside of the Page object pattern to realise the weaknesses of just using step definitions as your only modelling tool. Irrelevant of what metaphor you decide to organise around it’s a good habit to push the code out of the step definitions.

I was recently interviewed about Cucumber and whether it solves the problem of getting business people writing specifications.
You can read the article on SDTimes:
Cucumber puts plain English on requirements

I got a little bit carried away in answering some of the questions posed and Alex Handy was kind enough to post all my responses in detail:
Peeling Cucumber

A demonstration of working with Limited Red (previously known as CukeMax) on a Cucumber project.

Using Limited Red with a cucumber project from Joseph Wilk

At Songkick.com we have developed a number of patterns to make it easier to write Cucumber features. I thought I would start sharing some of those patterns here. So here is the first one:

Implicit Reference Pattern

Make use of implicit references to previously discussed topics to produce scenarios which are easier to read and write. Achieving this while avoiding highly coupled steps.

Problem

Storing and relying on state in a step definition can make it hard to reuse. So often state is avoided. This can lead to scenarios like the following:

Given there is a Artist named "XXs"
 And I visit the page for the Artist named "The XXs"

This leaves us with:

• Verbose steps which are not natural to read.
• Extra noise information purely for identity (referencing the name)

Solution

Map implicit references in the language to the objects being discussed.

Accept that we have to store state but do so in an encapsulated way where the feature language is the only thing needed within the step definition to provide a direct mapping to the relevant object from the state.

What we are aiming for is a scenario like this:

Given there is an Artist named "XXs"
 When I visit the page for the Artist

Requirements

This pattern is described in the context of ActiveRecord and Factory Girl.

Identification through a single meaningful name

Key to this pattern is mapping a single identifier (that we would happily talk about in our Features) to a model.

We have a method that provides all the mappings of the models to the field that uniquely identifies them.

def map_to_id_attribute(type_of_thing)
  {
    :user => :username,
    :concert => :title,
  }[type_of_thing] || :name
end

Domain model class names are meaningful to everyone

In our features when we talk about something in our domain we refer to its class name and we use the correct capitalization.

Non-technical people still understand what the references mean while allowing us to simplify identifying the model in a snippet of feature text.

This leads to steps such as:

Given the Artist
 Given the AdminUser

Implementation

Somewhere to store stuff

We store all the state in a specialised hash. This has a special find_things method which performs some validation and gives us nice error messages if we try and access incorrect types or non-existent objects.

class StuffContainer < Hash
  def find_thing(opts)
    expected_type = opts[:type]
    name = opts[:name]
    thing = self[name]

    raise("Unable to find any object in stuff[] with the name '#{name}' that you asked for, boss. I could however offer you one of the following: #{self.to_s}") unless thing

    raise("That thing you asked for, it appears to be a #{thing.class.name} when you asked for a #{expected_type.name}") unless thing.is_a?(expected_type)

    thing
  end

  def to_s
    result = ["#{self.length} items in total:"]
    self.each do |key, thing|
      result << "the #{thing.class.name} \"#{key}\""
    end

    result.join("\n")
  end
end

Recording the subjects under discussion

In order to record references to created Models we extend Factory Girl’s ‘Factory’ method which is used for all our model creations.

def Factory(type_of_thing, attributes = {})
  test_id = attributes[map_to_id_attribute(type_of_thing)]
  new_thing = super(type_of_thing, attributes)
  stuff[test_id] = new_thing if test_id
  new_thing
end

We will record created models in steps like these:

Given /^there is (?:one|an|a|another) ([^ ]+) named "([^"]+)"$/ do |entity_type, name|
  attributes = {:name => name}
  entity = Factory(entity_type.underscore.to_sym, attributes)
end

Resolving Implicit references

Starting with the step definition:

When /I (?:view|visit|go to) the page for (#{IDRE})$/ do |entity|
  visit model_path(identified_model(entity))
end 

IDRE represents the ID regexp which provides a way of identifying a model and is reused in many steps.

IDRE = /(?:the(?: first | last | )(?:[^ ]+)|the (?:[^ ]+) "(?:[^"]+)"|"(?:[^"]+)")/

The identified_model method turns an English string into a Model. It provides a number of ways of referencing a model.

the Artist
 the first Artist
 the last Artist
 the Artist "Jude"

The identified_model method:
(The key case we are focusing on in this example is where we have no identify just the type of the model – line 11 and 12)

def identified_model(str)
  case str
  when /^the (first|last) ([^ ]+)$/
    klass = safe_constantize($2)
    return klass.__send__($1.to_sym)
  when /^the ([^ ]+) "(.+)"$/
    klass = safe_constantize($1)
    instance = stuff.find_thing(:type => klass, :name => $2)
    instance.reload
    return instance
  when /^the ([^ ]+)$/
    return implicit_model($1)
  when /^"(.+)"$/
    instance = stuff[$1]
    instance.reload if instance
    return instance
  end
  raise "No such instance: '#{str}'.\n Current stuff: #{stuff.to_s}"
end

The implicit_model method

def implicit_model(str)
   klass = safe_constantize(str)
   raise "expected only one #{klass.name}" if klass.count > 1
   raise "expected one #{klass.name} to exist" if klass.count == 0
   klass.first
end

Notice to avoid ambiguity we restrict that only one model of the specified type must exist.

Full source

module StuffManagment
  def map_to_id_attribute(type_of_thing)
    {
      :user   => :username,
      :concert => :title,
    }[type_of_thing] || :name
  end

  class StuffContainer < Hash
    def find_thing(opts)
      expected_type = opts[:type]
      name = opts[:name]
      thing = self[name]
      raise("Unable to find any object in stuff[] with the name '#{name}' that you asked for, boss. I could however offer you one of the following: #{self.to_s}") unless thing
      raise("That thing you asked for, it appears to be a #{thing.class.name} when you asked for a #{expected_type.name}") unless thing.is_a?(expected_type)
      thing
    end

    def to_s
      result = ["#{self.length} items in total:"]
      self.each do |key, thing|
        result << "the #{thing.class.name} \"#{key}\""
      end
      result.join("\n")
    end
  end

  def clear_stuff
    @stuff = StuffContainer.new
  end

  def stuff
    return @stuff if @stuff
    clear_stuff
  end

  SEARCH_MODULES = ['']

  def identified_model(str)
    case str
    when /^the (first|last) ([^ ]+)$/
      klass = safe_constantize($2)
      return klass.__send__($1.to_sym)
    when /^the ([^ ]+) "(.+)"$/
      klass = safe_constantize($1)
      instance = stuff.find_thing(:type => klass, :name => $2)
      instance.reload
      return instance
    when /^the ([^ ]+)$/
      return implicit_model($1)
    when /^"(.+)"$/
      instance = stuff[$1]
      instance.reload if instance
      return instance
    end
    raise "No such instance: '#{str}'.\n Current stuff: #{stuff.to_s}"
  end

  def implicit_model(str)
    klass = safe_constantize(str)
    raise "expected only one #{klass.name}" if klass.count > 1
    raise "expected one #{klass.name} to exist" if klass.count == 0
    klass.first
  end

  def safe_constantize(str)
    begin
      recorded_exception = nil
      SEARCH_MODULES.each do |mod|
        begin
          return "#{mod}::#{str}".constantize
        rescue NameError => e
          recorded_exception = e
        end
      end
      error_message = "\"#{str}\" does not appear to be a valid object in the domain. Did you mean \"#{str.classify}\"?"\
                + "\nDetailed error message:\n#{recorded_exception.message}"
      raise NameError.new(error_message)
    end
  end
end

World(StuffManagment)