Joseph Wilk

Joseph Wilk

Things with code, creativity and computation.

Cucumber, Tags and Continuous Integration Oh My!

We want to be able to commit our code frequently to prevent merge headaches.

“the longer you wait, the more your code will diverge from your teammates. If you don’t commit often you rob them of the opportunity to reduce merge hell.” Aslak Hellesøy

When dealing with Cucumber and Features/Scenarios we may find we want to commit part way through a scenario but we won’t because:

  1. We don’t want to break the build
  2. We don’t want to pollute the build with lots of pending steps.

A common solution to this problem is to create two streams for running the features:

  1. In-progress

    • If an in-progress scenario fails then the build carries on.
    • If an in-progress scenario passes then the build fails (This is very similar to how Rspec works with pending)

  2. Finished

    • If a completed scenario fails it causes the build to fail.

We can implement this model using Cucumber’s new Tag feature. We can tag Scenarios and Features with @in-progress and use this tag to help exclude in-progress Features/Scenarios from the finished build.

@in-progress
Feature:
  In order to avoid merge headaches
  As a developer
  I want to tag my features and scenarios with a in-progress tag

  @in-progress
  Scenario: I'm not finished yet
    Given ...
    When ...
    Then ...

The Rake tasks

Finished features/scenarios task

We prefix tags with ~ to exclude features or scenarios having that tag

1
2
3
4

  desc "Run finished features"
  Cucumber::Rake::Task.new(:finished) do |t|
    t.cucumber_opts = "--format progress --tags ~in-progress"
  end

In-progress features/scenarios task

1
2
3
4

  desc "Run in-progress features"
  Cucumber::Rake::Task.new(:in_progress) do |t|
    t.cucumber_opts = "--require formatters/ --format Cucumber::Formatter::InProgress --tags in-progress"
  end

We require a special formatter Cucumber::Formatter::InProgress which is essential for making the task work. This formatter as well as giving helpful output changes the command line exit codes of Cucumber. This is kind of crazy but only within the formatter do we have enough information to decided if we should fail or pass. The formatter only returns a failure exit code if there were any scenarios which passed. So unlike the default exit codes failing steps will not cause a failure exit code.

Full Source Code

Also available at: http://github.com/josephwilk/cucumber_cocktails/tree/master

Rake Task

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

require 'cucumber/rake/task'

class BuildFailure < Exception;
  def initialize(message = nil)
    message ||= "Build failed"
    super(message)
  end
end;

Cucumber::Rake::Task.new do |t|
  t.cucumber_opts = "--format progress"
end

namespace :features do
  desc "Run finished features"
  Cucumber::Rake::Task.new(:finished) do |t|
    t.cucumber_opts = "--format progress --tags ~in-progress"
  end

  desc "Run in-progress features"
  Cucumber::Rake::Task.new(:in_progress) do |t|
    t.cucumber_opts = "--require formatters/ --format Cucumber::Formatter::InProgress --tags in-progress"
  end
end

desc "Run complete feature build"
task :cruise do
  finished_successful = run_and_check_for_exception("finished")
  in_progress_successful = run_and_check_for_exception("in_progress")

  unless finished_successful && in_progress_successful
    puts
    puts("Finished features had failing steps") unless finished_successful
    puts("In-progress Scenario/s passed when they should fail or be pending") unless in_progress_successful
    puts
    raise BuildFailure
  end
end

def run_and_check_for_exception(task_name)
  puts "*** Running #{task_name} features ***"
  begin
    Rake::Task["features:#{task_name}"].invoke
  rescue Exception => e
    return false
  end
  true
end

InProgress Formatter

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

module Cucumber
  module Formatter
    class InProgress < Progress
      FAILURE_CODE = 1
      SUCCESS_CODE = 0

      FORMATS[:invalid_pass] = Proc.new{ |string| ::Term::ANSIColor.blue(string) }

      def initialize(step_mother, io, options)
        super(step_mother, io, options)
        @scenario_passed = true
        @passing_scenarios = []
        @feature_element_count = 0
      end

      def visit_feature_element(feature_element)
        super

        @passing_scenarios << feature_element if @scenario_passed
        @scenario_passed = true
        @feature_element_count += 1

        @io.flush
      end

      def visit_exception(exception, status)
        @scenario_passed = false
        super
      end

      private

      def print_summary
        unless @passing_scenarios.empty?
          @io.puts format_string("(::) Scenarios passing which should be failing or pending (::)", :invalid_pass)
          @io.puts
          @passing_scenarios.each do |element|
            @io.puts(format_string(element.backtrace_line, :invalid_pass))
          end
          @io.puts
        end
        print_counts

        unless @passing_scenarios.empty?
          override_exit_code(FAILURE_CODE)
        else
          override_exit_code(SUCCESS_CODE)
        end
      end

      def override_exit_code(status_code)
        at_exit do
          Kernel.exit(status_code)
        end
      end

    end
  end
end

Comments