Joseph Wilk

Joseph Wilk

Things with code, creativity and computation.

Speaking at Scotland on Rails 2009

I’m really excited to be giving a talk at this years Scotland on Rails conference in Edinburgh.

I’ll be talking about working outside-in with Cucumber and RSpec. Having used Cucumber and as a member of the Cucumber core developer team I hope to share lots of experiences and lessons about getting the most out of the tool.

It’s looking like a great line up with with some really interesting presentations across a broad number of topics. The keynotes speakers are Michael Feathers and Marcel Molina, Jnr.

If you’re going to be in Edinburgh for the conference or have any burning questions about Cucumber, let me know.

Latent Semantic Analysis in Ruby

I’ve had lots of requests for a Ruby version to follow up my Latent Semantic Analysis in Python article. So I’ve rewritten the code and article for Ruby. I wrote LSA from scratch this time and test driven so it has some subtle differences from the Python version.

What is LSA?

Latent Semantic Analysis (LSA) is a mathematical method that tries to bring out latent relationships within a collection of documents. Rather than looking at each document isolated from the others it looks at all the documents as a whole and the terms within them to identify relationships.

An example of LSA: Using a search engine search for “ruby”.

Documents are returned which do not contain the search term “ruby” but contains terms like “rails”.

LSA has identified a latent relationship, “ruby” is semantically close to “rails”.

How does it work?

Given a set of word documents, each word in those documents represents a point in the semantic space. LSA uses a mathematical technique called Singular value decomposition to take the documents/words represented as a matrix and produce a reduced approximation of this matrix. In doing this it reduces the overall noise in the semantic space bringing words together. Hence after applying LSA some words share similar points in the semantic space, they are semantically similar.

These groups of semantically similar words form concepts and those concepts in turn relate to documents.

Term a <----------->
Term b <-----------> Concept d <---------> Document e
Term c <----------->

Background Reading on LSA

There are some very good papers which describe LSA in detail:

And on SVD

LSA Algorithm

This is an implementation of LSA in Ruby (v1.8.7). I made use of the linalg project which while not as mature as Python’s SciPy it does the job. Installing linalg can be rather challenging as it relies on LAPACK which in turn relies on some Fortran code.

1 Create the term-document matrix

We take a set of documents and map them to a vector space matrix. Each column represents a document and each row represents a term.

         Document n  Document n+1
term n    [   1           0 ]
term n+1  [   0           1 ]

Read more about the Vector Space Model at Wikipedia.

We use the DMatrix class from Linalg to represent our term-document matrix. This is like the Ruby standard library Matrix class but with some powerful matrix functions builtin.

2 tf-idf Transform

tf => Term frequency

idf => Inverse document frequency.

Apply the tf-idf transform to the term-document matrix. This generally tends to help improve results with LSA.

Github source file: tf_idf_transform.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
      def self.transform(matrix)
        number_of_documents = matrix.num_columns
        @@number_of_documents_with_term = []

        matrix.columns.each_with_index do |document, column_index|
          document_term_total = document.rows.inject(0.0) {|word_sum, word_count| word_sum + word_count.to_f }

          document.rows.each_with_index do |term_weight, row_index|
            unless term_weight.to_f == 0.0
              matrix[row_index, column_index] = (term_weight / document_term_total) *
              Math.log((number_of_documents / number_of_documents_with_term(row_index, matrix).to_f).abs)
            end
          end
        end
        matrix
      end

3 Latent Semantic transform

Github source file: lsa_transform.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
      def transform(matrix, number_of_dimensions_to_reduce = 1)
          columns = matrix.num_columns

          if dimensions <= columns: #Its a valid reduction
            u, sigma, vt = matrix.singular_value_decomposition

            sigma_prime = reduce_dimensions(number_of_dimensions_to_reduce, sigma)

            matrix_prime = u * sigma_prime * vt
          else
            raise Exception, "dimension reduction cannot be greater than %s" % columns
          end

          matrix_prime
        end

The LSA transform consists of 3 stages. Lets take a closer look at these 3 stages:

1. Singular Value Decomposition

SVD: http://en.wikipedia.org/wiki/Singular_value_decomposition Linalg’s DMatrix has a singular_value_decomposition method which saves us a lot of work!

1
 u, sigma, vt = matrix.singular_value_decomposition

2. Reduce the dimensions of Sigma

We generally delete the smallest coefficients in the diagonal matrix Sigma to produce Sigma’.

1
2
3
4
5
6
        def reduce_dimensions(number_of_dimensions_to_reduce, matrix)
          for diagonal_index in dimensions_to_be_reduced(matrix, number_of_dimensions_to_reduce)
            matrix[diagonal_index, diagonal_index] = 0
          end
          matrix
        end

The reduction of the dimensions of Sigma combines some dimensions such that they are on more than one term. The number of coefficients deleted can depend of the corpus used. It should be large enough to fit the real structure in the data, but small enough such that noise or unimportant details are not modelled.

The real difficulty and weakness of LSA is knowing how many dimensions to remove. There is no exact method of finding the right dimensions. Approaches which are used are L2-norm or Frobenius norm.

3. Calculate the Product with New Sigma’

Finally we calculate:

1
 matrix_prime = u * sigma_prime * vt

LSA In Action – rSemantic Gem

First install the rsemantic gem from Github

gem sources -a http://gems.github.com
sudo gem install josephwilk-rsemantic

Then we can run an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
require 'rubygems'
require 'semantic'

D1 = "The cat in the hat disabled"



D2 = "A cat is a fine pet ponies."
D3 = "Dogs and cats make good pets"
D4 = "I haven't got a hat."

#verbose mode so it shows us all the different stages
search = Semantic::Search.new([D1, D2, D3, D4], :verbose => true)

We start with out Model-Term frequency matrix with is generated from creating a Vector Space Search with four documents (D1-D4):

           D1    D2    D3    D4
hat     [ +1.00 +0.00 +0.00 +1.00 ]
dog     [ +0.00 +0.00 +1.00 +0.00 ]
cat     [ +1.00 +1.00 +1.00 +0.00 ]
make    [ +0.00 +0.00 +1.00 +0.00 ]
good    [ +0.00 +0.00 +1.00 +0.00 ]
pet     [ +0.00 +1.00 +1.00 +0.00 ]
fine    [ +0.00 +1.00 +0.00 +0.00 ]
poni    [ +0.00 +1.00 +0.00 +0.00 ]
disabl  [ +1.00 +0.00 +0.00 +0.00 ]

Apply tf-idf transform:

           D1    D2    D3    D4
hat     [ +0.23 +0.00 +0.00 +0.69 ]
dog     [ +0.00 +0.00 +0.28 +0.00 ]
cat     [ +0.10 +0.07 +0.06 +0.00 ]
make    [ +0.00 +0.00 +0.28 +0.00 ]
good    [ +0.00 +0.00 +0.28 +0.00 ]
pet     [ +0.00 +0.17 +0.14 +0.00 ]
fine    [ +0.00 +0.35 +0.00 +0.00 ]
poni    [ +0.00 +0.35 +0.00 +0.00 ]
disabl  [ +0.46 +0.00 +0.00 <span style="color: #ff0000;">+0.00</span> ]

Perform SVD – Reduce Sigmas dimensions (removing the 2 smallest coefficients)

           D1    D2    D3    D4
hat     [ +0.34 -0.01 -0.01 +0.63 ]
dog     [ +0.01 -0.00 +0.28 -0.01 ]
cat     [ +0.03 +0.08 +0.06 +0.04 ]
make    [ +0.01 -0.00 +0.28 -0.01 ]
good    [ +0.01 -0.00 +0.28 -0.01 ]
pet     [ +0.01 +0.17 +0.14 -0.01 ]
fine    [ +0.02 +0.35 -0.00 -0.01 ]
poni    [ +0.02 +0.35 -0.00 -0.01 ]
disabl  [ +0.11 +0.02 +0.02 <span style="color: #ff0000;">+0.19</span> ]

Note the Word ‘disabl’ despite not being in D4 now has a weighting in that document.

Dependencies

Linalg is dependent on LAPACK which can be a challenge to install.

This article on setting up linalg on Mac OS X helped: http://www.commonmediainc.com/2008/03/24/building-lapack-and-rubys-linalg-on-mac-os-x/

Problems

LSA assumes the Normal distribution where the Poisson distribution has actually been observed.

Source Code

The source code initially shown here has gone through a lot of refactoring and has become much more than just LSA. You can get the full project with source code from github:

git clone git://github.com/josephwilk/rsemantic.git

And you can read about rSemantic on Github: http://josephwilk.github.com/rsemantic

Rboss RubyGem for Yahoo! Search BOSS

With Search BOSS (Build your Own Search Service) Yahoo has freed up a lot of the restrictions on their previous search service. Like removing the cap on the number of searches and allowing re-purposing of results. I’ve been doing some work on using the service in Ruby. I wrote a little RubyGem called Rboss which wraps around the BOSS webservice. It makes life nice and easy using Ruby and BOSS.

require 'rubygems'
require 'boss'

api = Boss::Api.new('boss-api-key-got-from-yahoo')

#Find news articles that are not older than 7 days
results = api.search_news('monkeys', :age => '7d')
results.each do |news|
   puts news.title
   puts news.abstract
   puts news.date
   puts news.url
end

Install Gem from GitHub:

  1. Add github to gem sources

    gem sources -a http://gems.github.com

  2. Install the gem:

    sudo gem install eshopworks-rboss

  3. If you don’t already have a BOSS api key signup for one: http://developer.yahoo.com/wsregap

Checkout the Rboss documentation and example usage at: http://github.com/eshopworks/rboss-gem

Thanks to eShopworks for sponsoring this project.

Telling a Good Story - Rspec Stories From the Trenches

I’ve been developing multiple systems using Rspec stories for a little while now. There are a lot of great resources to get you started with a taste of what you can do with stories. Some of the resources I found useful where:

However once I had understood the basic idea I struggled to find practical examples and general guidance on writing real stories. So I’ve collected some of the lessons I’ve learnt along the way with story examples taken from real systems and how I’ve improved them as I learnt. Most examples are from web based applications.

There are no absolutes in these lessons, they are general rules. If the customer is NASA and they want an extremely explicit and low level story then that is fine. Really stories are the customers expressions of requirements and each customer may want to express these in different ways.

On the other hand it is often you helping the customer take those first baby steps into using stories. They turn to you for guidance until they can stand on their own two feet.

Stories are not specs!

Specs are your unit tests, they’re fast and lean and always running, helping you refactor and develop your code. They describe the system at the object level (and dependent on how you mock/stub they do this in isolation). Stories are like integration tests, they document how your system works. They help you know that your system meets the customers requirements.

Your spec test is going to care about testing each action in your controller.

Your story is going to care about usage of the system cutting through models/controllers/views.

Hence specs tend to have a high coverage while stories cover only that which is important to the customer!

Some people do run their stories continuously like their specs (which is only really practical if you are using only Webrat). Generally I only run my stories before a commit and I also have them running on a continuous integration server (CruiseControl.rb) on every commit.

Keep the story goals as real values for the customer

Story: Search
  As a website user
  I want a search box
  So that I can enter terms and click search

Why does the website user want to enter terms and click search? To find things!

Story: Search
  As a website user
  I want to search
  So that I can find content on the website

This is where what David Chelimsky mentions as ‘popping the “why”? stack’ helps us keep focused on real customer values.

Stories should not be exhaustive

Story: Changing pages urls
  As a website user
  I want to change the URLs of pages
  So that I can improve any bad urls

Scenario: valid url
Scenario: blank url
Scenario: symbol url
Scenario: character url
Scenario: numeric url

I find here we can get a bit more generic and concentrate of what is really important in this story. There are two different behaviours valid and invalid. (This does bring up the idea that stories can follow a behavioural or stateful approach, which I’ll leave as a discussion for another time)

Story: Changing page urls
  As a website user
  I want to change the URLs of pages
  So that I can improve any bad urls

Scenario: invalid url
  Given a title which is invalid in a url
  When I save
  Then it should show me an error message telling me why it was invalid

Scenario: valid url
  Given a title which is valid in a url
  When I save
  Then it should not show me an error message

This serves as a good base for the client to add to and refine as their requirements evolve. I prefer the stories to facilitate discussion rather than being documentation which starts to happen when you get too exhaustive.

Stories should speak with the customers domain terminology

Ideally your stories should be written by the customer in their terminology. Even if you don’t have direct access to a customer try putting yourself in their shoes.

With web applications if you find yourself mentioning things like ‘database’, ‘views’, ‘controllers’, ‘models’ and ‘sql’ your writing like a programmer and not an average customer (Well that’s what I’ve generally found). In the previous example note how we saved a whole lot of work by saying:

'valid in a url'

Rather than defining explicitly the different rules for what is a valid url in the story. So using the domain terminology can help make your stories more concise.

Stories should not be too low level.

Story: Viewing homepage
  As a logged in user
  I want to view my homepage
  So that I will see it

Scenario: login
  Given I'm logged in
  When I visit my homepage
  Then I will see it

If it takes you longer to write the story than complete the task its too small. When you end up with a couple of stories which are too small general its a good idea to combine them to form a more reasonably sized story. In this case I took the Viewing homepage story and combined it with a set of small stories:

Story: Important pages are accessible
  As a logged in user
  I want to visit key pages in the site
  So that I can successful navigate around it<span style="text-decoration: line-through;">
</span>
Scenario: Homepage
Scenario: Login page
Scenario: Payment page

Stories should not be too high level.

Stories that are too large are referred to as Epic stories. Epics tend to occur when the story scenarios are trying to be too exhaustive or the story is a:

  • Compound story – comprises multiple shorter stories

  • Complex story – Cannot easily be broken down into smaller stories

An example of an epic:

Story: Manage Content
  As a admin user
  I want to manage my sites pages
  So that I can keep my website up-to-date

Scenario: Add a page
Scenario: Delete a page
Scenario: Edit a page

I find a good guideline is that a story should be accomplishable in around 1 week.

There are two ways you can split a epic story into snappy short stories:

Split Along create, add, edit and delete. or Disaggregate along the boundaries of the data

So initially I went with disaggregating along the data boundaries but still found the stories where looking big, so I then tried splitting on actions which left me with some nice snappy stories.

Story: Add page
  As a admin user
  I want to add pages
  So that I can keep my website up-to-date

Scenario: ...

Story: Delete page
  As a admin user
  I want to delete pages
  So that I can remove bad or old articles

Scenario: ...

Story: Edit page
  As a admin user
  I want to edit pages
  So that I can refine my articles as time passes

Scenario: ...

Note how our understanding of the users goals has expanded here. We know in more detail what the user wants from the system.

When choosing which way to split stories I often write out what the stories and scenarios would look like which helps me get a feeling for which splitting method is best suited.

Stories should concentrate on outputs not inputs

Story: Creating an account
  As a new user to the site
  I want to create a account for the site
  So that I can login and comment on articles

Scenario: valid account details
  Given I'm logged out
  And I've entered my username 'spoink'
  And I've entered my website 'http://www.joesniff.co.uk'
  And I've entered my email address 'spoink@joesniff.co.uk'
  And I've entered my password 'poodle'
  And I've entered my password 'poodle'
  And I've entered a password reminder 'not a monkey'
  And I've selected 'do not send me emails'
  And I've selected 'I agree to terms and conditions'
  When I click the create account button
  Then my account will be created
  And I will be redirected to my account page

Outputs represent value to the customer. They tend to be far more interested in what they get out of the system than what they put in.

Story: Creating an account
  As a new user to the site
  I want to create a account for the site
  So that I can login and comment on articles

Scenario: valid account details
  Given I'm logged out
  When I create a new account
  Then I will be redirect to my account page
  And I will be presented with the last time I logged in
  And my current community ranking
  And the last comment I made.

In this example focusing on all the details of creating an account while useful missed the fact that the most important thing to the client was why the user was creating the account and what benefit/output they got that kept them there.

Stories should slice through multiple layers

Stories act as integration tests cutting through all the layers of your web application stack.

If you only cut through certain layers you introduce the possibility that the stories pass without parts of your system working!

Story: Fill in form
  As a logged in user
  I want to fill in a register form
  So that I can register

Story: Save in database
  As a logged in user
  I want my form information saved in the database
  So that the system remembers me

Yuck! Hearing the user talk about ‘the database’ is a sure sign these stories have problems.

Stories should be kept organised

I tend to use one of two ways of organising my stories in the file system:

1. Organise by Stakeholder

Contain the stories based on which stakeholder is acting in the story.

stories/
  customer/
    purchase.story
    purchase.rb
  web_master/
    ...
  advertiser/
    ...

2. Organise by Feature

Contain the stories based on which feature stories relate to. (This is particularly useful if you’re structure follows Epic –> Feature –> Story ).

stories/
  page_editing/
    edit_page.story
    edit_page.rb
  image_uploading/
    ...
  login
    ...

Use GivenScenario to make Stories easier to read and discuss

GivenScenario allows you to reuse scenarios which can help make the stories clean and in turn easier to discuss with the customer.

Story: maintaining blog
  As a web admin
  I want to manage pages within an admin system
  So I can keep my blog up to date

Scenario: Add page
  ...

Scenario: edit page with valid details
  GivenScenario: Add page
  When I edit pages details
  Then it should save without errors

Stories should not leave hidden magic in steps

When scenarios start to fail in your stories its often up to the developer to have a look why. You often turn to the story to check the steps.

...
  Given there is a page 'testing'
...




Given(/there is a page '(.+)'/) do |title|
  @page = Page.create!({ :title => title)
  @page.images = Image.create!({ :test )
end

Misleading or not revealing hidden steps can making debugging stories a painful process. I changed the story to make sure it was obvious what was going on:

...
  Given there is a page 'testing' with an image 'test'
...

Make your Given/When steps fail loudly

Stories can be hard to debug. Anything in the Given & When could have failed but we are only aware of a Then failing. So we have to search back through the Givens and Whens to spot an error.

The following could fail but the step would pass.

Given('there is a page') do
  Page.create {:title => 'explode early'}
end

By throwing an exception we can ensure that if anything goes wrong the step fails

Given('there is a page') do
 Page.create! {:title => 'explode early'}
end

Avoid relying heavily on instance values in steps

Relying on using instance values makes it difficult to reuse your steps. Also Interdependencies between steps can make them harder to debug.

Given('there is a page') do
 @page = Page.create! {:title => 'dont do it!'}
end

Ruby/Rails Interview Questions

I’ve recently been helping interview some ruby/rails developers. I searched the web for some inspiration but I could not find any example questions that had real depth to them. I like my questions to be a point of discussion rather than one word answers. Most importantly I want a wide enough scope to let those talented individuals shine through. So here are some of the questions I’ve been trying out recently.

1. You are getting the chance to fly in the worlds first unmanned Airplane who’s systems are written only in ruby. Would you fly in it? Explain your reasoning

2. Ruby is great and everything but its nothing more than a prototyping language. Its just too hacky and dynamic for any real production system. Why bother with its messy Perl lineage and the lack of internationalisation support when you could just use Python? Discuss.

3. Ruby takes a unique approach to the problem of multiple inheritance. Explain ruby’s approach and the strengths and weaknesses of it.

4. Do you think adding behaviour to the builtin core Ruby classes is a good idea? Can you give some examples to backup your opinion.

5. Explain why in ruby nil.object_id is equal to 4. (Rather nasty question, really asking about C)

>> nil.object_id
=> 4

6. Twitter(http://www.twitter.com) is a website built on Ruby on rails. It is a

“Social networking and microblogging service utilising instant messaging, SMS or a web interface.”

Why do you think twitter (http://www.twitter.com) used Ruby on Rails?

Do you think it was a good decision?

7. Explain what ‘has_many’ is and what happens when it is run.

class Monkey < ActiveRecord::Base
  has_many :bananas
end

8. Given a simple website focused on a REST model and produced solely by using script/generate scaffold. Explain what happens in the rails application when a user submits a form with a POST request to: ‘/images/1’. State any assumptions you make.


9. If each language was represented as a person what type of person would each be and why?

  • Ruby person

  • Python person

  • Php person

  • Java person

Rspec-rails Mock_model Helper for the RR Test Double Framework

Rspec-rails is a rails plugin which brings the Rspec Ruby Behaviour Driven Development framework to rails along with some rails specific helpers. One of these hugely useful helper functions is:

mock_model(model_class, options_and_stubs = {})

This creates a mock object with the common methods stubbed out. It also allows you to specify other methods you want to stub.

You can use Rspec with a number of mocking frameworks (Rspec 1.1.5: mocha, flexmock & RR). Unfortunately the Rspec-rails mock_model helper is not compatible with the RR test double framework. It throws this error message:

undefined method `to_sym' for {:count=>0}:Hash

as it uses a stub function:

:errors => stub("errors", :count => 0)

which does not match RR’s stub method.

I’ve written a patch which will allow mock_model to work with RR. Note: It’s still a work in progress but I am using it successfully in a number of projects.

You can get the latest code from GitHub: git clone git://github.com/josephwilk/rspec-rr.git

[viewcode] src=../projects/ruby/rr/mocks_rr_extensions.rb geshi=ruby showsyntax=display[/viewcode]

Textmate Bundle for RR Test Double Framework

A simple Textmate bundle for  RR the Ruby test double framework.  You can read about RR at http://github.com/btakita/rr/tree/master and look through the latest rdocs at Rubypub

Install with Git

(what on earth is Git…)

  1. Run this:

    mkdir -p ~/Library/Application\ Support/TextMate/Bundles/ cd ~/Library/Application\ Support/TextMate/Bundles/ git clone git://github.com/josephwilk/rr-tmbundle.git rr.tmbundle

  2. Reload bundles in Textmate

  3. Enjoy!

Rspec Stories - Keeping Steps Dry

When using Rspec stories you have plain text stories which we call the ‘story’ file and the ‘story steps’ file that maps the plain text story to programmatic code. Generally you end up with your story files not being DRY. This is not a worry, your stories are the domain specific languages detailing your acceptance/integration tests. Its like saying that your Rails Models are not DRY because they repeat lots of 'has_one'!

However within a single story file you can DRY things up a bit by reusing scenarios.

GivenScenario

We can use ‘GivenScenario: SCENARIO’ within a scenario to call another scenario. Pretty much like a method call, it will run the called scenario and then return to continue with the original scenario.

[viewcode] src=../projects/ruby/stories/given_scenario_story.rb geshi=ruby showsyntax=display[/viewcode]

Drying The Story Steps…

More so than stories we should try and keep our story steps DRY. Duplicating ourself in steps makes it harder to maintain the tests and all the other horrors of breaking DRY. So here are some simple steps to help ensure that your step files stay nice and DRY.

Regular Expression Steps

To make your steps a little more flexible you can use relative expressions. Within your Given/When/Then functions rather than using a string parameter you use a regular expression:

[viewcode] src=../projects/ruby/stories/regular_expression_steps.rb geshi=ruby showsyntax=display[/viewcode]

Common Step files

Creating common step files which contain frequently reused Given/Then/Whens. I generally end up with a Selenium and a Webrat common step files since they are both core to driving my stories.

The common steps reside with the other step files:

  • steps/common_webrat_steps.rb

  • steps/common_selenium_steps.rb

Create the steps file as you would with any other rspec step file. [viewcode] src=../projects/ruby/stories/common_webrat_steps.rb geshi=ruby showsyntax=display[/viewcode]

Then reference the common step file when you are creating the runner for your story: [viewcode] src=../projects/ruby/stories/example_story.rb geshi=ruby showsyntax=display[/viewcode]

Helpers

Within your stories folder generated by rspec you have your lovely helper file:

stories/helper.rb

Within here you can create functions that are accessible within all steps.

You may start to find that your helper becomes a gigantic list of helper methods. Hence I tend to organize helpers into modules.

You can mix these helpers into Rspec stories so all steps can access them:

[viewcode] src=../projects/ruby/stories/helper.rb geshi=ruby showsyntax=display[/viewcode]

JavaScript Acting as a Robotic Agent

We can think of JavaScript running within a clients browser as a robotic agent. It has an environment in which it can sense things. The ability to look at the environment and make decisions based on plans.

clientagent.JPG

So whys that useful, well why is a robot useful? You can produce many different complex plans and give them to the robot and forget about it while it does the work potentially over and over again. If we are really lucky the robot can demonstrate some intelligence and deal with uncertainty.

Well I tried out a small part of this idea to build a server side service which delivered plans in JavaScript to the client. The JavaScript planning agent followed the plans. Its not a intelligent robot but this is just a prototype. The plans where focused on validation conditions that a user needed to get through to post a form.

Example:

1
2
3
4
planCollection = Array();
planCollection[0][0]= Array();
planCollection[0][0]= document.forms['iwfmsForm'].elements['age'].value>18;
planCollection[0][1]= document.forms['iwfmsForm'].elements['ward'].value=='adult;

Timings where implied by ordering of plans in an array. i.e. for plan 0 – value > 18 must happen before ward == adult.

Array dimensions imply different plans or forks in the plan.

Its a rather crude plan referencing form names and lots of low level javascript but it helps simplify the boring bits!

The Environment

The environment represents the different states the form elements and the user can be in.

The See process

This function maps the form elements to their associated values.

The Message Store

The message store is used to represent events generated by the user. Events could be the result of clicking on a form element or clicking away from a form element. Such events generate messages which the plan executor responds to. These events are the points where the planner is activated.

The Plans

The plans represent a sequence of constraints within the JavaScript syntax on form elements. This type of action is not executed but used to help decided upon which plan path to follow.

The Action process

This process deals with selecting actions based on the see process which provides information about the form element’s values. There may only be a single plan and thus no decision to make. If there are multiple plans the decision to follow an action is only made for the first action of a plan. Which initial action to decide upon is decided by testing each of the first actions of the plans until one holds from the information provided by the see process. This plan is then used to further test the next constraint of the plan and so on. If multiple first actions hold each of the plans is tested to see if any of them reach the goal.

The Goals

The goals of the plans can be regarded as ensuring that all the conditions in the plan hold based on the data in the HTML form.

The Execution of Actions

Within the planner the actions are not executed. Since we are constraining the user all we need to do is find a valid plan path. The planner affects the environment by alerting the user if it cannot find any valid plan path.

The plan executor needs to be able to handle two different situations:

  • Form submission When the user inputs information into a form and then submits the form the planner will have to ensure that a valid plan path is reflected by the values that are held in the form. The form submission should be prevented if such a plan cannot be found.

  • Real time user input

      1. Guidance through a plan As the user inputs information the planner needs to be aware of the current position in multiple plans and what effect this has on the current form element being filled in. Alerting the user if they have failed to achieve any plan as a result of the data they entered into the form.

        1. Temporal ordering restrictions The planner needs to be able to handle temporal restrictions on the form elements. Indicating in which order form elements have to be completed. Therefore we have to look at the impact that a real time input action will have on the rest of the form. For example one form element being completed may have the effect that another disabled form element can be activated for input.

JavaScript Planning agent Code

https://github.com/josephwilk/iwfms/blob/master/javaScript/javaScriptPlanner.js

Rails Admins Plugins Review

A brief examination of some of the major Admin plugins for rails.

  • Lipsiaadmin

  • AutoAdmin

  • ActiveScaffold

  • Hobo

  • Streamlined

Lipsiadmin

http://rails.lipsiasoft.com/wiki/lipsiadmin

Google group Members: No group

Live Demo: http://demoadmin.lipsiasoft.org/admin/

Sample Projects: None

This admin framework mixes Ajax/JavaScript (library Ext 2.0 views) and old school HTML. The interface is presented in an application style reminiscent of Microsoft Outlook.

Lipsia Admin

Feature set

  • http://extjs.com/ JavaScript library.

  • Generator Admin code.

  • Permissions support

  • Live search

  • Creation of menus within migrations.

Weaknesses

  • Community seems non-existent

  • Sparse documentation


AutoAdmin

http://code.trebex.net/auto-admin

Google group Members (http://groups.google.com/group/rails-autoadmin): 37

Live Demo: None

Sample Projects: None

Heavily influenced by Djangos admin system in both theme and with the definition of admin display configuration within in models. Sub-projects within Django have already started to move away from this approach (newforms-admin).

Its goal is to generate views you otherwise wouldn’t bother to create.

AutoAdmin

Feature set

  • Themes

  • Basic Access Control – All or Nothing

Weaknesses

  • Lacks reliable set of Tests

  • Admin display configuration is stored within Models (Django branches newforms-admin have already moved aways from this)

  • Problems with editable sublists

  • Access control does not support more complex use cases.


ActiveScaffold (Version 1.1.1)

http://www.activescaffold.com

Google group Members (http://groups.google.com/group/activescaffold): 1075

Live Demo: http://demo.activescaffold.com/

Sample Projects: (SVN) http://activescaffold.googlecode.com/svn/applications/demo/

A Ajax driven admin system which promotes itself on being both configurable and customizable.

Configuration: “throwing flags and setting options and manipulating the setup.”

Customizing: “defining conventional methods or .rhtml overrides that it will intelligently use when available.”

ActiveScaffold

Admin customisation is performed in the controller.

Feature Set

  • Installed through plugin

  • Ajax driven interface

  • Embed ActiveScaffold like a widget in your web page.

  • High quality Documentation

  • Supports Third Party plugins within active scaffold : http://wiki.activescaffold.com/wiki/published/ThirdPartyPlugins

  • Graceful JavaScript degradation

  • Hugely Customizable

    • Admin interface

      • Grouping form inputs

      • Layout

  • Guaranteed to work on Firefox 1+, IE 6+ and Safari 2+

Weaknesses

  • No concept of semantic markup for models so configuration takes over when a little magic could help reduce the work in some usages.

  • Ajax interface is not amazingly usable.


Hobo (version 0.7.2)

http://hobocentral.net/

Google group Members: No Group

Live Demo: None

Sample Projects: http://hobocentral.net/blog/demos/

The Web App Builder for Rails. Focuses on the models providing magic to make the admin appear.

Hobo Admin

Feature Set

  • Installable through gem

  • DRYML – XML based markup language

  • Migration generation from Models

    • Define the table fields in the model

    • High level semantic definitions for model fields (eg. email address)

  • Permission System

  • Themes_ _

  • ActiveRecord extensions

Weaknesses

  • Documentation is a little incomplete.

  • Screencasts are out of date


Streamlined (version 0.9)

http://streamlinedframework.org/

Google group Members (http://groups.google.com/group/streamlined ): 474

Live Demo: None

Sample Projects: http://trac.streamlinedframework.org/wiki/SampleProject

Streamlined provides an instant, production-ready UI for your ActiveRecord model. It aims to bring the declarative goodness of ActiveRecord to the view tie. It has a nice balance of ajax functionality with traditional page loads.

Streamlined admin

Feature Set

  • Separates Admin declarations and Model

  • In place editing (edit content in listing)

  • Export content to XML, CSV, and JSON

  • Pluggable CSS styling

  • Live data filtering

  • Advanced Filtering

  • Ajax-powered widgets and transitions

  • Views are can be overwritten Globally and per model

  • Good documentation + screencasts

  • IBM guides – http://www.ibm.com/developerworks/java/library/j-cb09056/

Weaknesses

  • No attaching of semantic markup to model fields.

  • The documentation (while improving) is a little patchy in places.

Conclusions (winners)

ActiveScaffold

If your goal is to have as much flexibility as possible within your admin then ActiveScaffold is you best candidate. It is incredibly versatile and the documentation is a pleasure to work through. With developers contributing plugins its future growth looks promising.

Hobo

If your looking for an admin system and a framework and are less concerned about the configuration of the admin then Hobo is your is candidate. The power to attach semantic data to models has great potential for complex forms elements being generated automatically. Currently only the ‘email’ type is provided so this feature is yet to take off. Be prepared to deal with incomplete documentation. The Mirc #hobo group and the hobo forums are happy to help if you get stuck.