Joseph Wilk

Joseph Wilk

Programming bits and bobs

Live Coding - Repl Electric

Live coding is the act of turning a programming session into a performance. This can constitute improvisation, music, visuals, poetry, hardware, robots, dance, textiles and people. Pretty much anything with an input and output can be controlled live by programming.

This is not just a performance by programmers for programmers. While this is often where it starts as a live coder, the type of audience and the accessibility of the performance lies in the performers imagination. Abstraction can get us pretty much anywhere.

1
(def the-stars (dark-matter))

Repl Electric

Repl Electric is a project I started in order to discover more about music composition and Artificial intelligent based aids to creativity. Which in turn through the inspiration of people like Meta-ex lead me to live programming music.

Here is a performance live coding music and graphics, inspired by a performance in London:

The Stars

Open Live Coding

All the tools and code used to create this performance are open for all to see on Github: https://github.com/repl-electric

Three programming languages were used to create this piece:

  • Clojure (Sound)
  • GLSL (Visuals)
  • Emacs Lisp (Animations & Navigation)

Tools

Here are the tools used and a little detail around how they where used in performing “The Stars”:

Clojure: http://clojure.org

Clojure is a LISP language based on the JVM.

Clojure focuses on interactive REPL (Read, Evaluate, Print & Loop) driven development. Which makes it a good choice for interactively coding music. It also turns out functional programming is a good fit for operating over music as data.

Emacs Live: https://github.com/overtone/emacs-live

Emacs live is a Emacs release with packages and defaults that are Live Coding centric. Something I use for both for my work and for my live coding.

To execute our code, we launch a repl instance in our project (NEVER launch inside emacs, since then if emacs crashes the repl and music dies) and connect to it from emacs using cider https://github.com/clojure-emacs/cider.

A simple trick to combine Emacs code and visualizations is to launch an OpenGL window in full screen (see Shadertone) and then put a full screen transparent terminal window running emacs over it.

The tumbling text effect seen at the end of the performance is an emacs animation using Zone Mode which supports writing your own text destructors: http://www.emacswiki.org/emacs/ZoneMode

Overtone: https://github.com/overtone/overtone

Overtone is a Clojure based client to SuperCollider. Supercollider is an environment for real time audio synthesis and algorithmic composition.

Overtone provides us with:

  • Timing (beat generation – example timing code).
  • Building Synths (engineer sound).
  • Running samples (both your own and from Freesound).
  • Live Synth control (changing notes, durations, reverb, etc).
  • Hardware interaction (through midi or OSC).

An example of a synth used in The Stars:

1
2
3
4
5
6
7
8
9
10
(use 'overtone.live)
(defsynth dark-ambience [out-bus 0 amp 1 mul 0.2 room-size 70 rev-time 99 freq 60 ring-mul 55]
  (let [pink (hpf:ar (* (* 0.005 (pink-noise)) (line:kr 0 1 9)) 5)
        src1 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 0)) mul)
        src2 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 1)) mul)
        src3 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 2)) mul)
        src (tanh (g-verb (sum [src1 src2 src3]) room-size rev-time))]
    (out out-bus (* amp src))))

(def the-stars (dark-ambience))
Timing

Timing is a complicated issue but so important its worth touching on. You have a choice with Overtone to use Java for timing or Supercollider. I use Supercollider since I have found it to be much more reliable. Everything you need is here (copy and paste), thanks to the hard work of Sam Aaron.

The key concept to take away is there are two types of timing, a beat counter which is forever incrementing and a beat trigger which flips back and forth between 1/0.

1
2
3
4
5
6
7
(require '[cassiopeia.engine.timing :as time])

;;The beat trigger
(:beat time/main-beat) ;=> 0,1,0,1,0,1,0

;;The beat counter
(:count time/main-beat) ;=> 0,1,2,3,4,5,6,7,8,9

The counter is useful for indexing buffers, the trigger is useful in controlling the gate of an envelope (which turns a sound on or off).

In Clojure we can still get access to the beat, in our timing code we send a message using send-trig on every beat. We can hook a Clojure function to callback on this beat:

1
2
3
4
5
6
(on-trigger (:trig-id time/main-beat)
  (fn [beat-no]
    (when (= 0.0 (mod beat-no 32))
      ;;play a sample
      (boom-s)))
  ::on-beat-trigger)

I use this extensively to time graphic transitions with the music.

Buffers

Most of my live coding performance was writing to buffers which are hooked into synths. Buffers are just fixed size arrays but they are stored in Supercollider rather than in Clojure. Here is an example from The Stars where the midi notes are read from a buffer at a rate based on my beat timing signal (a 16th of the main beat here).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
(use 'overtone.live)
(use 'cassiopeia.engine.core)
(require '[cassiopeia.engine.timing :as time])

(defsynth growl [out-bus 0 note-buf 0 beat-bus 0 beat-trg-bus 0 amp 1]
  (let [cnt (in:kr beat-bus)
        trg (in:kr beat-trg-bus)
        note (buf-rd:kr 1 note-buf cnt)
        freq (midicps note)
        vol (> note 0)

        e (env-gen (perc :attack 10 :sustain 1 :release 1) :gate trg)
        src (lpf (mix [(saw (* 0.25 freq))
                       (sin-osc (* 1.01 freq))]))
        src (pitch-shift src 0.4 1 0 0.01)
        src (pan2:ar (* vol amp e src))]
    (out out-bus src)))

(defonce nebular (buffer 96))

(def nebula (growl :note-buf nebula-note-buf :beat-trg-bus (:beat time/beat-16th) :beat-bus (:count time/beat-16th)))

(pattern! nebula-note-buf (degrees [1 3 7] :major :A2))

GLSL + Shadertone: https://github.com/overtone/shadertone

Shaders generate imagery directly on your Graphics Processing Unit rather than going through your CPU. Through a language called GLSL (which is C like) we can express very simple functions which get called on every single pixel generating complex visuals. Here is a simple extract from The Stars that generates all the background small dots:

1
2
3
4
5
6
7
8
9
10
11
12
void main(void){
  vec2 current_pixel_position = mod(gl_FragCoord.xy, vec2(5.0)) - vec2(0.0);
  float distance_squared = dot(current_pixel_position, current_pixel_position);

  vec4 black = vec4(.0, .0, .0, 0.0);
  vec4 white = vec4(1.0, 1.0, 1.0, 1.0);

  //Test the current pixel position and if it should be a circle shade it.
  vec4 circles = (distance_squared < 0.6) ? white : black;

  gl_FragColor = circles;
}

For more examples of whats possible with Shaders checkout Shader Toy

Shadertone is the Clojure library that provides a convenient way of running shaders from Clojure and for feeding in data about our synths. It provides access in your Shader to:

  • Overtone’s Volume (iOvertoneVolume)
  • The frequency spectrum & audio waveform data (Passed as a 2D texture :textures [:overtone-audio])

To synchronize the graphics with the music I created a special Overtone synth which does not generate any sound, it instead feeds information in realtime to my shader.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
(use 'overtone.live)
(require '[shadertone.core :as t])

;;A synth that exposes through taps all the lovely timing information.
(defsynth buffer->tap [beat-buf 0 beat-bus 0 beat-size 16 measure 6]
  (let [cnt (in:kr beat-bus)
        beat (buf-rd:kr 1 beat-buf cnt)
        _  (tap "beat"          60 (a2k beat))
        _  (tap "beat-count"    60 (a2k (mod cnt beat-size)))
        _  (tap "measure-count" 60 (a2k (/ (mod cnt (* measure beat-size)) measure)))])
  (out 0 0))

;;; Used to store our drum beat, 1 for a hit 0 and for a miss
(defonce drum-sequence-buffer (buffer 256))

(def beats (buffer->tap drum-sequence-buffer (:count timing/main-beat)))

;;Open a OpenGL window running our shader
(t/start-fullscreen "resources/shaders/electric.glsl"
                    :user-data {
                    "iBeat"         (atom {:synth beats :tap "beat"})
                    "iBeatCount"    (atom {:synth beats :tap "beat-count"})
                    "iMeasureCount" (atom {:synth beats :tap "measure-count"})})

Inside our shader code:

1
2
3
4
uniform float iBeat;
uniform float iBeatCount;
uniform float iMeasureCount;
...

The other main way of controlling a shader from Clojure is using atoms.

1
2
3
4
5
6
7
(require '[shadertone.core :as t])

(defonce cellular-growth (atom 0.0))

(t/start-fullscreen "resources/shaders/electric.glsl" :user-data {"iCellularGrowth" cellular-growth})

(swap! cellular-growth + 0.01)

Hardware: Monome: http://monome.org

Something you don’t see in the video is that I’m using a 8x16 Monome. For this performance its primary function was a visual aid to show the beat/measure information.

Monome

The hardware is driven by Clojure communicating with the Monome through a serial port: https://github.com/josephwilk/monome-serial/tree/protocols

Live Coding

Live coding music and graphics combines skills in sound engineering, 3d graphics, geometry, physics, musical theory, composition, improvisation & hardware to name a few.

It is difficult, and requires a lot of work and practice.

But of all the code I’ve written over the years this is one of the things I’m most proud of. And i’m only at the beginning of discovering what’s possible.

Creative Machines

When Alan Turing asked if a machine can be intelligent one aspect of this question focused on “could machines be creative”?

Ada Lovelace seemed convinced that originality was not a feat a computer was capable of:

it can do whatever we know how to order it to perform, it has no pretensions whatever to originate anything

Before we outrightly dismiss the idea of creative machines do we even understand what creativity is?

Join me on a journey examining these questions while also meeting a new generation of artists born through code. Looking into their hearts and brains examining different algorithms/techniques and there effectiveness at exhibiting creativity.

Decide for yourself, can machines be creative?

My Strangeloop talk: Creative Machines – http://www.infoq.com/presentations/ai-machine-creativity

Slides:

Full source code: https://github.com/josephwilk/musical-creativity

Continuing the Journey

In order to continue my research into creativity and music I’ve started the project Repl Electric. A space for humans and machines to learn and create music together.

Further Reading

Sounds of the Human Brain

What does your brain sound like? Does it sound like “Rise of the Valkyrie” or more like “Hit me baby one more time”?

Step 1: Acquire a human brain (alive)

We are interested in capturing the brain waves. Specifically the:

  • Delta waves: Deepest stages of sleep.
  • Beta waves: Normal waking consciousness.
  • Alpha waves: Relaxation and meditation (creativity).
  • Theta waves: REM sleep (dreams).
  • Gamma waves: Hyper-alertness, perception, and integration of sensory input.

Step 2: EEG machine

I am using a EEG machine brought from Neurosky which is rated as Research grade (whatever that means). This measures voltage fluctuations resulting from ionic current flows within the neurons of the brain. While EEG machines are not the most accurate they are now reasonably cheap.

Step 3: EEG –> Overtone

In order to generate music I want to import the EEG brainwave data into Overtone.

We interact with the EEG machine over a serial port. The most mature library for this interface is in Python so there is a little jiggery pokery to get the data into Overtone.

The Brainwave Poller

We use https://github.com/akloster/python-mindwave to interface with the EEG machines data.

Writing all the data out to a FIFO file file as json.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
import re
import time
import json
import unicodedata

import gevent
from gevent import monkey

from pymindwave import headset
from pymindwave.pyeeg import bin_power

monkey.patch_all()

# connect to the headset
hs = None
hs = headset.Headset('/dev/tty.MindWave')
hs.disconnect()
time.sleep(1)
print 'connecting to headset...'
hs.connect()
time.sleep(1)
while hs.get('state') != 'connected':
    print hs.get('state')
    time.sleep(0.5)
    if hs.get('state') == 'standby':
        hs.connect()
        print 'retrying connecting to headset'

def raw_to_spectrum(rawdata):
    flen = 50
    spectrum, relative_spectrum = bin_power(rawdata, range(flen), 512)
    return spectrum

while True:
    t = time.time()
    waves_vector = hs.get('waves_vector')
    meditation = hs.get('meditation')
    attention = hs.get('attention')
    spectrum = raw_to_spectrum(hs.get('rawdata')).tolist()

    with open("/tmp/brain-data","w") as fp:
        s = {'timestamp': t,
             'meditation': meditation,
             'attention': attention,
             'raw_spectrum': spectrum,
             'delta_waves': waves_vector[0],
             'theta_waves': waves_vector[1],
             'alpha_waves': (waves_vector[2]+waves_vector[3])/2,
             'low_alpha_waves': waves_vector[2],
             'high_alpha_waves': waves_vector[3],
             'beta_waves': (waves_vector[4]+waves_vector[5])/2,
             'low_beta_waves': waves_vector[4],
             'high_beta_waves': waves_vector[5],
             'gamma_waves': (waves_vector[6]+waves_vector[7])/2,
             'low_gamma_waves': waves_vector[6],
             'mid_gamma_waves': waves_vector[7]}

        s = json.dumps(s)
        fp.write(s)
    gevent.sleep(0.4)

Reading from the FIFO file is simple in Clojure.

1
2
3
(while true
  (with-open [reader (clojure.java.io/reader "/tmp/brain-data")]
    (brainwave->music (json/decode (first (line-seq reader)) true))))

Step 3: Sonification

Sonification is the process of taking data and turning it into sound. Here is an example of the data we are now receiving:

A single JSON brainwave packet:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
 {"gamma_waves": 95408,
  "high_beta_waves": 205681,
  "beta_waves": 293928,
  "low_beta_waves": 382176,
  "mid_gamma_waves": 84528,
  "low_alpha_waves": 172417,
  "delta_waves": 117933,
  "low_gamma_waves": 106288,
  "alpha_waves": 112605,
  "theta_waves": 635628,
  "high_alpha_waves": 52793,
  "attention": 0,
  "meditation": 0,
  "timestamp": 1.375811275696894E9}

We will focus on the beta-waves for simplicity. Beta-waves fall between 16.5–20Hz.

EEG beta waves

Beta waves related to:

  • Alertness
  • Logic
  • Critical reasoning

We need to map a signal within 16.5-20Hz into the musical pitches of a sampled piano (21-108 pitches).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
(require '[clojure.math.numeric-tower :as math])

(defn linear-map [x0 x1 y0 y1 x]
  (let [dydx (/ (- y1 y0) (- x1 x0))
        dx (- x x0)]
    (+ y0 (* dydx dx))))

;; Piano range: 0..87
;; Beta wave range: 16500..20000

(defn beta-wave->pitch [signal] (-> (linear-map 16500 20000 21 108 signal) float math/round))

(beta-wave->pitch 16500) ;-> 21
(beta-wave->pitch 20000) ;-> 108

Now we extract the beta-waves from the brainwave data and play them. We play them live as they arrive rather than worrying about scheduling notes:

1
2
3
4
5
6
7
8
9
(use 'overtone.live)
(use 'overtone.inst.sampled-piano)
(require '[cheshire.core :as json])

(while true
  (with-open [reader (clojure.java.io/reader "/tmp/brain-data")]
    (let [beta-wave (-> (first (line-seq reader)) (json/decode true) :beta_waves)]
      (println beta-wave)
      (sampled-piano :note (beta-wave->pitch beta-wave) :sustain 0.2))))

Would you like to hear my brain?

The results, please listen to my brain.

Not really music is it? With beta-waves we get a serious of high to low transitions. While we can control at what pitch the transitions occur by performing activities that shape our brain waves the transitions don’t provide the order or structure we need to recognize this as music.

Brain controlled Dubstep

The only logical path left is to try and control Dubstep with our brain. Rather than generative music we can use our brain waves to control the tempo and volume of existing synthesized music.

Here is a Dubstep synth taken from Overtone:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
(use 'overtone.live)

(defsynth dubstep [bpm 120 wobble 1 note 50 snare-vol 1 kick-vol 1 volume 1 out-bus 0]
  (let [trig (impulse:kr (/ bpm 120))
        freq (midicps note)
        swr (demand trig 0 (dseq [wobble] INF))
        sweep (lin-exp (lf-tri swr) -1 1 40 3000)
        wob (apply + (saw (* freq [0.99 1.01])))
        wob (lpf wob sweep)
        wob (* 0.8 (normalizer wob))
        wob (+ wob (bpf wob 1500 2))
        wob (+ wob (* 0.2 (g-verb wob 9 0.7 0.7)))

        kickenv (decay (t2a (demand (impulse:kr (/ bpm 30)) 0 (dseq [1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0] INF))) 0.7)
        kick (* (* kickenv 7) (sin-osc (+ 40 (* kickenv kickenv kickenv 200))))
        kick (clip2 kick 1)

        snare (* 3 (pink-noise) (apply + (* (decay (impulse (/ bpm 240) 0.5) [0.4 2]) [1 0.05])))
        snare (+ snare (bpf (* 4 snare) 2000))
        snare (clip2 snare 1)]

       (out out-bus (* volume (clip2 (+ wob (* kick-vol kick) (* snare-vol snare)) 1)))))

Once the synth is running we can send it control signals which will vary any of the properties defined in the arguments to the dubstep function:

  • bpm
  • wobble
  • note
  • snare-vol
  • kick-vol
  • volume
1
2
3
4
5
6
7
(def d (dubstep))

(ctl d :snare-vol 0)
(ctl d :kick-vol 0)
(ctl d :wooble 0)
(ctl d :bpm 20)
(ctl d :v 0.2)

We again have to linearise the beta wave signal to the range of volume 0.0-1.1 and to the bpm 0-400.

Now all thats left to do is connect it to our brain.

Here’s what brain controlled Dubstep sounds like:

And for comparison what playing Go does to your brain activity (I turned the Dubstep down while playing, concentrating with that noise is hard):

Discovery through sound

Mapping brain waves into live music is a challenging task and while we can control music through an EEG machine that control is hard since we are using the brain to do many other things. What is interesting in the path of this experiment is not in fact the music generated but the use of sound to provide a way to hear the differences in datasets.

Hearing the difference between play Go or sleeping, between young people or old people.

Sound as a means of discovering patterns is a largely untapped source.

Building Clojure Services at Scale

At SoundCloud I’ve been experimenting over the last year with how we build the services that power a number of heavily loaded areas across our site. All these services have been built in Clojure with bits of Java tacked on the sides. Here are some of my personal thoughts on how to build Clojure services:

Netflix or Twitter

choose your own adventure

At some-point when you require a sufficient level of scaling you turn to the open source work of Twitter with Finagle or Netflix with Hystrix/RxJava. Netflix libs are written in Java while Twitters are written in Scala. Both are easy to use from any JVM based language but the Finagle route will bring in an extra dependency on Scala. I’ve heard little from people using interop between Clojure & Scala and that extra Scala dependency makes me nervous. Further I like the simplicity of Netflix’s libs and they have been putting a lot of effort into pushing support for many JVM based languages.

Hence with Clojure, Netflix projects are my preference. (I should add we do use Finagle with Scala at SoundCloud as well).

Failure, Monitoring & Composition Complexity

In a service reliant on other services, databases, caches any other external dependencies its a guarantee at some-point some of those will fail. When working with critical services we want to gracefully provide a degraded service.

While we can think about degrading gracefully in the case of failure, ultimately we want to fix wants broken as soon as possible. An eye into the runtime system allows us to monitor exactly whats going on and take appropriate action.

Your service needs to call other services. Dependent on those service results you might need to call other services which in turn might require calls to other services. Composing service calls is hard to get right without a tangle of complex code.

Fault Tolerance

How should we build fault tolerance into our Clojure services?

Single thread pool

Consider you have this line within a service response:

1
{:body @(future (client/get "http://soundcloud.com/blah/wah")) :status 200}

Now http://soundcloud.com/blah/wah goes down and those client requests start getting blocked on the request. In Clojure all future calls acquire a thread from the same thread pool. In our example the service is blocked up, is pilling new requests onto the blocked pool and we are in trouble.

My first solution to this problem was to introduce circuit breakers (https://github.com/josephwilk/circuit-breaker). I also stop using @ to dereference futures and used deref http://clojuredocs.org/clojure_core/clojure.core/deref which supports defaults and timeouts.

1
2
3
4
5
6
7
8
9
10
11
(defncircuitbreaker :blah-http {:timeout 30 :threshold 2})

(def future-timeout 1000)
(def timeout-value nil)

(defn http-get [url]
  (with-circuit-breaker :blah-http {
    :connected (fn [] (client/get "http://soundcloud.com/blah/wah"))
    :tripped (fn [] nil)}))

{:body (http-get http://www.soundcloud.com/blah/wah) :status 200}

Problem solved, now even though the thread pool may become blocked we back off the following requests and avoid pilling more work onto the blocked thread pool.

This worked pretty well, but then we decided we would to try and go even further in gracefully degrading. Why don’t we serve from a cache on failure, slightly stale data is better than none.

1
2
3
4
(defn http-get [url]
  (with-circuit-breaker :blah-http {
    :connected (fn [] (client/get "http://soundcloud.com/blah/wah"))
    :tripped (fn [] (memcache/get client url))}))

Now consider (client/get "http://soundcloud.com/blah/wah") starts failing, the thread pool gets blocked up, the circuit trigger flips and (memcache/get client url) is now fighting to get threads from the blocked thread pool.

Pants.

Scheduling over thread pools: Hystrix

Hystrix is Netflix library which I think of as circuit breakers on steroids.

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, 
services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems 
where failure is inevitable.

Dave Ray (http://darevay.com) has been doing lots of excellent work on producing Clojure bindings for Hystrix:

Hystrix gives me 2 big wins:

1. Separation of thread pools

Hystrix creates a separate thread pool for each Clojure namespace, if one thread pool becomes blocked due to a failure, a circuit breaker can be triggered with a fallback that uses a different thread pool.

This however does come with a cost:

  1. We have a performance hit due to moving to a scheduling based method for executing Hystrix commands.
  2. We cannot use Clojure’s concurrency primitives (futures/promises/agents).

Here is an example of our service rewritten with Hystrix:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(ns example
  (:require [[com.netflix.hystrix.core :as hystrix]]))

(hystrix/defcommand http-get
  {:hystrix/fallback-fn (fn [url] (memcache-get url)}
  [url]
  (client/get url))

(hystrix/defcommand memcache-get
  {:hystrix/fallback-fn (constantly nil)}
  [url]
  (memcache/get client key))

(defn http-get [url]
   {:body (http/get "http://soundcloud.com/blah/wah") :status 200})

Beautiful, Just adding the defcommand brings us fault tolerance with no other changes to the shape of our code.

See the Hystrix Clojure adapter for all the possible configuration: https://github.com/Netflix/Hystrix/tree/master/hystrix-contrib/hystrix-clj

2. Monitoring

Hystrix supports exposing metrics on all circuit breakers within a process. It exposes these calls through an event stream.

How you expose that Hystrix event stream depends slightly on which web server you are using with your Clojure app.

Netty and Hystrix Event Streams (without servlets)

https://github.com/josephwilk/hystrix-event-stream-clj

1
2
3
(:require [hystrix-event-stream-clj.core as hystrix-event])

(defroutes app (GET "/hystrix.stream" (hystrix-event/stream))

Jetty and Hystrix Event Streams (with servlets)

If they are using Jetty you will need to change your app to add your main web app as a servlet. Then we can also add the Hystrix event stream servlet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
(ns sc-clj-kit.hystrix.jetty
  (:import [com.netflix.hystrix.contrib.metrics.eventstream HystrixMetricsStreamServlet])
  (:import [org.eclipse.jetty.server Server])
  (:import [org.eclipse.jetty.servlet ServletContextHandler])
  (:import [org.eclipse.jetty.servlet ServletHolder])
  (:import [org.eclipse.jetty.server.bio SocketConnector])
  (:import [org.eclipse.jetty.server.ssl SslSocketConnector])

  (:import (org.eclipse.jetty.server Server Request)
           (org.eclipse.jetty.server.handler AbstractHandler)
           (org.eclipse.jetty.server.nio SelectChannelConnector)
           (org.eclipse.jetty.server.ssl SslSelectChannelConnector)
           (org.eclipse.jetty.util.thread QueuedThreadPool)
           (org.eclipse.jetty.util.ssl SslContextFactory)
           (javax.servlet.http HttpServletRequest HttpServletResponse))
  (:require
   [compojure.core          :refer :all]
   [ring.adapter.jetty      :as jetty]
   [ring.util.servlet :as servlet]))

(defn run-jetty-with-hystrix-stream [app options]
  (let [^Server server (#'jetty/create-server (dissoc options :configurator))
        ^QueuedThreadPool pool (QueuedThreadPool. ^Integer (options :max-threads 50))]
    (when (:daemon? options false) (.setDaemon pool true))
    (doto server (.setThreadPool pool))
    (when-let [configurator (:configurator options)]
      (configurator server))
    (let [hystrix-holder  (ServletHolder. HystrixMetricsStreamServlet)
          app-holder (ServletHolder. (servlet/servlet app))
          context (ServletContextHandler. server "/" ServletContextHandler/SESSIONS)]
      (.addServlet context hystrix-holder "/hystrix.stream")
      (.addServlet context app-holder "/"))
    (.start server)
    (when (:join? options true) (.join server))
    server))

(defroutes app (GET "/hello" [] {:status 200 :body "Hello"})

(run-jetty-with-hystrix app {:port http-port :join? false})

Aggregation and discovery

While you can monitor a single process using Hystrix in our example we have many processes serving an endpoint. To aggregate all these Hystrix metric services we use Turbine.

Physical endpoints for a service at SoundCloud are discovered using DNS lookup. We configured Turbine to use this method to auto discover which machines are serving an endpoint.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
(ns sc-turbine.discovery
  (:import [org.xbill.DNS Type]
           [com.netflix.turbine.discovery InstanceDiscovery Instance])
  (:require [clj-dns.core :refer :all]))

(gen-class
   :name ScInstanceDiscovery
   :implements [com.netflix.turbine.discovery.InstanceDiscovery])

(defn -getInstanceList [this]
  (map (fn [instance]
         (Instance. (str (:host instance) ":" (:port instance)) "example-prod" true))
       (map (fn [answer] {:host (-> answer .getTarget str) :port (.getPort answer)})
            (:answers (dns-lookup "" Type/SRV)))))

And the config.properties:

1
2
3
InstanceDiscovery.impl=ScInstanceDiscovery
turbine.aggregator.clusterConfig=example-prod
turbine.instanceUrlSuffix=/hystrix.stream

Putting this all together our monitoring looks like this:

Pretty graphs: Hystrix Dashboard

Finally we run the Hystrix Dashboard and watch our circuit breakers live.

And heres an example with triggered circuit breakers:

Since I cannot show you the dashboard running, you will have to make do with music generated from the metrics. I normalize the live Hystrix metrics to piano pitches and play the notes as the arrive from the stream.

Hystrix Metrics as Sounds

Complexity & Composition

Working with many services, composition of service calls becomes hard to think and write about. Callbacks try to solve this but nested callbacks leave us with a mess.

RxJava tries to solve this using the Reactive Functional model. While RxJava provides lots of features I see it primarily as a way of expressing concurrent actions as a directed graph which provides a single callback on success or failure. The graph is expressed in terms or maps/reduces/filters/etc, things we are familiar with in the functional world.

To separate the request thread from the response thread we use RxJava with Netty and Aleph.

Here is a very simple example firing 2 concurrent requests and then joining the results into a single map response:

1
2
3
4
5
6
7
8
9
;;Hystrix integrates with RxJava and can return Observables for use with Rx.
(defn- find-user-observable [id] (hystrix/observe #'find-user id))

(defn- meta-data [user-urn]
  (let [user-observable (-> (find-user-observable id) (.map (λ [user] ...)))
        meta-observable (-> (find-user-meta-observable id) (.map (λ [subscription] ...))))
    (-> (Observable/zip user-observable
                        meta-observable
                        (λ [& maps] {:user (apply merge maps)}))))

The function meta-data returns an Observerable which we subscribe to and using Aleph return the resulting JSON to a channel.

1
2
3
4
5
6
7
8
9
(defn- subscribe-request [channel request]
  (let [id (get-in request [:route-params :id])]
    (-> (meta-data id)
        (.subscribe
          #(enqueue channel {:status 200 :body %}))
          #(logging/exception %))))))

(defroutes app
  (GET "/users/:id" [id] (wrap-aleph-handler subscribe-request)))

The shape of the RxJava Clojure bindings are still under development.

That single thread pool again…

With RxJava we are also in a situation were we cannot use Clojure’s future. In order for RxJava to block optimally we don’t want to use a single thread pool. Hence we use Hystrix as our means of providing concurreny.

Clojure services at scale

I’m very happy with the shape of these services running at SoundCloud. In production they are performing very well under heavy load with useful near realtime monitoring. In part thanks to Netflix’s hard work there is no reason you cannot write elegant Clojure services at scale.

Creating Sampled Instruments With Overtone

Overtone is an open source audio environment which uses Clojure and SuperCollider (an environment and programming language for real time audio synthesis and algorithmic composition).

Overtone makes it very easy to define instruments based on sound samples from freesound. We will walk through getting samples for a flute and then using them to define the flute instrument in Overtone.

The Flute

Getting the samples

Firstly we need to manually grab all the IDs from freesound. Sounds are often grouped into packs making this a little less painful.

We will be using the “Transverse Flute: Tenuto Vibrato C4-C7” pack.

A little Ruby script I use to produce a map of freesound ids to notes (which are usually in the filename of the sample):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#!/usr/bin/env ruby -w
require 'rubygems'
require 'json'

raise Exception.new("You must pass in the pack id!") unless ARGV[0]

pack_id = ARGV[0]
note_extractor = λ{|filename| filename.gsub(/Transverse-Flute |\.wav|Tenuto|NonVibrato|-/, "").gsub(/ NonVibrato/, "")}

data = JSON.parse(`curl --silent "http://www.freesound.org/api/packs/#{pack_id}/sounds/?api_key=47efd585321048819a2328721507ee23"`)
number_of_pages = data["num_pages"] || 1

number_of_pages.times do |page|
  data = JSON.parse(`curl --silent "http://www.freesound.org/api/packs/#{pack_id}/sounds/?api_key=47efd585321048819a2328721507ee23&p=#{page + 1}"`)
  sound_ids = data["sounds"].map{|sound| sound["id"]}
  note_names = data["sounds"].map{|sound| note_extractor.call(sound["original_filename"])}

  sound_ids.each_with_index do |sound_id, index|
    puts "#{sound_id} :#{note_names[index]}"
  end
end

Adding these to our Clojure code we have a Freesound ID to note mapping:

1
2
3
4
5
6
7
8
(def FREESOUND-VIBRATO-FLUTE-SAMPLES
  "Freesound ids for all samples in the Vibrato Transverse Flute pack"
  {154274 :C7  154273 :B6 154272 :A#6 154271 :A6 154270 :G#6 154269 :G6  154268 :F#6
   154267 :F6  154266 :E6 154265 :D#6 154264 :D6 154263 :C#6 154262 :C6  154261 :B5
   154260 :A#5 154259 :A5 154258 :G#5 154257 :G5 154256 :F#5 154255 :F5  154254 :E5
   154253 :D#5 154252 :D5 154251 :C#5 154250 :C5 154249 :B4  154248 :A#4 154247 :A4
   154246 :G#4 154245 :G4 154244 :F#4 154243 :E4 154242 :F4  154241 :D#4 154240 :D4
   154239 :C#4 154238 :C4})

These samples are cached in the asset store (usually ~/.overtone/assets) so we don’t have to keep downloading them. We define a cache key for our downloaded samples.

1
2
3
4
(defn- registered-samples
  "Fetch flute samples from the asset store if they have been manually registered"
  []
  (registered-assets ::TransverseFluteTenutoVibrato))

Load all the samples into Overtone. This will automatically download the samples if they are not already present.

1
2
(def flute-samples
  (doall (map freesound-sample (keys FREESOUND-VIBRATO-FLUTE-SAMPLES))))

In order to play the samples we need to produce a mapping between the buffer that has the sound loaded and the midi pitch that buffer relates to.

First we convert a freesound-id into a midi note (pitch)

1
2
(defn- buffer->midi-note [buf]
  (-> buf :freesound-id FREESOUND-VIBRATO-FLUTE-SAMPLES name note))

Now we can generate the buffer id to midi note map.

1
2
3
4
5
6
7
8
9
(defn- note-index
  "Returns a map of midi-note values [0-127] to buffer ids."
  [buffers]
  (reduce (fn [index buf]
            (let [id (-> buf :id)
                  note (buffer->midi-note buf)]
              (assoc index note id)))
          {}
          buffers))

And we use this to set the buffers:

1
2
3
4
5
6
7
8
9
10
;Silent buffer used to ensure all 127 buffer spaces are filled 
(defonce ^:private silent-buffer (buffer 0))

(defonce index-buffer
  (let [tab (note-index flute-samples)
        buf (buffer 128)]
    (buffer-fill! buf (:id silent-buffer))
    (doseq [[idx val] tab]
      (buffer-set! buf idx val))
    buf))

Defining the Instrument

Now we have all our samples we define a instrument in Overtone using definst:

This allows us to specify a (huge) number of options on how our samples are played.

1
2
3
4
5
6
7
(definst sampled-flute-vibrato
  [note 60 level 1 rate 1 loop? 0 attack 0 decay 1 sustain 1 release 0.1 curve -4 gate 1]
  (let [buf (index:kr (:id index-buffer) note)
        env (env-gen (adsr attack decay sustain release level curve)
                     :gate gate
                     :action FREE)]
    (* env (scaled-play-buf 2 buf :level level :loop loop? :action FREE))))

There is quite a lot going on here. Lets focus on the configuration that effects how our sound samples are played.

Defining the Envelope Generator (env-gen).

An envelope generator (mapping to function env-gen) makes an audio signal that smoothly rises and falls. We are taking the recording of the real flute instrument as a digitized waveform, and then playing back its recordings at different speeds to produce different tones.

The most common form of envelope generator and the one we are using here is ADSR:

attack (A), decay (D), sustain (S) and release (R)

There are a number of other config settings in our example:

1
2
3
4
5
6
7
8
9
10
11
12
[
 note 60     ; Default pitch
 level 1     ; Volume
 rate 1      ; Playback rate 
 loop? 0     ; The note should loop after the first play
 attack 0    ; The way a sound is initiated. Fast attack (gunshot) vs Slow attack (closing a door slowly)
 decay 1     ; The time for notes to decay after the initial strike
 sustain 1   ; Once a sound has reached its peak, the length of time that the sound will sustain.
 release 0.1 ; The time for notes to decay after the key is released
 curve -4    ; Curve factor
 gate 1      ; note occurs when gate goes from nonpositive to positive. Note off occurs when it goes from positive to nonpositive
]

Look to the SuperCollider documentation on UGENs if you want to see all possible configuration options. Overtone is really a wrapper around the extremely powerful SuperCollider.

Grand finale

Finally we can start to play music in overtone using our flute:

1
2
3
(sampled-flute-vibrato 60) #=> pitch 60
(sampled-flute-vibrato 61) #=> pitch 61
(sampled-flute-vibrato 60) #=> pitch 60

Hear for yourself:

The Code in Full

Defining an instrument in Overtone:

Loading buffers
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
(ns overtone.samples.flute-vibrato
  (:use [overtone.core])
  (:require [clojure.string :as str]))

(defn- registered-samples
  "Fetch flute samples from the asset store if they have been manually
  registered"
  []
  (registered-assets ::TransverseFluteTenutoVibrato))

(def FREESOUND-VIBRATO-FLUTE-SAMPLES
  "Freesound ids for all samples in the Vibrato Transverse Flute pack"
  {154274 :C7  154273 :B6 154272 :A#6 154271 :A6 154270 :G#6 154269 :G6  154268 :F#6
   154267 :F6  154266 :E6 154265 :D#6 154264 :D6 154263 :C#6 154262 :C6  154261 :B5
   154260 :A#5 154259 :A5 154258 :G#5 154257 :G5 154256 :F#5 154255 :F5  154254 :E5
   154253 :D#5 154252 :D5 154251 :C#5 154250 :C5 154249 :B4  154248 :A#4 154247 :A4
   154246 :G#4 154245 :G4 154244 :F#4 154243 :E4 154242 :F4  154241 :D#4 154240 :D4
   154239 :C#4 154238 :C4})

(def FLUTE-SAMPLE-IDS (keys FREESOUND-VIBRATO-FLUTE-SAMPLES))

(def flute-samples
  (doall (map freesound-sample FLUTE-SAMPLE-IDS)))

(defn- buffer->midi-note [buf]
  (-> buf :freesound-id FREESOUND-VIBRATO-FLUTE-SAMPLES name note))

(defn- note-index
  "Returns a map of midi-note values [0-127] to buffer ids."
  [buffers]
  (reduce (fn [index buf]
            (let [id (-> buf :id)
                  note (buffer->midi-note buf)]
              (assoc index note id)))
          {}
          buffers))

;; Silent buffer used to fill in the gaps.
(defonce ^:private silent-buffer (buffer 0))

(defonce index-buffer
  (let [tab (note-index flute-samples)
        buf (buffer 128)]
    (buffer-fill! buf (:id silent-buffer))
    (doseq [[idx val] tab]
      (buffer-set! buf idx val))
    buf))
Defining the instrument
1
2
3
4
5
6
7
8
9
10
11
12
13
(ns overtone.inst.sampled-flute
  (:use [overtone.core])
  (:require
   [overtone.samples.flute-vibrato :as vibrato]))

(definst sampled-flute-vibrato
  [note 60 level 1 rate 1 loop? 0
   attack 0 decay 1 sustain 1 release 0.1 curve -4 gate 1]
  (let [buf (index:kr (:id vibrato/index-buffer) note)
        env (env-gen (adsr attack decay sustain release level curve)
                     :gate gate
                     :action FREE)]
    (* env (scaled-play-buf 2 buf :level level :loop loop? :action FREE))))

Sets in Elixir

I recently contributed the Set data structure to the Elixir programming language. The Set is implemented through a HashSet.

1
2
3
4
s1 = HashSet.new([1, 2, 3, 4])
s2 = HashSet.new([3, 4])

Set.difference(s1, s2) # => HashSet<[1, 2]>

Elixir’s Set implementation is often faster than Erlangs own sets. Which is pretty impressive since Elixir runs on the same Erlang VM. It does this by adapting the internal data structure based on the size of the sets.

Lets explore how Elixir does this while also explaining a little about Elixir:

Growing Data structures

For small sets the internal representation uses an ordered set (which really is just a list).

We refer to this as a Bucket.

1
[1, 2, 3, 4]

Being ordered allows us to perform faster access and modification.

For example finding a member of a set:

1
2
3
# Only needs a single comparison to know if 1 is in the Set.
# Since the first element is greater than 1 we know its not in the Set.
Set.member?(HashSet.new [2, 3, 4, 5], 1)

Order in the list is not maintained in the data structure but in the insertion/put:

Putting elements into a Bucket

Before we jump into some Elixir code its important to understand that the order of functions in Elixir is important. We start from the top function and if this function matches a criteria we call it otherwise we try the function below and so on. If we find no function that matches that’s a error and execution aborts.

Here is the Elixir code for inserting a member into a bucket.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# We have found the place to insert the new member.
defp bucket_put([m|_]=bucket, member) when m > member do 
  { [member|bucket], 1 }
end

# Member is already in the set, we don't added it. (notice how member is being used to pattern match here).
# Its the same as writing: `defp bucket_put([m|bucket], member) when m == member do
defp bucket_put([member|bucket], member) do
  { [member|bucket], 0 }
end

# Recursive case, keep calling bucket_put as this is not the point to insert.
defp bucket_put([e|bucket], member) do
  { rest, count } = bucket_put(bucket, member)
  { [e|rest], count }
end

# Empty set, insert the number
defp bucket_put([], member) do
  { [member], 1 }
end

Expanding into a Trie

Upon reaching a threshold the Set changes the internal representation into a Trie (What the hell is a trie).

Why a Trie?

For a Trie the time required for insert, delete and find operations is almost identical. As a result, for situations where code is inserting, deleting and finding in equal measure, tries can handily beat binary search trees.

In a Trie we don’t store any keys or hashes, instead a elements position in the tree defines the key with which it is associated.

Hence we can take advantage of Erlangs fast tuples to model the trie. A multi-depth trie is just tuples inside tuples.

Buckets growing into Tries

Lets work through an example. For simplicity we will assume the expansion happens at the very small size of 5 elements:

1
2
3
4
set = Set.new [0, 1, 2, 3]

# Apply an operation that forces the internal structure to change
set = Set.put(set, 4)

Create a new Erlang Tuple

bucket to trie

1
  :erlang.make_tuple(node_size = 4, [])

Redistribute the bucket elements into the Tuple

bucket to trie

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# This returns the index that a set value should have in the tuple.
defp bucket_nth_index(set_value, trie_depth) do
  node_shift = 2
  node_bitmap = 0b0011

  (:erlang.phash2(set_value) >>> (node_shift * trie_depth)) &&& node_bitmap
end

# Create a new node or add to an existing node.
# Based on the bucket_nth_index of the set member put it into a bucket at that index in the tuple.
defp node_relocate(node // :erlang.make_tuple(node_size = 4, []), bucket, depth) do
  :lists.foldl fn member, new_node ->
    position_in_tuple = bucket_nth_index(member, depth)
    set_elem(new_node, position_in_tuple, bucket_put(elem(new_node, position_in_tuple), member))
  end, new_node, bucket
end

Finally add the new set member into the Trie

bucket to trie

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Base case. We have found the right bucket, just insert the new member into it.
defp node_put(node, 0, member) do
  position = bucket_index(member)
  { new, count } = bucket_put(elem(node, position), member)
  { set_elem(node, position, new), count }
end

# Recursive case. Keeping trying to find the right bucket.
defp node_put(node, depth, member) do
  position = bucket_index(member)
  { new, count } = node_put(elem(node, position), depth - 1, bucket_next(hash), member)
  { set_elem(node, position, new), count }
end

# The bucket index of this member
defp bucket_index(member) do
  node_bitmap = 0b0011
  hash = :erlang.phash2(member)
  hash &&& node_bitmap
end

# The bucket index of this element with depth + 1.
defp bucket_next(member) do
  node_shift = 2
  hash = :erlang.phash2(member)
  hash >>> node_shift
end

Notice while we have a more complicated data structure with a trie the leaves are always plain old buckets. Hence insertion with a trie is a case of finding the right bucket and then just reusing our bucket_put function. Beautiful and fast :)

Tries expanding into deeper Tries

Things get a little more complicated with multi depth tries. If you are interested digging more into the implementation you can see all the source code of HashSet on Github.

Contraction

We can also remove elements from a set so in much the same as we expanded we have to contract the internal representation of the set.

Performance Benchmarks

Here is some sample data comparing Erlang sets to Elixir Sets. Smaller is better.

HashSet vs :sets

The benchmark scripts are on github (https://gist.github.com/josephwilk/5808525) if you are interested in running them yourself.

Isolating External Dependencies in Clojure

Isolating external dependencies helps make testing easier. We can focus on a specific unit of code and we can avoid slow tests calling real services or databases.

Clojure provides many different ways of achieving isolation. Lets explore what’s possible:

Redefining functions

We can redefine vars and hence functions in a limited scope with with-redefs

The documentation suggests its usefulness in testing:

Useful for mocking out functions during testing.

Lets look at an example where we want to isolate a function that logs to file:

1
2
3
(defn log-fn [] #(spit "report.xml" %))

(defn log [string] ((log-fn) string))

And the test removing the dependency on the filesytem:

1
2
(with-redefs [log-fn (fn [data] data)]
  (log "hello"))

Its important to note that with-redefs are visible in all threads and it does not play well with concurrency:

with-redefs can permanently change a var if applied concurrently:

1
2
3
4
5
6
(defn ten-sixty-six [] 1066)
(doall 
  (pmap #(with-redefs [ten-sixty-six (fn [] %)] (ten-sixty-six))
        (range 20 100)))

(ten-sixty-six) ; => 49 Ouch!

The parallel redefs conflict with each other when setting back the var to its original value.

Another option is alter-var-root which globally redefines a var and hence a function. alter-var-root can alter functions in other namespaces.

Writing our test to use alter-var-root:

1
2
3
4
(alter-var-root
 (var log-fn)
 (fn [real-fn] ; We are passed the function we are about to stub.
   (fn [data] (println data))))

Its important to note we have to reset the var if we want to restore the system to its previous state for other tests.

Redefining Dynamic Vars

Using dynamic vars we can rebind the value and hence we can use this as an injection point. Again if we can rebind vars we can rebind functions. Note though that we have to mark those functions as dynamic:

1
2
3
4
5
6
7
8
9
;The real http request
(defn ^:dynamic http-get-request [url] http/get url)
(defn get [url] (http-get-request [url]))

(defn fake-http-get [url] "{}")

(fact "make a http get request"
  (binding [http-get-request fake-http-get]
    (get "/some-resource")) => "{}"))

Unlike alter-var-root and with-redefs dynamic vars are bound at a thread-local level. So the stubbings would only be visible in that thread. Which makes this safe for tests being run concurrently!

Atoms & Refs (Global vars in disguise)

While insidious, evil, malicious and ugly we could use atom or refs to contain a dependency.

1
2
3
(def cache (atom (fn [method, & args] (apply (resolve (symbol (str "memcache/" method))) args))))

(defn get [key] (@cache get key))

And in our test:

1
(reset! cache (fn [method, & args] (apply (resolve (symbol (str "fake-cache/" method))) args)))

Yuck, lets never speak of that again.

Midje

The Midje testing framework provides stubbing methods through provided. In the core of Midje this uses our previously visited alter-var-root.

Lets see how our example would look using Midje:

The code:

1
2
3
(defn log-fn [] #(spit "report.xml" %))

(defn log [string] ((log-fn) string))

And our test that uses provided:

1
2
3
4
(fact "it should spit out strings"
  (log "hello") => "hello"
  (provided
    (log-fn) => (fn [data] data)))

Its important to note that the provided is scoped in effect. It is only active during the form before the Midje “=>” assertion arrow.

Conceptually think of it like this:

1
2
3
(provided
  (log-fn) => (fn [data] (println data))
  (captured-output (log "hello"))) => (contains "hello")

Flexibility

Midjes provided gives very fine grained control of when a stub is used:

1
2
3
4
5
6
(do
  (log "mad hatter")   ;will use the stub
  (log "white rabbit") ;will not use the stub
)
(provided
  (log "mad hatter") => (fn [data] (println data)))

And we can go further and define argument matcher functions giving huge flexibility in when a stub should be run.

Safety

Midje validates your stubs and checks your not doing anything too crazy which would fundamentally break everything.

Higher order functions

We can isolate dependencies by passing in functions which wrap that dependency. This abstracts the details of the dependency and provides a point where we can inject our own functions which bypasses the dependency.

For example:

1
2
3
(defn extract-urn [data]
  (let [urn-getter #(:urn data)]
 (do-it urn-getter)))

In our tests:

1
  (do-it (fn [] 10))

Simple and beautiful.

Substituting namespaces

We can switch the namespace that a block of functions are evaluated in. Hence we can swap in a completely new implementation (such as a fake) by changing the namespace.

An example faking out a key value store:

1
2
(defn get-it [arg & [namespace]]
  ((ns-resolve (or namespace *ns*) 'get) arg))

A fake implementation of this service:

1
2
3
4
5
6
7
8
9
(ns test.cache.fake)

(def cache (atom {}))

(defn get [arg]
  (@cache arg))

(defn put [arg value]
  (reset! @cache (assoc @cache arg value)))

And our test:

1
2
(fact "it should do something useful"
  (get-it "1234" 'test.cache.fake) => "1234")

Alternatively if we don’t want the mess of injecting new namespaces into our functions we could change namespace aliases to achieve the same effect:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(ns example
  (:require [cache.memcache :as memcache]))

(when (= (System/getenv "ENV") "TEST")
  (ns-unalias 'example 'memcache)
  (require 'test.cache.fake)
  (alias 'memcache 'test.cache.fake))

(defn get [arg]
  (memcache/get arg))

; ...

(when (= (System/getenv "ENV") "TEST")
  ;Cleanup our rebinding of memcache alias
  (ns-unalias 'memcache 'example))

When running our tests we mutate the behaviour of the system by setting the ENV environment variable to TEST.

Runtime Polymorphism

Switch behaviour based on a the type of an argument. During testing we can inject a specific type which will behave differently from the real system.

Clojure gives us protocols and multimethods to achieve this:

Protocols

1
2
3
4
5
6
7
8
9
10
11
12
13
(:require [[fake-service :as fake]
           [service      :as service]])

(defprotocol Service
  (do-get [this arg]))

(deftype FakeService []
  Service
  (do-get [this arg] (fake/do-get arg)))

(deftype RealService []
  Service
  (do-get [this arg] (service/do-get arg)))

And in our test:

1
(do-get (FakeService.) "cheshire")

Multimethods

Similar we can use the type of arguments to indicate different behaviour.

1
2
3
4
5
6
7
8
9
10
(:require [[fake-service :as fake]
           [service      :as service]])

(defmulti do-get (fn [service param] [(:Service service) param]))

(defmethod do-get [:FakeService] [service param]
  (fake/do-get param))

(defmethod do-get [:RealService] [service param]
  (service/do-get param))

And in our test:

1
(do-get (FakeService.) "rabbit")

Defining functions at Runtime

Using environment variables its possible to switch what functions are defined at runtime. def always defines a method at the top level of a namespace.

Here is an example inspired from Midje’s source code:

1
2
3
4
5
6
7
8
9
10
(defn init! []
  (case (System/getenv "ENV")
    "TEST"
    (do
      (def get [key]       (fake/get key))
      (def set [key value] (fake/set key value))
    ;; else
    (do
      (def get [key]       (memcache/get key))
      (def set [key value] (memcache/set key value))))))

We would run our tests with ENV=test lein test.

How should I isolate dependencies in Clojure?

Having explored what we can do, what should we do?

There are a number of choices and a lot depends on you’re programming and testing style:

The Purest form of isolation

Passing a function, functions that wrap our dependencies means we do not have to mutate the code under test. This is the ideal form of isolating. This is where we want to be.

But sometimes either aesthetics or control might make us look elsewhere.

Functions with many parameters can become ugly and cumbersome to use.

Using external libraries where we cannot have design the way we want it (though we can try by wrapping the heck out of any library).

Finally integration tests are hard if not impossible to do with this form of dependency isolation.

The Aesthetic small touch form of isolation

var-alter-root is (very) scary, but the guard rails of Midje make it an easy way to isolate dependencies. It also supports flexibility in how we stub functions based on the arguments they are called with (or completely ignore the arguments). This flexibility is extremely powerful and is a big plus for Midjes provided.

The danger ever present with this form of isolation is ignoring your tests telling you about a problem in your design.

The Simple small touch form of isolation

While Midje provides lots of power and flexibly it does so at the cost of slightly awkward syntax and a lot of crazy macros (I say this having stared into the heart of Midje). For example parallel functions do not work with provided.

with-redefs, binding and var-alter-root provide flexibly to handle different testing scenarios. and no prior knowledge of an external tool is required.

If you don’t need the power of Midje or fear its complexity you can happily use nothing but Clojure’s standard library. Maybe you will even learn something about how Clojure works!

The Java small touch form of isolation.

Since Clojure supports java interop its always possible to fall back to using Java, OO and dependency injection. If you love Java, this is a good path.

The Crazy Large touch form of isolation

Namespace switching is a shortcut to having to stub out every single method. In one sweep we redefine all the functions of a namespace. This might be more useful for integration tests than unit tests.

That shortcut does come at a cost, we still have to maintain our fake ns every time something changes in the real namespace and our production code is left wrapped in ns-resolve or a ugly switch based on Environment settings. Ugly!

I don’t recommend using this form of isolation regularly but in edge cases it can be very convenient, though people will still think you are crazy.

A Developers Guide to Creating Presentations

So your talk got selected. Great!

Oh Crap! Now hits the panic, you have to actually create a presentation.

Every presenter wants to give the best possible presentation they can that sticks in peoples minds.

Here are some hard earned lessons for getting the best out of your presentation.

Accept that most of our preconceptions of how we learn are wrong.

Forget those high information dense, black and white acetate slides with professors droning on about solving the towers of Hanoi while you scribble your own notes while downing your fifth coffee.

“Attending a lecture is a passive experience for the student. Of all teaching events, the lecture is most likely to promote basic assumption dependence and sleep”

http://www.dailyprincetonian.com/2009/10/15/24142/

The good news

That does not mean you cannot give a super dense, deeply detailed presentation. It means if you want people to get the most out of your presentation you need to think outside a boring lecture. Luckily Fun, humor, creativity, color, interaction, invoking emotions are a key part of helping people learn effectively. Think back to your favourite teacher, why were they your favourite? I’m guessing they introduced some of those things in their teaching.

Creating a presentation

Lets look at some general ideas to help make your presentation great:

1. Dedicate time

Creating a presentation, content, theme, styling, finding pictures, practicing all take time. As you get better you can speed up and hone your tactics for presentations.

My first presentation took 3 months. Make sure you start early and give yourself enough time!

2. Research the current state of the art

Doing a talk on Continuous Deployment? Search for every Continuous Deployment presentation that has been given. Watch them all, steal the good ideas and throw away the bad. Try and make sure you are saying something new or presenting a different slant on the topic.

3. Don’t be on your own

Experienced speakers have developed a repertoire of little tricks, ideas and thoughts for presentations. They have also sat through a lot of presentations and have good experience of being the audience. Don’t be shy finding someone who has spoken before for some advice, even finding someone who would be happy to be your mentor.

If nothing else watch some of the most popular speakers on Confreaks and see what tricks/styles they use.

Conferences are not very good at helping get the best out of you when it comes to creating your talk. Once your talk is submitted and accepted you are often left on your own until the day of your talk. I believe this is something fundamentally broken with conferences today. Realise this and find someone to be your mentor.

4. Practice frees the subconscious

Practicing a presentation is an important part of getting better and more confident at giving presentations. Practice wherever and whenever you can, at work, at your local meetup and in the shower (yes I’ve done this). Don’t forget you are not learning if you don’t get feedback from your audience.

There is however another secret reason to practice.

When you know a presentation and are confident in giving it you become freer to improvise, to play to the crowd, to react to the room. Your mind is no longer concerned with messing it up, its free to improvise and be creative. Understand what I mean? Watch any of Jim Weirich talks.

5. Borrow confidence from the Samurais

The hard truth is its easier to pay attention to someone who seems confident about what they are saying than someone who is very nervous. There is an old and secret trick I learnt to calm nerves before a presentation:

“When faced with a crisis, if one puts some spittle on his earlobe and exhales deeply through his nose, he will overcome anything at hand. This is a secret matter.”

Bushido: The Way of the Samurai

While it sounds a little silly, I found this trick to be very useful. It gives you something to focus on to calm yourself, a ritual with which you find a sense of comfort and deeply breathing is a well know way of increasing oxygen to reduce stress.

6. Talk to your audience

If you want to engage your audience you need to talk to them, not your notes or your laptop. Look around the audience as you speak. This engenders engagement with the whole audience, eye contact draws us into a conversation and draws the audience into your story.

Going further always try and remove any obstacles between you and the audience like those pesky lecterns or table. You want the audience to connect with you and associate you as one of them. This can help encourage people to listen to what you have to say. Barriers create a separation between you and the audience, and while that can be overcome why add the challenge in the first place.

7. Time

If you have a 40 minute slot for talking, you do not have to fill it. In fact most peoples attention span dips massively towards that figure. Personally I believe 30 minutes is the sweet spot for a presentation. Though you should not feel pressured to pad your talk. Use what time you need to get your point across in the most concise way you can.

8. The final curtain, end you presentation

The end of the presentation is one of the most powerful moments to set the mood of the room and help fire the discussion in the after talk chit chat. You have done all the hard work to get here, don’t wimper out with a quiet “thats all” or a jarring quick finish.

You want too leave a short summary of the ideas of your presentation, big questions the audience can discuss afterwards. The best presentations leave the room alive with discussions.

Creating your slides

Now lets take a deeper look at some ideas to help you create great slides:

1. Learn your tools

You know your developers tools right? Spend any time practicing and getting sharper with your IDE?

If you want to make great presentations you have to invest time in mastering your presentation tools.

Have you ever watched the Keynote tutorials? How about spending an hour playing with animations and find out whats possible. Why not look and see what is possible with presentation tools from watching other peoples talks.

2. Shape first, design later

Before getting too caught up in making your slides beautiful and full of pictures of cats, first think out the flow, order and shape of your presentation. A good presentation has a natural flow where each slide leads into the other.

I usually start first by brain storming all the ideas I have about a presentation on sticky notes. No need for each sticky to contain exact details, just words, ideas or thoughts. Leaving a mass of mess on the wall.

Then on another wall try and extract a story putting the stickies into a beginning, middle and end

3. Deviate from the default

Keynote and Powerpoint provide lots of nice default (boring) themes. Default themes are good way to knock something together quickly. But if you use a default theme its hard to stand out from the crowd.

Do you want your presentation to stand out and be memorable? Do you want people leaving your presentation remembering the funny use of star wars Lego characters.

4. Experiment with design

Don’t be afraid to play around with the design of your slides. Experiment lots until you start to like what you have. I spend the most time on the first slide and I usually have hundreds of different designs until I’m happy.

5.  Pick an original theme

Pick a theme/style for your presentation. Try and choose a theme that has sufficient material so you can find lots of pictures. Look to other peoples presentations for ideas.

Examples:

Street art

Tv Programs / Mad Men

Silent Movies

6. Kill them with your first slide

Your very first slide will probably be looked at longer than any of your other slides. You want to immediately capture the audiences attention and excite them. Show them how you are going to tell them the story of your presentation, introducing your theme.

Which of these first slides would most engage you?

7. Fonts make the theme

Default fonts can work in a presentation if you have little text on the slide or you have very powerful images/colours. If you want to make your presentation standout try different fonts. There are thousands of readable fonts free to download. Explore which ones work for your presentation and your theme.

For example this slide uses a font which fits very well with its DIY/tools theme:

Always remember to pick a font that does not compromise the readability of your content. Use that font consistently throughout your whole presentation.

8. Live and die by your theme

Maintain consistency of your theme throughout. The more daring and original the theme the harder it can be to find media. With great risk can come great reward.

9. Invest time or money into images

There are two paths to great images that help make your presentation stand out.

Buy images

On average I spend £100 per presentation on buying images (Mostly on http://www.istockphoto.com). That is one of my key secrets for standing out in a presentation. Spending money buys you great, high resolution and original images.

Hunt or create images

There are lots of free sources of images on the Internet such as flickr. It can take more time to hunt around and find the right images but its possible. Or create your own images to give your presentation that personal, unique touch.

10. Minimalism is king with content

When we first start a presentation we often overfill the slides with content. It makes sense as we are thinking out what we are going to talk about.  As you practice your presentation, slowly that content sinks into your brain (or your notes) allowing you to ween out as much content as possible from the slides. Leaving the minimal possible text on the slide. The audience is left listening to you rather than trying to read overly detailed, complex slides.

Anti-patterns of presentations

There are some common anti-patterns in presentations. Lets look at some with examples:

1. Bullet points of death

You can get away with a few of these slides mixed into an interesting presentation. Rely on them too much and it gets boring quick.

Do you want to keep my attention?

2. Breaking continuity of images, text and content

One of the hardest challenges of adding images to your presentation is ensuring they feel part of the presentation, fitting with your content and style.

Lets look at an example where images do not fit in the presentation:

Now improve that slide by making the image feel less jarring with the rest of the content.

Using framing or picture frames is a easy way to help a image fit in a presentation.

Without a frame:

With a frame:

3. Inconsistent design

If you want to make your presentation beautiful having consistency in design is important.

Consider how the diagrams in this presentation break consistency.

We have non-shaded, hard edged blocks of colour in one diagram.

Then in following diagrams we have round blocks with shading.

4. The death of colour

Colour stimulates my brain. Do you want to stimulate it or send it to sleep?

5. Live code demo fail

You have 30ish minutes of my time. I don’t want to spend that time watching you make typos and debugging an error. You immediately make me lose my concentration, I’ll start flicking through my twitter feed. You have been working hard through your presentation to engage me, why throw it away?

Either practice a heck of a lot or pre-record your demos. I personally use Screenflick for all my recorded demos.

6. Black and white code.

How often do you read source code without syntax highlighting?

For me, that’s never. Highlighting code is also a great excuse to add some colour and life to your slides.

Its not tricky, for example extracting syntax highlighting from Textmate to Keynote.

7. Hello my name is

The slides where you tell me who you are, what company you work for, what open source projects you work on, your cats name, your dogs name, and what you ate for breakfast.

Earn me wanting to know who you are through giving a stimulating presentation.

Content over character.

Avoid breaking the flow of the presentation with a slide which has nothing to do with your topic or content.

Final words

Creating a presentation is hard work. Creating a great presentation is lots and lots of work and even then you are never sure if your audience will think its great. Public speaking is stressful and takes a lot of concentration and confidence.

No matter what happens with your talk, be proud of yourself. You decided to put yourself out there, up on stage trying to explain your ideas to people. This is already a great achievement. Don’t be too dishearten or critical of yourself if you are not happy with your talk.

Take a moment in that euphoric buzz of the applause to enjoy yourself.

Then work out how you can do better!

Good luck!

The Aesthetics of Density

Programming languages can be described as Dense.

What does dense mean?

Closely compacted in substance. Having the constituent parts crowded closely together:

What does it mean for a programming language to be dense?

I consider there are two axis for the density of programming languages:

Density of syntax The syntax is very dense/compact.

For example Assembly has a very dense syntax, abbreviate commands, small register names, etc… It takes a lot to express simple expressions

A simple “for” loop in Assembly

1
2
3
4
5
6
7
8
9
10
11
12
13
mov cx, 3
startloop:
   cmp cx, 0
   jz endofloop
   push cx
loopy:
   Call ClrScr
   pop cx
   dec cx
   jmp startloop
endofloop:
   ; Loop ended
   ; Do what ever you have to do here

Density of expression The means of expressing simple concepts or solutions is very compact.

This is a little fuzzier than syntax, it can depend on what you are trying to express and languages often provide many different ways to express something. For example string processing in Erlang is a lot messier than say, Ruby. Paul Graham measures this form of density by the number of elements

As an example PROLOG scores highly in expressive density. One of the main reasons why is when you give up control of execution (imperative style) and describe the problem (declarative style) you increase the expressive density.

The towers of hanoi in Prolog:

1
2
3
4
5
6
7
8
9
10
11
12
move(1,X,Y,_) :-
    write('Move top disk from '),
    write(X),
    write(' to '),
    write(Y),
    nl.
move(N,X,Y,Z) :-
    N>1,
    M is N-1,
    move(M,X,Z,Y),
    move(1,X,Y,_),
    move(M,Z,Y,X).

Programming languages fit along a spectrum within these forms of density. Ruby provides the means to express concepts very syntactically densely. Just look at Ruby Golf (solving a problem with the smallest possible number of characters) for example:

1
2
3
def fizzbuzz(n)
n%3<1&&f="Fizz";n%5<1?"#{f}Buzz":f||n.to_s
end

It is also always posible to build a DSL within a programming language to maximise density.

Where does Density fit with Literate Programming?

Dense syntax moves code away from being an easily accessible form of documentation.

Density of expression can move code away from being easily accessible as documentation. For example do you understand how that PROLOG towers of hanoi works?

The more focused a language is on expressive/syntactical density the further it moves the art of programming away from Literate programming where we focus on our code being the documentation. Much like writing an essay:

Instead of writing code containing documentation, the literate programmer writes documentation containing code.

Ross Williams. FunnelWeb Tutorial Manual, pg 4.

The readability of the code to humans is the priority.

Under the literate programming paradigm, the central activity of programming becomes that of conveying meaning to other intelligent beings rather than merely convincing the computer to behave in a particular way.

Ross Williams. FunnelWeb Tutorial Manual, pg 4.

Density within our heads

One could argue that dense code can still be literate in style. Its just that you have to fit all the programming languages syntax into your head. Its not unrealistic to ask developers to know the syntax/api of their language. Though holding it all in memory when its particularly dense can be challenging.

If your a Clojure programmer you might have a good understanding of this code as documentation:

1
2
3
(def ^{:dynamic true
       :doc "some doc here"}
     *allow-default-prerequisites* false)

And if you’re a Ruby or Perl programming you might read this with ease:

1
$!.is_a?(MonkeyError)

Can dense languages be a good idea?

“The quantity of meaning compressed into a small space by algebraic signs, is another circumstance that facilitates the reasonings we are accustomed to carry on by their aid.”

  • Charles Babbage, quoted in Iverson’s Turing Award Lecture

Is there a trade-off in moving to a more dense form of expression in helping shape the way we think and the kind of thoughts we have?

How easy is it to hold a dense language in our heads, remembering all the syntax in order to easily read code?

Regular Expressions

While regular expressions are not a programming language they are one of the best examples of a very dense language both syntactically and expressively  that has persisted in its syntax through many programming languages.

Is that a sign that regular expressions have succeeded in encoding pattern matching text?

Write Once

Do you understand this pattern?

1
/^[\w]$/

How about we push the complexity level and try some of the more esoteric symbols in regular expressions:

Do you know what this does?

1
/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b/i

How about this?

Full email detection regular expression (RFC822)

While regular expressions are very well suited to small patterns, with a very dense language our ability to parse complex statements is reduced.

Which has a knock on effect for maintenance, its read-only and even then its not easy to read.

Readability

In fact its considered bad practice to write a regular expression of the form above. Its understood that its hard to read and hence programmers have to add to the dense language to increase readability:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
/
^                                             # start of string
(                                             # first group start
  (?:
    (?:[^?+*{}()[\]\\|]+                      # literals and ^, $
     | \\.                                    # escaped characters
     | \[ (?: \^?\\. | \^[^\\] | [^\\^] )     # character classes
          (?: [^\]\\]+ | \\. )* \]
     | \( (?:\?[:=!]|\?<[=!]|\?>)? (?1)?? \)  # parenthesis, with recursive content
     | \(\? (?:R|[+-]?\d+) \)                 # recursive matching
     )
    (?: (?:[?+*]|\{\d+(?:,\d*)?\}) [?+]? )?   # quantifiers
  | \|                                        # alternative
  )*                                          # repeat content
)                                             # end first group
$                                             # end of string
/

This is definitely not literate programming, comments and code are clearly separate things.

Named captures groups are also an optional feature to improve and document the readability.

1
2
3
4
5
6
7
8
9
10
user_regexp = %r{
   (?<username> [a-z]+ ){0}

   (?<ip_number> [0-9]{1,3} ){0}
   (?<ip_address> (\g<ip_number>\.){3}\g<ip_number> ){0}

   (?<admin> true | false ){0}

   \g<username>:\g<ip_address>:\g<admin>
 }x

Mnemonics

Our memory also struggles to find mnemonics or associations to remember the full vocabary of regexps:

1
2
3
4
5
6
7
8
9
#Some easy ones
/w #word
/s #space

#Harder ones
(?<!pat)
(?<=pat)
(?!pat)
(?=pat)

Reducing the Density of Regular Expressions

Creating a DSL for parsing text is a huge domain. The power of regular expressions is very clear.

Yet there have been attempts in various languages to move regular expressions towards a more verbose form to improve readability.

Regexp::English

The Perl community has attempted to provide a more English, verbose syntax for regular expressions:

Regexp::English provides an alternate regular expression syntax, one that is slightly more verbose than the standard mechanisms

Lets look at an example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
        use Regexp::English;

        my $re = Regexp::English
                -> start_of_line
                -> literal('Flippers')
                -> literal(':')
                -> optional
                        -> whitespace_char
                -> end
                -> remember
                        -> multiple
                                -> digit;

        while (<input />) {
                if (my $match = $re->match($_)) {
                        print "$match\n";
                }
        }

Better?

Loving the Density of Regular Expressions

Clearly there has been some recognition among developers that regexp could be improved by being more verbose. Its interesting that these attempts are considered failures. It would imply the majority of developers prefer dense regexps.

“you can document them with comments, named capture groups, composing them from well-named variables. of course, no one does those things.” Tom Stuart

In the Perl community some people have given up completely on the humans and their dense, hard to maintain regular expressions. They create tools to decode the density automatically:

1
2
3
4
use YAPE::Regex::Explain;

print YAPE::Regex::Explain->new(qr/this.*(?:that)?(?!another)/)
      ->explain;

Outputs:

The regular expression:

(?-imsx:this.*(?:that)?(?!another))

matches as follows:

NODE                     EXPLANATION
----------------------------------------------------------------------
(?-imsx:                 group, but do not capture (case-sensitive)
                         (with ^ and $ matching normally) (with . not
                         matching \n) (matching whitespace and #
                         normally):
----------------------------------------------------------------------
  this                     'this'
----------------------------------------------------------------------
  .*                       any character except \n (0 or more times
                           (matching the most amount possible))
----------------------------------------------------------------------
  (?:                      group, but do not capture (optional
                           (matching the most amount possible)):
----------------------------------------------------------------------
    that                     'that'
----------------------------------------------------------------------
  )?                       end of grouping
----------------------------------------------------------------------
  (?!                      look ahead to see if there is not:
----------------------------------------------------------------------
    another                  'another'
----------------------------------------------------------------------
  )                        end of look-ahead
----------------------------------------------------------------------
)                        end of grouping
----------------------------------------------------------------------

Some snippets on developers thoughts about regular expressions:

“Love them, probably because they’re a form of arcane magic and they make me feel special for being able to control them”

“I think they are geek candy … sometimes used to show off maximum geekness”

“Love them because they make me look clever and LIKE A H4XX0rrrrr!”

The Aesthetics of Density

Regular expressions have succeeded and they are one of the few very dense languages to have done so.

The density of regular expressions make the initial barrier to getting started and moving to expert high. They are far from what we imagine in a literate programming style, yet once you have the syntax in your head, movement becomes fluid, you think in regular expressions. Dense languages with a very limited common syntax set allow experimenting rapidly. Without practice dense languages quickly drop from your mind and you struggle to fit the problem into the right expressive form.

There is an undeniable beauty in the density of regular expressions, in both syntax and expression.

Finding prime numbers using a single Regexp:

1
/^1?$|^(11+?)\1+$/

It’s also drink absinthe and cut off your own ear crazy.

I wrote this PROLOG code for my thesis. I have no idea how it works now and it would take me about a month of playing with it to get back to a state where the dense language was back in my head and I could express ideas in the PROLOG way.

I spent over a month adding nothing more than a single “!” mark in the code.

1
2
3
4
5
6
7
abdemo_holds_ats([holds_at(F,T)|Gs],R1,R3,N1,N3,D) :-
     !,
     abdemo([holds_at(F,T)],R1,R2,N1,N2,D),

     %cut added Joseph Wilk 16/03/2004
     !,
     abdemo_holds_ats(Gs,R2,R3,N2,N3,D).

I still feel its some of the most beautiful code I’ve written. Why?

I revel in the expressive density. Bending my brain to express my thoughts in the densely expressive PROLOG way.

I guiltily dip into the syntactical density because it’s like the detailing on the brush strokes of a painting.

Would I ever write this code in a production system that developers would have to maintain? Hell no.

Would I consider this literate programming? Hell no. Just look at the 100 of lines of comments.

But for the sake of art and realising a form of flow I’ve not encountered since, I would happily revel in the aesthetics of density.

Michael Wolf “Architecture of Density no.36”: http://www.flickr.com/photos/worldeconomicforum/6751247749/

Why Are You SHOUTING Programmer?

Being shouted at is not a lot of fun. So why do we shout in code?

Shouting code

Compare

"how_many_monkeys_can_a_monkey_eat_before_it_explodes"

and:

"HOW_MANY_MONKEYS_CAN_A_MONKEY_EAT_BEFORE_IT_EXPLODES"

How do you read those differently in your head?

We associate capital letters with someone shouting.

Now lets turn to two functionally equivalent pieces of Ruby code:

1
2
3
4
5
6
7
8
9
class Monkey
  def capacity
    10
  end

  def stomach
      "I can fit #{capacity} monkeys in my belly"
  end
end

And

1
2
3
4
5
6
7
class Monkey
  CAPACITY = 10

  def stomach
    "I can fit #{CAPACITY} monkeys in my belly"
  end
end

Constants are shouted. Why do we shout? Because:

  • We are angry

  • We have something we think is important and we want everyone to hear it.

Why in Ruby are constants uppercase? Well they don’t have to be, Ruby constrains us to ensuring the first letter is a capital. We get warnings if we try and reassign their value but ultimately they are just Fixnums. In order to stick with our Ruby naming convention we use ‘_’ and all caps.

So its a combination or telling the compiler that this value is a constant and fitting with the naming scheme in Ruby.

We shout because society indicates to us thats the normal behaviour and we all want to be nice citizens of the Ruby republic.

As a side effect constants feel like they are more important than the other variables or methods. They should take our attention first.

Reading uppercase is slow

What wait a minute isn’t uppercase text harder to read? There is evidence [1] to show that all-caps is less legible and less readable than lower case. So constants are harder for us to read.

lowercase permits reading by word units, while all capitals tend to be read letter by letter

Numbers, letters and uppercase

A common use of shouting case is constants used to remove magic numbers from calculations.

1000 * 667895 / LIMIT + 475436

The brain recognises numbers and letters very differently. The brain in general can recognise words faster than a sequence of digits since with a word we do not need to read each character in order to recognise the word.

To see for yourself try and read the following:

The hmaun mind does not raed eervy letter by istlef but the word as aa woelh.

Compare how much more time it takes you to read the numbers.

124 3456 3234 5443 3342 55334 66554 47567

By uppercasing the constant we are slowing down this natural ability to read words.

Compare again:

1000 * 667895 / LIMIT + 475436

With:

1000 * 667895 / limit + 475436

Do we really need this further uppercase difference? With the instinctive separation between words and letters the further effort and cognitive slow down has little value.

Immutability vs Mutability

A legitimate case were it does become useful to use shouting case is where you want to distinguish between variables/functions and constants. Expressing that a value is immutable in comparison to a mutable variable is important in a language like Ruby were immutability is not the norm.

1000 * 667895 + scale / LIMIT + 475436 - radius

Shouting in technicolor

Editors often provide color markup for words in uppercase. For example for Ruby in Textmate:

Though they also provide the difference between variables/function and numbers. Shouting provides us with a discernible way to see all constants within the code at glance based on colour.

History of shouting in code

How or where did this convention of uppercasing constants come from? Why is it a convention? When did we start shouting in our code?

What made us so mad?

Assembly

It started with assembly, the convention was to uppercase variables names and lowercase instructions.

ADCTL  = 0x30
staa ADCTL,X

Variable names were limited in length, so often variable names were acronyms, which in English are often capitalised (we just skipped the dot). Registers, memory & caches all had nice acronyms which you could reference in your assembly.

Assembly was more machine centric than human centric. Not quite shouting as we know it now (though its understandable we were angry with all that ugly code).

FLOW-MATIC

The birth of programming in English. It was not a pretty birth, this thing was born shouting very loudly. EVERYTHING is in capitals, even the programming languages name!

INPUT  INVENTORY FILE=A
 PRICE FILE=B,
 OUTPUT PRICED-INV FILE=C
 UNPRICED-INV FILE=D,
 HSP D.

Flow-Matic had the builtin in constant ZERO. Our first example of a Constant but where everything is capitalised so it is not distinguished from other code.

FORTRAN

FORTRAN was a confused language when it came to shouting. The use of lowercase letters in keywords was strictly nonstandard.

IF (IA+IC-IB) 777,777,705
IF (IB+IC-IA) 777,777,799
STOP 1
C USING HERON'S FORMULA WE CALCULATE THE
C AREA OF THE TRIANGLE
S = FLOATF (IA + IB + IC) / 2.0
AREA = SQRT( S * (S - FLOATF(IA)) * (S - FLOATF(IB)) + (S - FLOATF(IC)))
WRITE OUTPUT TAPE 6, 601, IA, IB,
STOP
END

But then the liberation came and after a bloody battle FORTRAN was renamed Fortran.

In this new post revolution age Fortran’s compiler was a liberal one, not caring about shouting or case at all.

program helloworld
     print *, "Hello, world."
end program helloworld

The society on the other hand was still very keen to tell its citizens when they should shout. It was a social coding convention that local variables be in lowercase and language keywords be in uppercase.

Language keywords were more important and hence shouted. Far more important than those pesky human named local variables. This inverted the previous Assembly conventions on the use of case.

LISP

Common Lisp is case sensitive but the Common Lisp reader converts all uppercase to lower case:

(defun hi () "Hi!)
(hi) ;; outputs "Hi"
(HI) ;; outputs "Hi"
(Hi) ;; outputs "Hi"

LISP had a social convention to only use lowercase (one might think to avoid confusing situations like the one above). Did LISPeans shout at all? They did when it came to documentation strings:

“In a documentation string, when you need to refer to function arguments, names of classes, or other lisp objects, write these names in all uppercase, so that they are easy to find”

This was to help humans easily find them and because documentation generation tools could detect them.

Shouting the references in unstructured text made them clearly visible to both machines and humans.

COBOL

COBOL is another of those shouting languages which liked everything in uppercase. Which makes reading a COBOL program akin to having someone shout very loudly in your face. Until you cry. Lots.

01 RECORD-NAME.
02 DATA-NAME-1-ALPHA PIC X(2).
02 DATA-NAME-2.
03 DATA-NAME-3-NUMERIC PIC 99.
03 DATA-NAME-4.
04 DATA-NAME-5-ALPHA PIC X(2).
04 DATA-NAME-6-NUMERIC PIC 9(5).
02 DATA-NAME-7-ALPHA PIC X(6).

The only thing that was not upper case was comments.

It helps if all comments are in lower-case, to differentiate from actual commands which should always be in upper-case

Comments where not important, so no need to shout them. Which in turn makes them easier to read. Perhaps there is an understanding here that shouting makes code hard to read. Comments which might contain a lot of text should also be easier to read.

If you were still in doubt about COBOL’s evilness: user defined constants were distinguished by using a single character variable name. MAD. YES THAT WAS WORTH SHOUTING.

Basic

In basic keywords were capitalised to distinguish between variables names. The case is insignificant, it’s for the humans not the compiler.

LET m = 2
LET a = 4
LET force = m*a
PRINT force
END

Keys words are important, so shout them. But in turn make it easier to read the user named variables by leaving them lower case.

C

C uses uppercase by convention for object-like Macros which get replaced during pre-processing.

#define BUFFER_SIZE 1024
foo = (char *) malloc (BUFFER_SIZE);

Uppercase is used to defined a templating language within C, which we can quickly distinguish from the code. It also makes the job of the pre-processor easier, parsing macros.

So is shouting a bad thing?

When it comes to expressing ideas in nothing but text we use everything we can to provide structure and separation to help improve clarity. Shouting or uppercasing words provides a very powerful way of rapidly distinguishing certain aspects of text.

How programming languages spend this limited currency of instinctive separation reflects the languages understanding of readability (i’m looking at you COBOL) and what they find to be important enough to earn shouting case.

However most modern languages provide you with the choice of shouting. In Ruby we can skip it all together.

Try not shouting for a while. See how it makes you feel.

And always:

TRY AND AVOID SHOUTING FOR TOO LONG AS IT IS HARD TO READ.

Keep it sharp, short and loud.

References

[1] Type and Layout: How Topography and Design Can Get Your Message Across – Or get in the Way