Joseph Wilk

Joseph Wilk

Things with code, creativity and computation.

Visuals With Overtone and Shadertone

Exploring techniques for creating live coded performances with Overtone and OpenGL Fragment Shaders. Much learnt from my work performing as Repl Electric. All my shader source code for these performances is open: https://github.com/repl-electric/cassiopeia/tree/master/resources/shaders

shaders

To bring OpenGl to Clojure I use Shadertone written by rogerallen. This utilises LWJGL (Java Light Weight Java Game Library https://www.lwjgl.org).

The Bridge between Clojure and Shaders

A vital feature of Shadertone is a map between Clojure atoms and shader Uniforms. What is a shader Uniform? Well think of it as a read-only global variable in your shader. A Clojure watcher ensures any updates to your Clojure atom persist into your Uniform. A little clunky but all uniforms start with the letter i.

The shader:

1
uniform float iExample;

And in Clojure

1
2
3
4
5
6
(def example-weight (atom 0.5))
(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data {"iExample" example-weight})

;;iExample Uniform will also be updated.
(reset! example-weight 0.2)

Live editing shaders

When a shader file is edited Shadertone is watching the file (using watchtower) and will reload/recompile the changed file. This results in a slight freeze as the new code is run (This might be down to my graphics card). Hence most of the time I prefer alternatives to live editing the shader to create smoother transitions.

Injecting movement

To make static images move we need a continuously changing value.

Shadertone gives us iGlobalTime using the number of seconds since the shader was started:

1
2
3
4
5
6
uniform iGlobalTime

void main(void){
  //Use the continuously changing time signal as the value for a color.  
  gl_FragColor = vec4(sin(iGlobalTime)*0.5+0.5);
}

Putting a continuously changing value through a function like sin/cos is the bread and butter of creating animations with shaders.

Randomness

We often need a cheap and fast way to generate random floats in Shaders. Without persistent state and preservation of a seed it can be difficult. One solution is to use a noise image and the current pixel coordinates as an index into the image for a float value.

Shadertone supports loading custom textures into your shaders:

1
2
(shadetone/start "shaders/example.glsl"
         :textures ["/josephwilk/textures/noise.png"])

The noise texture:

And finally the shader:

1
2
3
//Turning the current pixel coordinates (uv) into a random float. 
vec2 uv = gl_FragCoord.xy / iResolution.xy;
return texture2D(iChannel0, vec2(uv)/256.0, 0.0);

Composing visual effects

I attach a weight to each function or visual phase of the shader. Through this we can select which visual effect is visible or combine multiple effects. Its a bit messy, since I have all my functions in a single shader file. I’ve not explored including of external files with shaders.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
uniform float iCircularWeight;
uniform float iPopulationWeight;

vec4 circular(){...}
vec4 population(){..}

void main(void){
  vec4 circleResult     = vec4(0.0);
  vec4 populationResult = vec4(0.0);
  if(iCircularWeight > 0.0){
    circleResult = circular() * iCircularWeight;
  }
  if(iPopulationWeight > 0.0){
    populationResult = population() * iPopulationWeight;
  }
  gl_FragColor = (populationResult + circleResult);
}

And within Clojure:

1
2
3
4
5
6
(def circular-w (atom 1.0))
(def population-w (atom 1.0))
(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data {"iCircularWeight" circular-w
              "iPopulationWeight" population-w})
(reset! circular-weight 1.0)

Synchronisation

Shadertone uses the seconds since start (iGlobalTime) while Overtone via Supercollider uses the soundcard’s clock. Hence there is no guarantee these two sources will be in sync.

Replacing iGlobalTime is the only option. We create a special synth called data-probes which sole function is to transfer data from the Supercollider world to the Clojure world. Overtone provides a Supercollider to Clojure binding called a tap. We add a tap into our Overtone synth which is polling our global timing signal (this powers all synths and is how we co-ordinate everything).

1
2
3
4
5
6
7
8
9
10
11
(defsynth data-probes [timing-signal-bus 0]
  (let [beat-count (in:kr timing-signal-bus)
        _  (tap "global-beat-count" 60(a2k beat-count))
      (out 0 0))

(def active-data-probes (data-probes (:count time/beat-1th)))

(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data
  ;;An atom wrapping the tap and the running synth instance
   "global-beat-count" {"iGlobalBeatCount" (atom {:synth active-data-probes :tap "global-beat-count"})})

Using our iGlobalBeatCount in our shader now means anything requiring a continuously increasing value flows to our beat.

Shaders & global mutable state

Persistent mutable state between executions is not possible in OpenGL Shaders. Uniforms are read-only.

Lets look at an example. On a drum hit I want the color of a visual to change and persist until the next drum hit. The drum signal is 1 for a hit 0 for silence:

1
[1 0 0 0 1 0 0 0 1 0 0 0]

The current value based on the global clock is passed into the Shader as the iBeat Uniform.

1
2
3
4
5
6
7
8
9
uniform float iBeat;
float color = 0.0;

vec4 function showColor(){
  if(iBeat == 1.0){
    color += 1.0;
  }
  return vec4(color);
}

Will return:

1
2
3
4
5
vec4(1.0);
vec4(0.0);
vec4(0.0);
vec4(0.0);
vec4(1.0);

What we were after is actually:

1
2
3
4
5
vec4(1.0);
vec4(1.0);
vec4(1.0);
vec4(1.0);
vec4(2.0);

My solution is to move to Clojure where mutable state using atoms is simple.

Our timing is guided by Supercollider and a global clock. The value of our kick buffer at anyone time is only known inside the synth and hence inside Supercollider. But if we want to have mutable state we need access to this value in Clojure. So we create a custom synth that taps the value of the kick buffer based on the global clock signal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(defsynth drum-data-probe [kick-drum-buffer timing-signal-bus 0]
  (let [beat-count (in:kr timing-signal-bus)
    drum-beat (buf-rd:kr 1 kick-drum-buffer beat-count)
    _  (tap "drum-beat" 60 (a2k drum-beat))])
  (out 0 0))

(defonce kick-drum-buffer (buffer 256))

;;Create the synth with the "drum-beat" tap
(def drum-data-probe (drum-data-probe kick-drum-buffer (:count time/beat-1th)))

;;Bind the running synth and the tap
(def kick-atom (atom {:synth drum-data-probe :tap "drum-beat"}))

;;Extract the tap atom
(def kick-tap (get-in (:synth @kick-atom) [:taps (:tap @kick-atom)]))

Now in the Clojure world its simple to watch our tap atom and hence get alerted when it changes value. Overtone is dealing with the magic of updating the atom under the covers, the watcher is a nice implementation independent way of hooking into this. We now know the value of our kick buffer in Clojure. If we use another atom as our accumulator we can update it when the tap atom changes. Finally pushing this new accumulator to the Shader.

1
2
3
4
5
6
7
8
9
(def active-color (atom 0.0))

(add-watch kick-tap :cell-color
  (fn [_ _ old new]
    (when (and (= old 0.0) (= 1.0 new))
    (reset! active-color (mod (+ @active-color 1.0) 100)))))

(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data {:iActiveColor active-color}

Within the shader:

1
2
3
4
5
uniform float iActiveColor;

vec4 function showColor(){
  return vec4(iActiveColor);
}

Thats a lot of work, but I’m very happy with the results in my (end-of-buffer) performance.

Buffer events

Writing to a buffer is a common way of live coding in Overtone. Its very useful to attach some visual effect based on the settings of a buffer.

1
2
3
;;Setting notes to a buffer
(def notes-buf (buffer 256))
(pattern! notes-buf (degrees-seq [:f3 1314]))

We could put a tap into the synth and grab the current note and pass this into the shader. As I’ve mentioned taps are expensive and they are always on while we may not always be using them. This also gets more complicated when say we have 3 instances of the same synth running playing simultaneous to form a chord.

An alternative is to invent an atom which is used as a signal on every buffer write.

1
2
3
4
5
6
;;Could also do this with OSC messages...
(do
  (defonce buffer-change-event-notes-buf (atom 0.0))
  (pattern! notes-buf (degrees-seq [:f3 1314]))
  (swap! buffer-change-event-notes-buf + 1)
)

And adding a watcher

1
2
3
4
5
6
7
8
9
10
11
12
(add-watch
  buffer-change-event-notes-buf
  :buffer-change-event-notes-buf
  (fn [& _]
    (let [n (int (buffer-get notes-buf 0))]
      (case n
        29 (reset! color 0.4)
           (reset! color 0.0)))))

(def buffer-atoms (atom {}))
(pattern! notes-buf)
(reset! buffer-atoms (assoc @buffer-atoms (:id notes-buf) (inc (@buffer-atoms (:id notes-buf)))))

I use this trick in (end-of-buffer) to use the bass note to control the level of distortion of the visuals (source). Its wonderful to focus on the notes and feel the visuals following you automatically.

Midi notes to visuals

I often want to map a midi note to a visual effect. All my notes are mapped to buffers. Much like we did with the drums I can use a tap to get access to the current note being played in a buffer. Skipping over the details, when we have a midi note we send it as an float (to support crazy 42.333 like notes) to the shader via an atom.

We then map it to a nice range value to effect visuals:

1
2
3
4
5
6
7
uniform float iLeadNote;

//midi note: 42 => F#2  
//midi note: 78 => F#5
float noteMapped = smoothstep(42.0, 78.0, iLeadNote);

//noteMapped now spans => 0..1

A cheap way to scale effects based on the height of the note.

Gradual transitions

Often I want a smooth fading it or out of a shader function. Say for example fading to black. Pretty simple, just fire a thread which sleeps and ticks an atom. The atom is fed into the Shader.

1
2
3
4
5
uniform float iColor;

void main(void){
  gl_color = vec4(iColor);
}

Since I use this a lot I created a helper fn in MUD:

1
2
3
(def color (atom 1.0))
;;         atom / target /  rate
(overtime! color   0.0      0.001)

Text

In end-of-buffer I spell the word Repl Electric out of floating lights. We are bound to only a few data structures with fragment Shaders. I used a simple 3x3 matrix mapping each part of a character. Then using this to decided the position of the lights.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
const mat3 LETTER_R        = mat3(1, 1, 1,
                                  1, 1, 0,
                                  1, 0, 1);
const mat3 LETTER_E        = mat3(1, 1, 1,
                                  1, 1, 0,
                                  1, 1, 1);

vec4 letter(mat3 letter, vec2 offset, vec2 uv){
  vec2 point = vec2(0,0);
  vec4 helloPoint = vec4(0,0,0,0);
  vec3 xPos = vec3(0.01, 0.03, 0.05);
  vec3 yPos = vec3(0.05, 0.03, 0.01);

  for(int y=0; y < 3; y++){
    for(int x=0; x < 3; x++){
      if(letter[y][x] == 1){// Show this part of the letter
        point = vec2(xPos[x]+offset.x, offset.y+yPos[y]);
        helloPoint += buildCell(uv, point, STATIC_LETTERS);
      }
    }
  }
  return helloPoint;
}

letter(LETTER_R, 0.2, uv);

And the visual.

Font example

Visuals effected by frequencies

Shadertone provides a 2x512 array with the frequency spectrum (FFT) and audio waveform data. It does this by loading the data into a 2D Texture. The audio data is taken from tapping the main Overtone audio bus.

1
2
;;Tell Shadertone to fill iChannel0 with audio data
(shadetone/start "shaders/example.glsl" :textures [:overtone-audio])

It’s always a challenge to utilise this without creating something jerky or causing motion sickness. Hence I tend to use the waveform or FFT as a distorter rather than a base for animations.

It also helps to concentrate on specific ranges of frequencies of the waveform data to create a stronger connection between a synth and the visuals.

1
2
3
4
5
6
7
8
9
10
float sound = texture2D(iChannel0, vec2(max(uv.x,0.9),.75)).x;

//uv.xy => current x,y coordinates of pixel.

//First argument is an index into the 512 values of the waveform.
//By limiting the first argument we can ignore certain ranges.

//Second argument selects:
//    0.25 => FFT
//    0.75 => audio waveform

Here is an example where I use the audio waveform to distort the scale & jaggedness of a series of circle shapes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
const float tau = 6.28318530717958647692;
vec3 wave = vec3(0.0);
float width = 4.0/500;
for (int i=0; i < 60; i++){
  float sound = texture2D(iChannel0, vec2(uv.x,.75)).x;

  float a = 0.1*float(i)*tau/float(n);
  vec3 phase = smoothstep(-1.0,.5, vec3(cos(a), cos(a-tau/3.0), cos(a-tau*2.0/3.0)));
  wave += phase * smoothstep(width, 0.0, abs(uv.y - ((sound*0.9)+0.2)));

  //This shift of uv.x means our index into the sound data also 
  //moves along, examining a different part of the audio wave. 
  uv.x += 0.4/float(n);
  uv.y -= 0.05;
}
wave *= 10.0/float(n);
return vec4(wave,1);

And the resulting visual:

FFT example

Final thoughts on Live coding visuals

Through Clojure’s binding of atoms with Fragment shaders we have the power to live code visuals and music. Though it comes at a cost of complexity having to wrap lots of functions in order to have powerfully connected visuals. Fragment shaders are extremely terse, and can be pushed to replicate many advanced effects but they are also performance intense, and often taking a non-shader route will be much more performant.

Stability

My knowledge of LWJGL is small, but crashes in the fragment shaders often occur leaving the JVM wedged. This has happened to me quite a lot practicing, but never in a performance. Its worth reflecting that something (be it fixable) leaves a risk of a freeze in a performance.

Combining State & Shaders

I’ve started to explore what a shader application might look like if it was a server and provided a state machine so the live coding language does have this complexity. In turn producing a freer and more spontaneous interaction. This project is Shaderview and steals all the good ideas of Shadertone while adding some new features like vertex shader art. I’ll be writing up more about Shaderview soon.

Animations With Emacs

Emacs is designed for fast, highly customisable manipulation of text. ASCII animation requires manipulating text at a sufficient speed that it appears animated. Emacs is also used by a number of performers to live code musical & visual performances (and many other things). Where the audience can see the code in emacs and hear it.

Live Coding with Emacs

In my live coding performances as Repl Electric I’ve used emacs animations to augment emacs with more feedback for the performer and a chance to destroy the order and structure the programmer has spent the entire performance building. Reminding us that we are looking at thoughts expressed through code that seem magical but are ultimately nothing more than text.

Maybe something akin to the creation and destruction of Sand Mandalas.

Sand Mandala

Framework for Emacs Animation

Zone Mode is an Emacs plugin which provides a framework for screensaver like animations.

http://www.emacswiki.org/emacs/ZoneMode

Importantly it allows us to turn on an animation using our current code buffer as input and to terminate the animation, returning to the original code on a key press. So we can safely mangle the text knowing we can also return to safety. Well so far I’ve always found it to be safe but there is a small risk as mentioned in the zoning warning:

1
(message "...here's hoping we didn't hose your buffer!")

A nice property of taking our buffer as input is we are never quite sure what text will be there and hence the properties of the animation.

Example: Uppercase all letters

A simple function that finds non-whitespace in the buffer and tries to uppercase the char. It knows nothing about the zoning framework, its just a plain old function that operates on the active buffer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
(defun zone-upper-case-text ()
  (zone-fill-out-screen (window-width) (window-height))
  (random t)
  (goto-char (point-min))
  (while (not (input-pending-p))
    (let ((wbeg (window-start))
          (wend (window-end)))

      ;;Keep moving the char cursor until its not whitespace
      (while (looking-at "[ \n\f]")
        (goto-char (+ wbeg (random (- wend wbeg))))))

    ;;If we are at the end of the buffer go to the last char
    (when (eobp) (goto-char (point-min)))

    ;;Read the char at the cursor
    (let ((c (char-after (point))))
      (delete-char 1)           ;; Remove the char
      (insert-char (upcase c))) ;; Reinsert with caps      

    ;;Sleep
    (zone-park/sit-for (point-min) 0.1)))

The animation in all its glory:

Zoning Setup

We can override all other zoning programs and just specify our zone-fn. When we activate zoning our animation will be run.

1
2
3
(eval-after-load "zone"
  '(unless (memq 'zone-upper-case-text (append zone-programs nil))
         (setq zone-programs [zone-upper-case-text])))

Zoning Examples:

Zoning mode comes with lots of example animations that are good starting points:

http://www.opensource.apple.com/source/emacs/emacs-51/emacs/lisp/play/zone.el

  • zone-pgm-jitter
  • zone-pgm-putz-with-case
  • zone-pgm-dissolve
  • zone-pgm-whack-chars
  • zone-pgm-rotate
  • zone-pgm-rotate-LR-lockstep
  • zone-pgm-rotate-RL-lockstep
  • zone-pgm-rotate-LR-variable
  • zone-pgm-rotate-RL-variable
  • zone-pgm-drip
  • zone-pgm-drip-fretfully
  • zone-pgm-five-oclock-swan-dive
  • zone-pgm-martini-swan-dive
  • zone-pgm-paragraph-spaz
  • zone-pgm-stress

Open Sound Control Protocol Based animation

OSC is a handy protocol for sending data between networked devices using url like endpoints. Emacs has a plugin to run an OSC server (http://delysid.org/emacs/osc.html). Hence if we have some kind of beat signal we could send a message to Emacs and in turn it could render changes based on our musics timing.

With my Overtone setup for Repl-Electric I have the following flow of OSC messages:

1
[Supercollider] -> OSC -> [Clojure] -> OSC -> [Emacs]

Within Emacs setup an OSC server and define two call backs which change the color of the window face number

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
(require 'osc)
(require 'cl)

(defvar osc-server nil "Connection to receive msgs")
(defvar flip-state t)

(defun osc-connect ()
  "Create an OSC server and bind our fallback functions"
  (when (not osc-server)
    (setq osc-server
          (osc-make-server
           "localhost" 4558
           (lambda (path &rest args)
             (cond
              ((string-match "/beat" path)
               (progn (if flip-state (on-beat) (off-beat))
                      (setq flip-state (not flip-state))))))))))

(defun osc-make-server (host port default-handler)
  "Settings for OSC server"
  (make-network-process
   :name "emacs OSC server"
   :host host
   :server t
   :service port
   :filter #'osc-filter
   :type 'datagram
   :family 'ipv4
   :plist (list :generic default-handler)))

(defun on-beat ()
  (custom-set-faces
   '(window-number-face ((t (:foreground "deeppink"))))))

(defun off-beat ()
  (custom-set-faces
   '(window-number-face ((t (:foreground "#FDDD0C"))))))

(osc-connect)
(provide 'osc-server)

In Overtone/Clojure the sending signal:

1
2
(defonce emacs-client (osc-client "localhost" 4558))
(def emacs-trigger    (on-beat-trigger 8 #(do (osc-send emacs-client "/beat"))))

Heres a little demo with the brackets and window number changing colour based on the Overtone beat.

Emacs rendering to the beat

Synchronisation

Given some small local lag we now have a timing signal which is threaded through all our tools. Supercollider, Overtone and Emacs.

Which means our emacs animations can start to change to the beat of the music…

Sound in ASCII

Now that we have ways to animate and to connect audio data with emacs we can go a little further (way too far) and start to visualise the data about our sound in ASCII.

From Overtone or SuperCollider we can create a synth which tracks the peak and power of an audio signal. It sends us messages back with the data which we then forward on as OSC messages to Emacs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#Triggers a Sin Wave Oscillator and sends signals about power/peak
SynthDef(\pulse,{
  var sig, chain, onsets;
  sig = SinOsc.ar(Rand(220.0,440.0))
  *EnvGen.ar(Env.perc(releaseTime:0.5),Dust.ar(0.5))*0.7;
  Out.ar(0,sig !2);
  //
  chain = FFT({LocalBuf(512, 1)}, sig);
  onsets = Onsets.kr(chain,0.1,\power);
  SendTrig.kr(onsets);
  SendPeakRMS.kr(sig, 20, 3, "/replyAddress");
}).add;
#Run the crazy synth above
Synth(\pulse)

#Forward the data on as an OSC message
#to emacs
~host = NetAddr("localhost", 4859);
p = OSCFunc({ |msg|
  ~host.sendMsg("/peakpower",msg[3], msg[4]);
  "peak: %, rms: %".format(msg[3], msg[4]).postln
}, '/replyAddress');

And in our emacs OSC server:

1
2
3
4
5
6
7
8
9
((string-match "/peakpower" path)
  (progn
    (with-current-buffer "flatiron.clj"
      (let ((sig (round (* 100.0 (first args)))))
        (message (format "%f" (first args)))
        (dotimes (n sig)
          (insert "▓"))
        (insert "▒░"))
      (insert "\n"))))

Repl Electric Emacs animations

All my Emacs animations are used to conclude the performance. Heres lies the source code, some screenshots and tricks & tips that made the animations possible.

Here’s a demo of all the Repl Electric animations discussed in action:

End of Buffer

https://github.com/repl-electric/view-pane/blob/master/animations/end-of-buffer.el

end-of-buffer-01end-of-buffer-02end-of-buffer-03

In this animations the text gets slowly broken up with white spaces and then like the wind, blows the characters away. Sometimes words are ripped apart as they blow in the wind (if we get lucky).

Two main phases:

  • Injection of spaces. This starts to distort the text while keeping it readable. It provides a way to increase the effect of expanding whitespace in the next stage.

  • Transforming whitespace into lots of whitespace. A Regex matches whitespace and replaces it with a randomly increasing amount of whitespace. Which leads to the effect of the characters of the code blowing away. I spent a while trying to improve the speed of this phase and Regexs proved to be the fastest way.

If we move the text fast enough soft word wrapping means the text appears to re-enter from the left side of the screen and eventually disappear out of the buffer. Without soft wrapping we get a horrible jitter as emacs moves back and forth between left and right side of the buffer.

A couple of other tricks/tactics used:

  • Continually incrementing integer. Useful for injecting movement or using sin/cos fn with a continuous value.
  • Perserving the syntax highlighting of the original code in an attempt to maintain some of the meaning of the code.

The Stars

https://github.com/repl-electric/view-pane/blob/master/animations/the-stars.el

This was my first animation and was based heavily on zone-pgm-drip-fretfully.

It randomly picks a single character and moves it down the screen until it hits another letter or exits the screen.

When running Emacs + Overtone + OpenGL, Emacs starts to slow down so part of the challenge was ensuring the animation ran as fast as possible.

A nice property of this is that as the OpenGL shaders shutdown, the speed of the animation increases and the code destroys itself more quickly.

Waves

https://github.com/repl-electric/view-pane/blob/master/animations/waves.el

This animations attempts to simulate the effect of waves using line wrapping and mixing deletions with insertions of different sizes to create lines that seem to move at different speeds.

Breaking Tools

While it may seem silly to bend Emacs to do things it was never intended to do, it’s an important part of discovering for yourself how you want your tools to work. Not just doing what you are expected but breaking them apart and discovering for yourself how you want to use them.

Clojure and Kinesis at Scale

I’ve been working over the last year in the data team at SoundCloud building a realtime data pipeline using Clojure and Amazon’s Kinesis. Kinesis is Amazons equivalent to Kafka, “Real-Time data processing on the Cloud”. This is a summary of what was built, some lessons learnt and all the details in-between.

Kinesis pipeline
Kinesis pipeline at SoundCloud

Tapping Real traffic

The first step was to tee the traffic from a live system to a test system without comprising its function. The main function of the live system is logging JSON events to file (which eventually end up somewhere like HDFS). Tailing the logs of the live system gives us access to the raw data we want to forward on to our test system. A little Go script watches the logs, parses out the data and then forwards them in batch to test instances that will push to Kinesis. Hence we had live data flowing through the system and after launch a test setup to experiment with. Sean Braithwaite was the mastermind behind this little bit of magic.

Canary Kinesis pipeline
Tapping Traffic

Sending to Kinesis

All Kinesis sending happens in an application called the EventGateway (also written in Clojure). This endpoint is one of the most heavily loaded services in SoundCloud (at points it has more traffic than the rest of SoundCloud combined). The Eventgateway does a couple of things but at its core it validates and broadcasts JSON messages. Hence this is where our Kinesis client slots in.

Squeezing Clojure Reflection

Its worth mentioning that in order for the Eventgateway service to be performant we had to remove all reflection in tight loops through type hints. It simply could not keep up without this. It became a common pattern to turn reflection warnings on while working in Clojure.

Project.clj

1
:profiles {:dev {:global-vars {*warn-on-reflection* true *assert* false}}}

Kinesis

The Eventgateway posts to Kinesis in batch using a ConcurrentLinkedQueue and separate producers and consumers. Messages are pushed into a ConcurrentLinkedQueue. We rolled our own Clojure Kinesis client using Amazons Java library rather than using Amazonica.

1
2
3
;; Java Amazon libraries used
[com.amazonaws/aws-java-sdk "1.9.33"         :exclusions [joda-time]]
[com.amazonaws/amazon-kinesis-client "1.1.0" :exclusions [joda-time]]

Amazonica was good to get started quickly in the initial phase but there are a couple of reasons we switched to our own unique snowflake (which still looked a little like Amazonica):

  • Amazonica did not support batch mode for Kinesis. Under initial tests it was impossible to scale this without batch.
  • Injecting our own telemetry at low levels to learn more about Kinesis running.
  • Some of its sensible defaults where not so sensible (for example default encoding the data using nippy).
  • Ultimately most of any Kinesis client/server is configuration and tuning.
  • Amazonica’s source is hard to read with a little too much alter-var-root going on.
1
2
3
4
5
6
;;Ugh. Its not just me right?
(alter-var-root
  #'amazonica.aws.kinesis/get-shard-iterator
  (fn [f]
    (fn [& args]
      (:shard-iterator (apply f args)))))

Pushing Messages in a Queue

Very simple, just adding a message to the ConcurrentLinkedQueue. A environment variable allows us to gradually scale up or down the percentage of traffic that is added to the queue.

1
2
3
4
5
6
7
8
9
10
11
12
(require '[environ.core :refer :all])

(def kinesis-message-queue (ConcurrentLinkedQueue.))
(def hard-limit-queue-size 1000)
(def queue-size (atom 0))

(defn send [message]
  (when-let [threshold (env :kinesis-traffic)]
    (when (> (rand-int 100) (- 100 (or (Integer/valueOf ^String threshold) 0)))
    (when (<= @queue-size hard-limit-queue-size)
      (.add kinesis-message-queue message)
      (swap! queue-size inc)))))
Failure

The queue pusher operates within a wider system and any failures due to Amazon being unreachable should not impede the function of the system. For the client this means:

  • Not exceeding memory limits with a hard queue size (since ConcurrentLinkedQueue is unbound in size).
  • Backing off workers if the queue is full to prevent cpu throttling.

When we cannot send messages to Kinesis we instead log them to disk, and into our normal logging pipeline (usually ending up in HDFS). Hence we could replay at a later date if required.

Sending batches to Kinesis

The workers, operating in separate threads consuming messages from the ConcurrentLinkedQueue collecting them into a batch:

1
2
3
4
5
6
7
8
9
10
11
12
(loop [batch []
       batch-start-time (time/now)
       poll-misses 0]
  (if (batch-ready-to-send? batch batch-start-time)
    batch
    (if-let [event (.poll kinesis-message-queue)]
      (do
        (swap! queue-size dec)
        (recur (conj batch event) batch-start-time 0))
      (do
        (Thread/sleep (exponential-backoff-sleep poll-misses))
        (recur batch batch-start-time (inc poll-misses))))))))

When polling from the queue an exponential back-off if no messages are on the queue.

1
2
3
4
5
6
7
8
9
10
(defn exponential-backoff-sleep
  "Exponential backoff with jitter and a max "
  [misses]
  (let [max-timeout 1000
        jitter-order 10]
    (Math/min
     max-timeout
     (Math/round (+ (Math/exp misses)
                    (* (Math/random)
                       jitter-order))))))

Once the batch is ready (in terms of age or size) its sent to Kinesis.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
(defn- failures->events [^PutRecordsResult put-results events]
  "Find the events that are marked as failed in the PutRecordsResult"
  (->> put-results
       .getRecords
       (map vector (range))
       (filter (fn [[idx ^PutRecordsResultEntry e]] (.getErrorCode e)))
       (map (fn [[idx _]] (nth events idx)))))

(defn- send-events
  "Perform putRecords request to send the batch to Kinesis
   returns a list of events that failed."
  [^AmazonKinesisClient client stream-name events]
  (try+
   (let [result (.putRecords client (events->put-records-request events stream-name))]
     (if (pos? (.getFailedRecordCount result))
       (failures->events result events))
       [])))

Note this is where we also decided the partition key. In our case its important for the same user to be located on the same partition. For example when consuming from Kinesis a worker is allocated a partition to work from and would miss events if they where across multiple partitions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
(defn- events->put-records-request
  "Take client and a vector of JsonNodes and produce a PutRecord"
  [batch event-stream]
  (let [batch-list  (java.util.ArrayList.)
        put-request (PutRecordsRequest.)]
    (.setStreamName put-request event-stream)
    (doseq [^ObjectNode event batch]
      (.remove event ^String failure-metadata)
      (let [request-entry (PutRecordsRequestEntry.)
            request-data  (.getBytes (str event))
            request-buf   (ByteBuffer/wrap request-data 0 (alength request-data))
            partition-key (:user-id event)]
        (doto request-entry
          (.setData         request-buf)
          (.setPartitionKey partition-key))
        (.add batch-list request-entry)))
    (.setRecords put-request batch-list)
    put-request))
Failure

Failure can occur on individual records within a batch or in the batch as a whole.

Individual failures
  1. These messages are re-added to the queue so we can try again. If the messages fail for some nth time they are considered invalid and rejected from Kinesis and logged as an error.
Batch level
  1. Amazon had an Internal Failure. We don’t know what went wrong. (We see this regularly in normal function).

  2. Amazon Kinesis is not resolvable (AmazonClientException/AmazonServiceException).

  3. Exceeding the read/write limits of Kinesis (ProvisionedThroughputExceededException).

This is our backpressure signal, in which case at worst we need to log to disk for replay later.

Consuming Messages from Kinesis

With the consumption of events we have a different application stream for every worker. All workers have their own streams, and own checkpoints so they operate independently of each other. Some example of the workers we gave running:

  • Logging Events to s3
  • Calculating listening time
  • Forwarding certain messages on to various other systems (like RabbitMQ).

Launching a worker is pretty simple with the Amazon Java Kinesis library.

1
2
3
4
5
6
7
8
(:import [com.amazonaws.services.kinesis.clientlibrary.lib.worker Worker])

(defn -main [& args]
  (let [worker-fn (fn [events] (print events))
        config (KinesisClientLibConfiguration.   worker-fn) ;;I'm airbrushing over the Java classes
        processor (reify IRecordProcessorFactory worker-fn) ;;Ultimately this is a lot of config wrapped in Java fun
        [^Worker worker uuid] (Worker. processor config)]
          (future (.run worker))))

One of the hardest parts of setting up the a worker is getting the configuration right to ensure that the consumers are getting through the events fast enough. Events are held in Amazon for 24 hours after entry, and hence there is a minimum consumption rate.

Counting events in and events out with Prometheus made it easier to get the correct consumption rates. Entry/exit rates

Via the Amazon console you also get access to various graphs around read/write rates and limits:

Finally you can also look at Amazon’s Dynamodb instance for the Kinesis stream providing insight into metrics around leases, how many where revoked, stolen, never finished, etc.

Here is an example of one of our Kinesis workers configuration covered in scribblings of me trying to work out the right settings.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
  {
   ;;default 1 sec, cannot be lower than 200ms
   ;;If we are not reading fast enough this is a good value to tweak
   :idle-time-between-reads-in-millis 1800

   ;;Clean up leases for shards that we've finished processing (don't wait
   ;;until they expire)
   :cleanup-leases-upon-shard-completion true

   ;;If the heartbeat count does not increase within the configurable timeout period,
   ;;other workers take over processing of that shard.
   ;;*IMPORTANT* If this time is shorter than time for a worker to checkpoint all nodes
   ;;will keep stealing each others leases producing a lot of contention.
   :failover-time-millis 80000

   ;;Max records in a single returned in a `GetRecords`. Cannot exceed 10,000
   :max-records 4500

   ;;Process records even if GetRecords returned an empty record list.
   :call-process-records-even-for-empty-record-list false

   ;;Sleep for this duration if the parent shards have not completed processing,
   ;;or we encounter an exception.
   :parent-shard-poll-interval-millis 10000

   ;;By default, the KCL begins withs the most recently added record.
   ;;Instead always reads data from the beginning of the stream.
   :initial-position-in-stream  :TRIM_HORIZON}

Monitoring

Prometheus (http://prometheus.io/) a monitoring tool built at SoundCloud was core to developing, scaling and monitoring all of this pipeline. Amazon does provide some useful graphs within the AWS console but more detailed feedback was very helpful even if it was removed later.

Exception Logging pattern

All Exceptions are counted and sent to log. This was a very useful pattern for driving out errors and spotting leaks in the interactions with Kinesis and consumption:

(Using a Clojure wrapper around Prometheus: https://github.com/josephwilk/prometheus-clj)

1
2
3
4
5
6
(try+
  (worker-fn raw-events)
(catch Exception e
  ;;Count based on exception class
  (inc-counter :failed-batches {:type worker-type :error-code (str (.getClass e))})
  (log/error e)))]

Note Kinesis regularly spits out “InternalFailure” Exceptions. Thats all you get…

Kinesis Internal failures

A Cloud Pipeline in Pictures

In my previous post about Building Clojure services at scale I converted the system metrics to sound. This time I convert the metrics across all the machine into vertexes on a mesh using OpenFrameworks. So you start to see a visual representation in 3D of the function of the system.

Thanks

This work constitues a team effort by the Data team at SoundCloud. A lot of advice, collaboration and hard work. Kudos to everyone.

Clojure and Overtone Driving Minecraft

Using Clojure we create interesting 3D shapes in Minecraft to the beat of music generated from Overtone.

We achieve this by embedding a Clojure REPL inside a Java Minecraft server which loads Overtone and connects to an external Supercollider instance (What Overtone uses for sound).

Demo

Demo Source code: https://github.com/josephwilk/clj-minecraft/blob/master/src/cljminecraft/overtone.clj

Tools

(Dependent on your IDE of choice)

Building the world

We need to install Spigot which is an optimized version of the Craftbukkit Java Minecraft server and install clj-minecraft project as a plugin. Things are complicated by Bukkit no longer being registered in Maven.

Read through the Makefile install:

https://github.com/josephwilk/clj-minecraft/blob/master/Makefile

If your happy clone and run:

1
git clone git@github.com:josephwilk/clj-minecraft.git && make install

Bukkit + clj-minecraft will be installed for you.

Running the world

You will need to edit the minecraft/eula.txt indicating you agree with the license. Then you can run your Minecraft server:

1
java -XX:MaxPermSize=1G -jar spigot-1.8.jar

Seeing the world

Buy and install a Minecraft client: https://minecraft.net

Select >“Multiplayer” >“Direct connect” and enter the “Server Address” as localhost.

Bring music to the world

Install and Boot Supercollider

http://supercollider.github.io

Once installed:

1
2
3
4
5
6
7
#Find the Supercollider binary (scsynth)
$ sudo find /. -name "scsynth"

/Applications/SuperCollider/SuperCollider.app/Contents/Resources/scsynth

#Run Supercollider server
/Applications/SuperCollider/SuperCollider.app/Contents/Resources/scsynth -u 57110

Speaking to the Minecraft REPL

clj-minecraft opens a REPL on localhost port 4005. Using emacs and cider connect to this REPL instance.

Boot and connect Overtone:

1
2
3
(use 'overtone.core)

(connect-external-server)  #=> :happy-hacking

Interaction

Using MUD we have some useful wrappers around Overtone for scheduling functions on beats.

To coordinate graphics and sound we schedule both within a single function.

1
2
3
4
5
6
7
8
9
10
(require '[mud.core :as mud])

(defonce highhat-sample (freesound 53532))

(def ride-trigger
  (mud/on-beat-trigger 8       ;; Every 8th beat
   (fn []
     (highhat-sample)      ;; Play sample
     (block 2 10 2 :grass) ;; Place a block into the Minecraft world
)))

Most functions that change the state of the Minecraft world need to be run in the main GUI thread. To achieve this we wrap any state changing function within ui-sync (https://github.com/CmdrDats/clj-minecraft/blob/a3331e925b56becf88d9ef96cab225856e2f7ead/src/cljminecraft/bukkit.clj#L39-L42).

For example a function to place a block into the Minecraft world:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
(import '[org.bukkit Location Material])
(require '[cljminecraft.bukkit :as bk])
(require '[cljminecraft.items :as i])

(def player (first (.getOnlinePlayers (bk/server))))

(defn block
  "place a block relative to current players position
  Example:
    (block 2 10 2 :grass)
  "
  [x y z material]
  (let [l (.getLocation player))
        m (i/get-material material)]
    (doto l
      (.setX (+ x (.getX l)))
      (.setY (+ y (.getY l)))
      (.setZ (+ z (.getZ l))))
    (bk/ui-sync
     @cljminecraft.core/clj-plugin
     (fn []
       (doto (.getBlock l)
         (.setData 0)
         (.setType (.getItemType m))
         (.setData (.getData m)))))))

For further documentation of whats possible, the Bukkit Java docs: https://hub.spigotmc.org/javadocs/bukkit/

clj-minecaft has lots of helpers + examples: https://github.com/CmdrDats/clj-minecraft/tree/master/src/cljminecraft

Fun

Create, play and share all the crazy things you can come up with using Clojure and Minecraft.

The Pig Algo Rave

For more live programming music and sound checkout my performances as Repl Electric: http://www.repl-electric.com

Credits

Built on the back of lots of great open source projects. Thanks to the Craftbukkit/Spigot contributors, @CmdrDats for clj-minecraft and @samaaron for Overtone and inspiring this crazy journey with the musical Sonic Pi (which supports combining music and Minecraft on the RaspberryPi).

Live Coding - Repl Electric

Live coding is the act of turning a programming session into a performance. This can constitute improvisation, music, visuals, poetry, hardware, robots, dance, textiles and people. Pretty much anything with an input and output can be controlled live by programming.

This is not just a performance by programmers for programmers. While this is often where it starts as a live coder, the type of audience and the accessibility of the performance lies in the performers imagination. Abstraction can get us pretty much anywhere.

1
(def the-stars (dark-matter))

Repl Electric

Repl Electric is a project I started in order to discover more about music composition and Artificial intelligent based aids to creativity. Which in turn through the inspiration of people like Meta-ex lead me to live programming music.

Here is a performance live coding music and graphics, inspired by a performance in London:

The Stars

Open Live Coding

All the tools and code used to create this performance are open for all to see on Github: https://github.com/repl-electric

Three programming languages were used to create this piece:

  • Clojure (Sound)
  • GLSL (Visuals)
  • Emacs Lisp (Animations & Navigation)

Tools

Here are the tools used and a little detail around how they where used in performing “The Stars”:

Clojure: http://clojure.org

Clojure is a LISP language based on the JVM.

Clojure focuses on interactive REPL (Read, Evaluate, Print & Loop) driven development. Which makes it a good choice for interactively coding music. It also turns out functional programming is a good fit for operating over music as data.

Emacs Live: https://github.com/overtone/emacs-live

Emacs live is a Emacs release with packages and defaults that are Live Coding centric. Something I use for both for my work and for my live coding.

To execute our code, we launch a repl instance in our project (NEVER launch inside emacs, since then if emacs crashes the repl and music dies) and connect to it from emacs using cider https://github.com/clojure-emacs/cider.

A simple trick to combine Emacs code and visualizations is to launch an OpenGL window in full screen (see Shadertone) and then put a full screen transparent terminal window running emacs over it.

The tumbling text effect seen at the end of the performance is an emacs animation using Zone Mode which supports writing your own text destructors: http://www.emacswiki.org/emacs/ZoneMode

Overtone: https://github.com/overtone/overtone

Overtone is a Clojure based client to SuperCollider. Supercollider is an environment for real time audio synthesis and algorithmic composition.

Overtone provides us with:

  • Timing (beat generation – example timing code).
  • Building Synths (engineer sound).
  • Running samples (both your own and from Freesound).
  • Live Synth control (changing notes, durations, reverb, etc).
  • Hardware interaction (through midi or OSC).

An example of a synth used in The Stars:

1
2
3
4
5
6
7
8
9
10
(use 'overtone.live)
(defsynth dark-ambience [out-bus 0 amp 1 mul 0.2 room-size 70 rev-time 99 freq 60 ring-mul 55]
  (let [pink (hpf:ar (* (* 0.005 (pink-noise)) (line:kr 0 1 9)) 5)
        src1 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 0)) mul)
        src2 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 1)) mul)
        src3 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 2)) mul)
        src (tanh (g-verb (sum [src1 src2 src3]) room-size rev-time))]
    (out out-bus (* amp src))))

(def the-stars (dark-ambience))
Timing

Timing is a complicated issue but so important its worth touching on. You have a choice with Overtone to use Java for timing or Supercollider. I use Supercollider since I have found it to be much more reliable. Everything you need is here (copy and paste), thanks to the hard work of Sam Aaron.

The key concept to take away is there are two types of timing, a beat counter which is forever incrementing and a beat trigger which flips back and forth between 1/0.

1
2
3
4
5
6
7
(require '[cassiopeia.engine.timing :as time])

;;The beat trigger
(:beat time/main-beat) ;=> 0,1,0,1,0,1,0

;;The beat counter
(:count time/main-beat) ;=> 0,1,2,3,4,5,6,7,8,9

The counter is useful for indexing buffers, the trigger is useful in controlling the gate of an envelope (which turns a sound on or off).

In Clojure we can still get access to the beat, in our timing code we send a message using send-trig on every beat. We can hook a Clojure function to callback on this beat:

1
2
3
4
5
6
(on-trigger (:trig-id time/main-beat)
  (fn [beat-no]
    (when (= 0.0 (mod beat-no 32))
      ;;play a sample
      (boom-s)))
  ::on-beat-trigger)

I use this extensively to time graphic transitions with the music.

Buffers

Most of my live coding performance was writing to buffers which are hooked into synths. Buffers are just fixed size arrays but they are stored in Supercollider rather than in Clojure. Here is an example from The Stars where the midi notes are read from a buffer at a rate based on my beat timing signal (a 16th of the main beat here).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
(use 'overtone.live)
(use 'cassiopeia.engine.core)
(require '[cassiopeia.engine.timing :as time])

(defsynth growl [out-bus 0 note-buf 0 beat-bus 0 beat-trg-bus 0 amp 1]
  (let [cnt (in:kr beat-bus)
        trg (in:kr beat-trg-bus)
        note (buf-rd:kr 1 note-buf cnt)
        freq (midicps note)
        vol (> note 0)

        e (env-gen (perc :attack 10 :sustain 1 :release 1) :gate trg)
        src (lpf (mix [(saw (* 0.25 freq))
                       (sin-osc (* 1.01 freq))]))
        src (pitch-shift src 0.4 1 0 0.01)
        src (pan2:ar (* vol amp e src))]
    (out out-bus src)))

(defonce nebular (buffer 96))

(def nebula (growl :note-buf nebula-note-buf :beat-trg-bus (:beat time/beat-16th) :beat-bus (:count time/beat-16th)))

(pattern! nebula-note-buf (degrees [1 3 7] :major :A2))

GLSL + Shadertone: https://github.com/overtone/shadertone

Shaders generate imagery directly on your Graphics Processing Unit rather than going through your CPU. Through a language called GLSL (which is C like) we can express very simple functions which get called on every single pixel generating complex visuals. Here is a simple extract from The Stars that generates all the background small dots:

1
2
3
4
5
6
7
8
9
10
11
12
void main(void){
  vec2 current_pixel_position = mod(gl_FragCoord.xy, vec2(5.0)) - vec2(0.0);
  float distance_squared = dot(current_pixel_position, current_pixel_position);

  vec4 black = vec4(.0, .0, .0, 0.0);
  vec4 white = vec4(1.0, 1.0, 1.0, 1.0);

  //Test the current pixel position and if it should be a circle shade it.
  vec4 circles = (distance_squared < 0.6) ? white : black;

  gl_FragColor = circles;
}

For more examples of whats possible with Shaders checkout Shader Toy

Shadertone is the Clojure library that provides a convenient way of running shaders from Clojure and for feeding in data about our synths. It provides access in your Shader to:

  • Overtone’s Volume (iOvertoneVolume)
  • The frequency spectrum & audio waveform data (Passed as a 2D texture :textures [:overtone-audio])

To synchronize the graphics with the music I created a special Overtone synth which does not generate any sound, it instead feeds information in realtime to my shader.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
(use 'overtone.live)
(require '[shadertone.core :as t])

;;A synth that exposes through taps all the lovely timing information.
(defsynth buffer->tap [beat-buf 0 beat-bus 0 beat-size 16 measure 6]
  (let [cnt (in:kr beat-bus)
        beat (buf-rd:kr 1 beat-buf cnt)
        _  (tap "beat"          60 (a2k beat))
        _  (tap "beat-count"    60 (a2k (mod cnt beat-size)))
        _  (tap "measure-count" 60 (a2k (/ (mod cnt (* measure beat-size)) measure)))])
  (out 0 0))

;;; Used to store our drum beat, 1 for a hit 0 and for a miss
(defonce drum-sequence-buffer (buffer 256))

(def beats (buffer->tap drum-sequence-buffer (:count timing/main-beat)))

;;Open a OpenGL window running our shader
(t/start-fullscreen "resources/shaders/electric.glsl"
                    :user-data {
                    "iBeat"         (atom {:synth beats :tap "beat"})
                    "iBeatCount"    (atom {:synth beats :tap "beat-count"})
                    "iMeasureCount" (atom {:synth beats :tap "measure-count"})})

Inside our shader code:

1
2
3
4
uniform float iBeat;
uniform float iBeatCount;
uniform float iMeasureCount;
...

The other main way of controlling a shader from Clojure is using atoms.

1
2
3
4
5
6
7
(require '[shadertone.core :as t])

(defonce cellular-growth (atom 0.0))

(t/start-fullscreen "resources/shaders/electric.glsl" :user-data {"iCellularGrowth" cellular-growth})

(swap! cellular-growth + 0.01)

Hardware: Monome: http://monome.org

Something you don’t see in the video is that I’m using a 8x16 Monome. For this performance its primary function was a visual aid to show the beat/measure information.

Monome

The hardware is driven by Clojure communicating with the Monome through a serial port: https://github.com/josephwilk/monome-serial/tree/protocols

Live Coding

Live coding music and graphics combines skills in sound engineering, 3d graphics, geometry, physics, musical theory, composition, improvisation & hardware to name a few.

It is difficult, and requires a lot of work and practice.

But of all the code I’ve written over the years this is one of the things I’m most proud of. And i’m only at the beginning of discovering what’s possible.

Creative Machines

When Alan Turing asked if a machine can be intelligent one aspect of this question focused on “could machines be creative”?

Ada Lovelace seemed convinced that originality was not a feat a computer was capable of:

it can do whatever we know how to order it to perform, it has no pretensions whatever to originate anything

Before we outrightly dismiss the idea of creative machines do we even understand what creativity is?

Join me on a journey examining these questions while also meeting a new generation of artists born through code. Looking into their hearts and brains examining different algorithms/techniques and there effectiveness at exhibiting creativity.

Decide for yourself, can machines be creative?

My Strangeloop talk: Creative Machines – http://www.infoq.com/presentations/ai-machine-creativity

Slides:

Full source code: https://github.com/josephwilk/musical-creativity

Continuing the Journey

In order to continue my research into creativity and music I’ve started the project Repl Electric. A space for humans and machines to learn and create music together.

Further Reading

Sounds of the Human Brain

What does your brain sound like? Does it sound like “Rise of the Valkyrie” or more like “Hit me baby one more time”?

Step 1: Acquire a human brain (alive)

We are interested in capturing the brain waves. Specifically the:

  • Delta waves: Deepest stages of sleep.
  • Beta waves: Normal waking consciousness.
  • Alpha waves: Relaxation and meditation (creativity).
  • Theta waves: REM sleep (dreams).
  • Gamma waves: Hyper-alertness, perception, and integration of sensory input.

Step 2: EEG machine

I am using a EEG machine brought from Neurosky which is rated as Research grade (whatever that means). This measures voltage fluctuations resulting from ionic current flows within the neurons of the brain. While EEG machines are not the most accurate they are now reasonably cheap.

Step 3: EEG –> Overtone

In order to generate music I want to import the EEG brainwave data into Overtone.

We interact with the EEG machine over a serial port. The most mature library for this interface is in Python so there is a little jiggery pokery to get the data into Overtone.

The Brainwave Poller

We use https://github.com/akloster/python-mindwave to interface with the EEG machines data.

Writing all the data out to a FIFO file file as json.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
import re
import time
import json
import unicodedata

import gevent
from gevent import monkey

from pymindwave import headset
from pymindwave.pyeeg import bin_power

monkey.patch_all()

# connect to the headset
hs = None
hs = headset.Headset('/dev/tty.MindWave')
hs.disconnect()
time.sleep(1)
print 'connecting to headset...'
hs.connect()
time.sleep(1)
while hs.get('state') != 'connected':
    print hs.get('state')
    time.sleep(0.5)
    if hs.get('state') == 'standby':
        hs.connect()
        print 'retrying connecting to headset'

def raw_to_spectrum(rawdata):
    flen = 50
    spectrum, relative_spectrum = bin_power(rawdata, range(flen), 512)
    return spectrum

while True:
    t = time.time()
    waves_vector = hs.get('waves_vector')
    meditation = hs.get('meditation')
    attention = hs.get('attention')
    spectrum = raw_to_spectrum(hs.get('rawdata')).tolist()

    with open("/tmp/brain-data","w") as fp:
        s = {'timestamp': t,
             'meditation': meditation,
             'attention': attention,
             'raw_spectrum': spectrum,
             'delta_waves': waves_vector[0],
             'theta_waves': waves_vector[1],
             'alpha_waves': (waves_vector[2]+waves_vector[3])/2,
             'low_alpha_waves': waves_vector[2],
             'high_alpha_waves': waves_vector[3],
             'beta_waves': (waves_vector[4]+waves_vector[5])/2,
             'low_beta_waves': waves_vector[4],
             'high_beta_waves': waves_vector[5],
             'gamma_waves': (waves_vector[6]+waves_vector[7])/2,
             'low_gamma_waves': waves_vector[6],
             'mid_gamma_waves': waves_vector[7]}

        s = json.dumps(s)
        fp.write(s)
    gevent.sleep(0.4)

Reading from the FIFO file is simple in Clojure.

1
2
3
(while true
  (with-open [reader (clojure.java.io/reader "/tmp/brain-data")]
    (brainwave->music (json/decode (first (line-seq reader)) true))))

Step 3: Sonification

Sonification is the process of taking data and turning it into sound. Here is an example of the data we are now receiving:

A single JSON brainwave packet:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
 {"gamma_waves": 95408,
  "high_beta_waves": 205681,
  "beta_waves": 293928,
  "low_beta_waves": 382176,
  "mid_gamma_waves": 84528,
  "low_alpha_waves": 172417,
  "delta_waves": 117933,
  "low_gamma_waves": 106288,
  "alpha_waves": 112605,
  "theta_waves": 635628,
  "high_alpha_waves": 52793,
  "attention": 0,
  "meditation": 0,
  "timestamp": 1.375811275696894E9}

We will focus on the beta-waves for simplicity. Beta-waves fall between 16.5–20Hz.

EEG beta waves

Beta waves related to:

  • Alertness
  • Logic
  • Critical reasoning

We need to map a signal within 16.5-20Hz into the musical pitches of a sampled piano (21-108 pitches).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
(require '[clojure.math.numeric-tower :as math])

(defn linear-map [x0 x1 y0 y1 x]
  (let [dydx (/ (- y1 y0) (- x1 x0))
        dx (- x x0)]
    (+ y0 (* dydx dx))))

;; Piano range: 0..87
;; Beta wave range: 16500..20000

(defn beta-wave->pitch [signal] (-> (linear-map 16500 20000 21 108 signal) float math/round))

(beta-wave->pitch 16500) ;-> 21
(beta-wave->pitch 20000) ;-> 108

Now we extract the beta-waves from the brainwave data and play them. We play them live as they arrive rather than worrying about scheduling notes:

1
2
3
4
5
6
7
8
9
(use 'overtone.live)
(use 'overtone.inst.sampled-piano)
(require '[cheshire.core :as json])

(while true
  (with-open [reader (clojure.java.io/reader "/tmp/brain-data")]
    (let [beta-wave (-> (first (line-seq reader)) (json/decode true) :beta_waves)]
      (println beta-wave)
      (sampled-piano :note (beta-wave->pitch beta-wave) :sustain 0.2))))

Would you like to hear my brain?

The results, please listen to my brain.

Not really music is it? With beta-waves we get a serious of high to low transitions. While we can control at what pitch the transitions occur by performing activities that shape our brain waves the transitions don’t provide the order or structure we need to recognize this as music.

Brain controlled Dubstep

The only logical path left is to try and control Dubstep with our brain. Rather than generative music we can use our brain waves to control the tempo and volume of existing synthesized music.

Here is a Dubstep synth taken from Overtone:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
(use 'overtone.live)

(defsynth dubstep [bpm 120 wobble 1 note 50 snare-vol 1 kick-vol 1 volume 1 out-bus 0]
  (let [trig (impulse:kr (/ bpm 120))
        freq (midicps note)
        swr (demand trig 0 (dseq [wobble] INF))
        sweep (lin-exp (lf-tri swr) -1 1 40 3000)
        wob (apply + (saw (* freq [0.99 1.01])))
        wob (lpf wob sweep)
        wob (* 0.8 (normalizer wob))
        wob (+ wob (bpf wob 1500 2))
        wob (+ wob (* 0.2 (g-verb wob 9 0.7 0.7)))

        kickenv (decay (t2a (demand (impulse:kr (/ bpm 30)) 0 (dseq [1 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0] INF))) 0.7)
        kick (* (* kickenv 7) (sin-osc (+ 40 (* kickenv kickenv kickenv 200))))
        kick (clip2 kick 1)

        snare (* 3 (pink-noise) (apply + (* (decay (impulse (/ bpm 240) 0.5) [0.4 2]) [1 0.05])))
        snare (+ snare (bpf (* 4 snare) 2000))
        snare (clip2 snare 1)]

       (out out-bus (* volume (clip2 (+ wob (* kick-vol kick) (* snare-vol snare)) 1)))))

Once the synth is running we can send it control signals which will vary any of the properties defined in the arguments to the dubstep function:

  • bpm
  • wobble
  • note
  • snare-vol
  • kick-vol
  • volume
1
2
3
4
5
6
7
(def d (dubstep))

(ctl d :snare-vol 0)
(ctl d :kick-vol 0)
(ctl d :wooble 0)
(ctl d :bpm 20)
(ctl d :v 0.2)

We again have to linearise the beta wave signal to the range of volume 0.0-1.1 and to the bpm 0-400.

Now all thats left to do is connect it to our brain.

Here’s what brain controlled Dubstep sounds like:

And for comparison what playing Go does to your brain activity (I turned the Dubstep down while playing, concentrating with that noise is hard):

Discovery through sound

Mapping brain waves into live music is a challenging task and while we can control music through an EEG machine that control is hard since we are using the brain to do many other things. What is interesting in the path of this experiment is not in fact the music generated but the use of sound to provide a way to hear the differences in datasets.

Hearing the difference between play Go or sleeping, between young people or old people.

Sound as a means of discovering patterns is a largely untapped source.

Building Clojure Services at Scale

At SoundCloud I’ve been experimenting over the last year with how we build the services that power a number of heavily loaded areas across our site. All these services have been built in Clojure with bits of Java tacked on the sides. Here are some of my personal thoughts on how to build Clojure services:

Netflix or Twitter

choose your own adventure

At some-point when you require a sufficient level of scaling you turn to the open source work of Twitter with Finagle or Netflix with Hystrix/RxJava. Netflix libs are written in Java while Twitters are written in Scala. Both are easy to use from any JVM based language but the Finagle route will bring in an extra dependency on Scala. I’ve heard little from people using interop between Clojure & Scala and that extra Scala dependency makes me nervous. Further I like the simplicity of Netflix’s libs and they have been putting a lot of effort into pushing support for many JVM based languages.

Hence with Clojure, Netflix projects are my preference. (I should add we do use Finagle with Scala at SoundCloud as well).

Failure, Monitoring & Composition Complexity

In a service reliant on other services, databases, caches any other external dependencies its a guarantee at some-point some of those will fail. When working with critical services we want to gracefully provide a degraded service.

While we can think about degrading gracefully in the case of failure, ultimately we want to fix wants broken as soon as possible. An eye into the runtime system allows us to monitor exactly whats going on and take appropriate action.

Your service needs to call other services. Dependent on those service results you might need to call other services which in turn might require calls to other services. Composing service calls is hard to get right without a tangle of complex code.

Fault Tolerance

How should we build fault tolerance into our Clojure services?

Single thread pool

Consider you have this line within a service response:

1
{:body @(future (client/get "http://soundcloud.com/blah/wah")) :status 200}

Now http://soundcloud.com/blah/wah goes down and those client requests start getting blocked on the request. In Clojure all future calls acquire a thread from the same thread pool. In our example the service is blocked up, is pilling new requests onto the blocked pool and we are in trouble.

My first solution to this problem was to introduce circuit breakers (https://github.com/josephwilk/circuit-breaker). I also stop using @ to dereference futures and used deref http://clojuredocs.org/clojure_core/clojure.core/deref which supports defaults and timeouts.

1
2
3
4
5
6
7
8
9
10
11
(defncircuitbreaker :blah-http {:timeout 30 :threshold 2})

(def future-timeout 1000)
(def timeout-value nil)

(defn http-get [url]
  (with-circuit-breaker :blah-http {
    :connected (fn [] (client/get "http://soundcloud.com/blah/wah"))
    :tripped (fn [] nil)}))

{:body (http-get http://www.soundcloud.com/blah/wah) :status 200}

Problem solved, now even though the thread pool may become blocked we back off the following requests and avoid pilling more work onto the blocked thread pool.

This worked pretty well, but then we decided we would to try and go even further in gracefully degrading. Why don’t we serve from a cache on failure, slightly stale data is better than none.

1
2
3
4
(defn http-get [url]
  (with-circuit-breaker :blah-http {
    :connected (fn [] (client/get "http://soundcloud.com/blah/wah"))
    :tripped (fn [] (memcache/get client url))}))

Now consider (client/get "http://soundcloud.com/blah/wah") starts failing, the thread pool gets blocked up, the circuit trigger flips and (memcache/get client url) is now fighting to get threads from the blocked thread pool.

Pants.

Scheduling over thread pools: Hystrix

Hystrix is Netflix library which I think of as circuit breakers on steroids.

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, 
services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems 
where failure is inevitable.

Dave Ray (http://darevay.com) has been doing lots of excellent work on producing Clojure bindings for Hystrix:

Hystrix gives me 2 big wins:

1. Separation of thread pools

Hystrix creates a separate thread pool for each Clojure namespace, if one thread pool becomes blocked due to a failure, a circuit breaker can be triggered with a fallback that uses a different thread pool.

This however does come with a cost:

  1. We have a performance hit due to moving to a scheduling based method for executing Hystrix commands.
  2. We cannot use Clojure’s concurrency primitives (futures/promises/agents).

Here is an example of our service rewritten with Hystrix:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
(ns example
  (:require [[com.netflix.hystrix.core :as hystrix]]))

(hystrix/defcommand http-get
  {:hystrix/fallback-fn (fn [url] (memcache-get url)}
  [url]
  (client/get url))

(hystrix/defcommand memcache-get
  {:hystrix/fallback-fn (constantly nil)}
  [url]
  (memcache/get client key))

(defn http-get [url]
   {:body (http/get "http://soundcloud.com/blah/wah") :status 200})

Beautiful, Just adding the defcommand brings us fault tolerance with no other changes to the shape of our code.

See the Hystrix Clojure adapter for all the possible configuration: https://github.com/Netflix/Hystrix/tree/master/hystrix-contrib/hystrix-clj

2. Monitoring

Hystrix supports exposing metrics on all circuit breakers within a process. It exposes these calls through an event stream.

How you expose that Hystrix event stream depends slightly on which web server you are using with your Clojure app.

Netty and Hystrix Event Streams (without servlets)

https://github.com/josephwilk/hystrix-event-stream-clj

1
2
3
(:require [hystrix-event-stream-clj.core as hystrix-event])

(defroutes app (GET "/hystrix.stream" (hystrix-event/stream))

Jetty and Hystrix Event Streams (with servlets)

If they are using Jetty you will need to change your app to add your main web app as a servlet. Then we can also add the Hystrix event stream servlet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
(ns sc-clj-kit.hystrix.jetty
  (:import [com.netflix.hystrix.contrib.metrics.eventstream HystrixMetricsStreamServlet])
  (:import [org.eclipse.jetty.server Server])
  (:import [org.eclipse.jetty.servlet ServletContextHandler])
  (:import [org.eclipse.jetty.servlet ServletHolder])
  (:import [org.eclipse.jetty.server.bio SocketConnector])
  (:import [org.eclipse.jetty.server.ssl SslSocketConnector])

  (:import (org.eclipse.jetty.server Server Request)
           (org.eclipse.jetty.server.handler AbstractHandler)
           (org.eclipse.jetty.server.nio SelectChannelConnector)
           (org.eclipse.jetty.server.ssl SslSelectChannelConnector)
           (org.eclipse.jetty.util.thread QueuedThreadPool)
           (org.eclipse.jetty.util.ssl SslContextFactory)
           (javax.servlet.http HttpServletRequest HttpServletResponse))
  (:require
   [compojure.core          :refer :all]
   [ring.adapter.jetty      :as jetty]
   [ring.util.servlet :as servlet]))

(defn run-jetty-with-hystrix-stream [app options]
  (let [^Server server (#'jetty/create-server (dissoc options :configurator))
        ^QueuedThreadPool pool (QueuedThreadPool. ^Integer (options :max-threads 50))]
    (when (:daemon? options false) (.setDaemon pool true))
    (doto server (.setThreadPool pool))
    (when-let [configurator (:configurator options)]
      (configurator server))
    (let [hystrix-holder  (ServletHolder. HystrixMetricsStreamServlet)
          app-holder (ServletHolder. (servlet/servlet app))
          context (ServletContextHandler. server "/" ServletContextHandler/SESSIONS)]
      (.addServlet context hystrix-holder "/hystrix.stream")
      (.addServlet context app-holder "/"))
    (.start server)
    (when (:join? options true) (.join server))
    server))

(defroutes app (GET "/hello" [] {:status 200 :body "Hello"})

(run-jetty-with-hystrix app {:port http-port :join? false})

Aggregation and discovery

While you can monitor a single process using Hystrix in our example we have many processes serving an endpoint. To aggregate all these Hystrix metric services we use Turbine.

Physical endpoints for a service at SoundCloud are discovered using DNS lookup. We configured Turbine to use this method to auto discover which machines are serving an endpoint.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
(ns sc-turbine.discovery
  (:import [org.xbill.DNS Type]
           [com.netflix.turbine.discovery InstanceDiscovery Instance])
  (:require [clj-dns.core :refer :all]))

(gen-class
   :name ScInstanceDiscovery
   :implements [com.netflix.turbine.discovery.InstanceDiscovery])

(defn -getInstanceList [this]
  (map (fn [instance]
         (Instance. (str (:host instance) ":" (:port instance)) "example-prod" true))
       (map (fn [answer] {:host (-> answer .getTarget str) :port (.getPort answer)})
            (:answers (dns-lookup "" Type/SRV)))))

And the config.properties:

1
2
3
InstanceDiscovery.impl=ScInstanceDiscovery
turbine.aggregator.clusterConfig=example-prod
turbine.instanceUrlSuffix=/hystrix.stream

Putting this all together our monitoring looks like this:

Pretty graphs: Hystrix Dashboard

Finally we run the Hystrix Dashboard and watch our circuit breakers live.

And heres an example with triggered circuit breakers:

Since I cannot show you the dashboard running, you will have to make do with music generated from the metrics. I normalize the live Hystrix metrics to piano pitches and play the notes as the arrive from the stream.

Hystrix Metrics as Sounds

Complexity & Composition

Working with many services, composition of service calls becomes hard to think and write about. Callbacks try to solve this but nested callbacks leave us with a mess.

RxJava tries to solve this using the Reactive Functional model. While RxJava provides lots of features I see it primarily as a way of expressing concurrent actions as a directed graph which provides a single callback on success or failure. The graph is expressed in terms or maps/reduces/filters/etc, things we are familiar with in the functional world.

To separate the request thread from the response thread we use RxJava with Netty and Aleph.

Here is a very simple example firing 2 concurrent requests and then joining the results into a single map response:

1
2
3
4
5
6
7
8
9
;;Hystrix integrates with RxJava and can return Observables for use with Rx.
(defn- find-user-observable [id] (hystrix/observe #'find-user id))

(defn- meta-data [user-urn]
  (let [user-observable (-> (find-user-observable id) (.map (λ [user] ...)))
        meta-observable (-> (find-user-meta-observable id) (.map (λ [subscription] ...))))
    (-> (Observable/zip user-observable
                        meta-observable
                        (λ [& maps] {:user (apply merge maps)}))))

The function meta-data returns an Observerable which we subscribe to and using Aleph return the resulting JSON to a channel.

1
2
3
4
5
6
7
8
9
(defn- subscribe-request [channel request]
  (let [id (get-in request [:route-params :id])]
    (-> (meta-data id)
        (.subscribe
          #(enqueue channel {:status 200 :body %}))
          #(logging/exception %))))))

(defroutes app
  (GET "/users/:id" [id] (wrap-aleph-handler subscribe-request)))

The shape of the RxJava Clojure bindings are still under development.

That single thread pool again…

With RxJava we are also in a situation were we cannot use Clojure’s future. In order for RxJava to block optimally we don’t want to use a single thread pool. Hence we use Hystrix as our means of providing concurreny.

Clojure services at scale

I’m very happy with the shape of these services running at SoundCloud. In production they are performing very well under heavy load with useful near realtime monitoring. In part thanks to Netflix’s hard work there is no reason you cannot write elegant Clojure services at scale.

Creating Sampled Instruments With Overtone

Overtone is an open source audio environment which uses Clojure and SuperCollider (an environment and programming language for real time audio synthesis and algorithmic composition).

Overtone makes it very easy to define instruments based on sound samples from freesound. We will walk through getting samples for a flute and then using them to define the flute instrument in Overtone.

The Flute

Getting the samples

Firstly we need to manually grab all the IDs from freesound. Sounds are often grouped into packs making this a little less painful.

We will be using the “Transverse Flute: Tenuto Vibrato C4-C7” pack.

A little Ruby script I use to produce a map of freesound ids to notes (which are usually in the filename of the sample):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#!/usr/bin/env ruby -w
require 'rubygems'
require 'json'

raise Exception.new("You must pass in the pack id!") unless ARGV[0]

pack_id = ARGV[0]
note_extractor = λ{|filename| filename.gsub(/Transverse-Flute |\.wav|Tenuto|NonVibrato|-/, "").gsub(/ NonVibrato/, "")}

data = JSON.parse(`curl --silent "http://www.freesound.org/api/packs/#{pack_id}/sounds/?api_key=47efd585321048819a2328721507ee23"`)
number_of_pages = data["num_pages"] || 1

number_of_pages.times do |page|
  data = JSON.parse(`curl --silent "http://www.freesound.org/api/packs/#{pack_id}/sounds/?api_key=47efd585321048819a2328721507ee23&p=#{page + 1}"`)
  sound_ids = data["sounds"].map{|sound| sound["id"]}
  note_names = data["sounds"].map{|sound| note_extractor.call(sound["original_filename"])}

  sound_ids.each_with_index do |sound_id, index|
    puts "#{sound_id} :#{note_names[index]}"
  end
end

Adding these to our Clojure code we have a Freesound ID to note mapping:

1
2
3
4
5
6
7
8
(def FREESOUND-VIBRATO-FLUTE-SAMPLES
  "Freesound ids for all samples in the Vibrato Transverse Flute pack"
  {154274 :C7  154273 :B6 154272 :A#6 154271 :A6 154270 :G#6 154269 :G6  154268 :F#6
   154267 :F6  154266 :E6 154265 :D#6 154264 :D6 154263 :C#6 154262 :C6  154261 :B5
   154260 :A#5 154259 :A5 154258 :G#5 154257 :G5 154256 :F#5 154255 :F5  154254 :E5
   154253 :D#5 154252 :D5 154251 :C#5 154250 :C5 154249 :B4  154248 :A#4 154247 :A4
   154246 :G#4 154245 :G4 154244 :F#4 154243 :E4 154242 :F4  154241 :D#4 154240 :D4
   154239 :C#4 154238 :C4})

These samples are cached in the asset store (usually ~/.overtone/assets) so we don’t have to keep downloading them. We define a cache key for our downloaded samples.

1
2
3
4
(defn- registered-samples
  "Fetch flute samples from the asset store if they have been manually registered"
  []
  (registered-assets ::TransverseFluteTenutoVibrato))

Load all the samples into Overtone. This will automatically download the samples if they are not already present.

1
2
(def flute-samples
  (doall (map freesound-sample (keys FREESOUND-VIBRATO-FLUTE-SAMPLES))))

In order to play the samples we need to produce a mapping between the buffer that has the sound loaded and the midi pitch that buffer relates to.

First we convert a freesound-id into a midi note (pitch)

1
2
(defn- buffer->midi-note [buf]
  (-> buf :freesound-id FREESOUND-VIBRATO-FLUTE-SAMPLES name note))

Now we can generate the buffer id to midi note map.

1
2
3
4
5
6
7
8
9
(defn- note-index
  "Returns a map of midi-note values [0-127] to buffer ids."
  [buffers]
  (reduce (fn [index buf]
            (let [id (-> buf :id)
                  note (buffer->midi-note buf)]
              (assoc index note id)))
          {}
          buffers))

And we use this to set the buffers:

1
2
3
4
5
6
7
8
9
10
;Silent buffer used to ensure all 127 buffer spaces are filled 
(defonce ^:private silent-buffer (buffer 0))

(defonce index-buffer
  (let [tab (note-index flute-samples)
        buf (buffer 128)]
    (buffer-fill! buf (:id silent-buffer))
    (doseq [[idx val] tab]
      (buffer-set! buf idx val))
    buf))

Defining the Instrument

Now we have all our samples we define a instrument in Overtone using definst:

This allows us to specify a (huge) number of options on how our samples are played.

1
2
3
4
5
6
7
(definst sampled-flute-vibrato
  [note 60 level 1 rate 1 loop? 0 attack 0 decay 1 sustain 1 release 0.1 curve -4 gate 1]
  (let [buf (index:kr (:id index-buffer) note)
        env (env-gen (adsr attack decay sustain release level curve)
                     :gate gate
                     :action FREE)]
    (* env (scaled-play-buf 2 buf :level level :loop loop? :action FREE))))

There is quite a lot going on here. Lets focus on the configuration that effects how our sound samples are played.

Defining the Envelope Generator (env-gen).

An envelope generator (mapping to function env-gen) makes an audio signal that smoothly rises and falls. We are taking the recording of the real flute instrument as a digitized waveform, and then playing back its recordings at different speeds to produce different tones.

The most common form of envelope generator and the one we are using here is ADSR:

attack (A), decay (D), sustain (S) and release (R)

There are a number of other config settings in our example:

1
2
3
4
5
6
7
8
9
10
11
12
[
 note 60     ; Default pitch
 level 1     ; Volume
 rate 1      ; Playback rate 
 loop? 0     ; The note should loop after the first play
 attack 0    ; The way a sound is initiated. Fast attack (gunshot) vs Slow attack (closing a door slowly)
 decay 1     ; The time for notes to decay after the initial strike
 sustain 1   ; Once a sound has reached its peak, the length of time that the sound will sustain.
 release 0.1 ; The time for notes to decay after the key is released
 curve -4    ; Curve factor
 gate 1      ; note occurs when gate goes from nonpositive to positive. Note off occurs when it goes from positive to nonpositive
]

Look to the SuperCollider documentation on UGENs if you want to see all possible configuration options. Overtone is really a wrapper around the extremely powerful SuperCollider.

Grand finale

Finally we can start to play music in overtone using our flute:

1
2
3
(sampled-flute-vibrato 60) #=> pitch 60
(sampled-flute-vibrato 61) #=> pitch 61
(sampled-flute-vibrato 60) #=> pitch 60

Hear for yourself:

The Code in Full

Defining an instrument in Overtone:

Loading buffers
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
(ns overtone.samples.flute-vibrato
  (:use [overtone.core])
  (:require [clojure.string :as str]))

(defn- registered-samples
  "Fetch flute samples from the asset store if they have been manually
  registered"
  []
  (registered-assets ::TransverseFluteTenutoVibrato))

(def FREESOUND-VIBRATO-FLUTE-SAMPLES
  "Freesound ids for all samples in the Vibrato Transverse Flute pack"
  {154274 :C7  154273 :B6 154272 :A#6 154271 :A6 154270 :G#6 154269 :G6  154268 :F#6
   154267 :F6  154266 :E6 154265 :D#6 154264 :D6 154263 :C#6 154262 :C6  154261 :B5
   154260 :A#5 154259 :A5 154258 :G#5 154257 :G5 154256 :F#5 154255 :F5  154254 :E5
   154253 :D#5 154252 :D5 154251 :C#5 154250 :C5 154249 :B4  154248 :A#4 154247 :A4
   154246 :G#4 154245 :G4 154244 :F#4 154243 :E4 154242 :F4  154241 :D#4 154240 :D4
   154239 :C#4 154238 :C4})

(def FLUTE-SAMPLE-IDS (keys FREESOUND-VIBRATO-FLUTE-SAMPLES))

(def flute-samples
  (doall (map freesound-sample FLUTE-SAMPLE-IDS)))

(defn- buffer->midi-note [buf]
  (-> buf :freesound-id FREESOUND-VIBRATO-FLUTE-SAMPLES name note))

(defn- note-index
  "Returns a map of midi-note values [0-127] to buffer ids."
  [buffers]
  (reduce (fn [index buf]
            (let [id (-> buf :id)
                  note (buffer->midi-note buf)]
              (assoc index note id)))
          {}
          buffers))

;; Silent buffer used to fill in the gaps.
(defonce ^:private silent-buffer (buffer 0))

(defonce index-buffer
  (let [tab (note-index flute-samples)
        buf (buffer 128)]
    (buffer-fill! buf (:id silent-buffer))
    (doseq [[idx val] tab]
      (buffer-set! buf idx val))
    buf))
Defining the instrument
1
2
3
4
5
6
7
8
9
10
11
12
13
(ns overtone.inst.sampled-flute
  (:use [overtone.core])
  (:require
   [overtone.samples.flute-vibrato :as vibrato]))

(definst sampled-flute-vibrato
  [note 60 level 1 rate 1 loop? 0
   attack 0 decay 1 sustain 1 release 0.1 curve -4 gate 1]
  (let [buf (index:kr (:id vibrato/index-buffer) note)
        env (env-gen (adsr attack decay sustain release level curve)
                     :gate gate
                     :action FREE)]
    (* env (scaled-play-buf 2 buf :level level :loop loop? :action FREE))))

Sets in Elixir

I recently contributed the Set data structure to the Elixir programming language. The Set is implemented through a HashSet.

1
2
3
4
s1 = HashSet.new([1, 2, 3, 4])
s2 = HashSet.new([3, 4])

Set.difference(s1, s2) # => HashSet<[1, 2]>

Elixir’s Set implementation is often faster than Erlangs own sets. Which is pretty impressive since Elixir runs on the same Erlang VM. It does this by adapting the internal data structure based on the size of the sets.

Lets explore how Elixir does this while also explaining a little about Elixir:

Growing Data structures

For small sets the internal representation uses an ordered set (which really is just a list).

We refer to this as a Bucket.

1
[1, 2, 3, 4]

Being ordered allows us to perform faster access and modification.

For example finding a member of a set:

1
2
3
# Only needs a single comparison to know if 1 is in the Set.
# Since the first element is greater than 1 we know its not in the Set.
Set.member?(HashSet.new [2, 3, 4, 5], 1)

Order in the list is not maintained in the data structure but in the insertion/put:

Putting elements into a Bucket

Before we jump into some Elixir code its important to understand that the order of functions in Elixir is important. We start from the top function and if this function matches a criteria we call it otherwise we try the function below and so on. If we find no function that matches that’s a error and execution aborts.

Here is the Elixir code for inserting a member into a bucket.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# We have found the place to insert the new member.
defp bucket_put([m|_]=bucket, member) when m > member do 
  { [member|bucket], 1 }
end

# Member is already in the set, we don't added it. (notice how member is being used to pattern match here).
# Its the same as writing: `defp bucket_put([m|bucket], member) when m == member do
defp bucket_put([member|bucket], member) do
  { [member|bucket], 0 }
end

# Recursive case, keep calling bucket_put as this is not the point to insert.
defp bucket_put([e|bucket], member) do
  { rest, count } = bucket_put(bucket, member)
  { [e|rest], count }
end

# Empty set, insert the number
defp bucket_put([], member) do
  { [member], 1 }
end

Expanding into a Trie

Upon reaching a threshold the Set changes the internal representation into a Trie (What the hell is a trie).

Why a Trie?

For a Trie the time required for insert, delete and find operations is almost identical. As a result, for situations where code is inserting, deleting and finding in equal measure, tries can handily beat binary search trees.

In a Trie we don’t store any keys or hashes, instead a elements position in the tree defines the key with which it is associated.

Hence we can take advantage of Erlangs fast tuples to model the trie. A multi-depth trie is just tuples inside tuples.

Buckets growing into Tries

Lets work through an example. For simplicity we will assume the expansion happens at the very small size of 5 elements:

1
2
3
4
set = Set.new [0, 1, 2, 3]

# Apply an operation that forces the internal structure to change
set = Set.put(set, 4)

Create a new Erlang Tuple

bucket to trie

1
  :erlang.make_tuple(node_size = 4, [])

Redistribute the bucket elements into the Tuple

bucket to trie

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# This returns the index that a set value should have in the tuple.
defp bucket_nth_index(set_value, trie_depth) do
  node_shift = 2
  node_bitmap = 0b0011

  (:erlang.phash2(set_value) >>> (node_shift * trie_depth)) &&& node_bitmap
end

# Create a new node or add to an existing node.
# Based on the bucket_nth_index of the set member put it into a bucket at that index in the tuple.
defp node_relocate(node // :erlang.make_tuple(node_size = 4, []), bucket, depth) do
  :lists.foldl fn member, new_node ->
    position_in_tuple = bucket_nth_index(member, depth)
    set_elem(new_node, position_in_tuple, bucket_put(elem(new_node, position_in_tuple), member))
  end, new_node, bucket
end

Finally add the new set member into the Trie

bucket to trie

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Base case. We have found the right bucket, just insert the new member into it.
defp node_put(node, 0, member) do
  position = bucket_index(member)
  { new, count } = bucket_put(elem(node, position), member)
  { set_elem(node, position, new), count }
end

# Recursive case. Keeping trying to find the right bucket.
defp node_put(node, depth, member) do
  position = bucket_index(member)
  { new, count } = node_put(elem(node, position), depth - 1, bucket_next(hash), member)
  { set_elem(node, position, new), count }
end

# The bucket index of this member
defp bucket_index(member) do
  node_bitmap = 0b0011
  hash = :erlang.phash2(member)
  hash &&& node_bitmap
end

# The bucket index of this element with depth + 1.
defp bucket_next(member) do
  node_shift = 2
  hash = :erlang.phash2(member)
  hash >>> node_shift
end

Notice while we have a more complicated data structure with a trie the leaves are always plain old buckets. Hence insertion with a trie is a case of finding the right bucket and then just reusing our bucket_put function. Beautiful and fast :)

Tries expanding into deeper Tries

Things get a little more complicated with multi depth tries. If you are interested digging more into the implementation you can see all the source code of HashSet on Github.

Contraction

We can also remove elements from a set so in much the same as we expanded we have to contract the internal representation of the set.

Performance Benchmarks

Here is some sample data comparing Erlang sets to Elixir Sets. Smaller is better.

HashSet vs :sets

The benchmark scripts are on github (https://gist.github.com/josephwilk/5808525) if you are interested in running them yourself.