Joseph Wilk

Joseph Wilk

Things with code, creativity and computation.

Live Coding - Repl Electric

Live coding is the act of turning a programming session into a performance. This can constitute improvisation, music, visuals, poetry, hardware, robots, dance, textiles and people. Pretty much anything with an input and output can be controlled live by programming.

This is not just a performance by programmers for programmers. While this is often where it starts as a live coder, the type of audience and the accessibility of the performance lies in the performers imagination. Abstraction can get us pretty much anywhere.

1

(def the-stars (dark-matter))

Repl Electric

Repl Electric is a project I started in order to discover more about music composition and Artificial intelligent based aids to creativity. Which in turn through the inspiration of people like Meta-ex lead me to live programming music.

Here is a performance live coding music and graphics, inspired by a performance in London:

The Stars

Open Live Coding

All the tools and code used to create this performance are open for all to see on Github: https://github.com/repl-electric

Three programming languages were used to create this piece:

  • Clojure (Sound)
  • GLSL (Visuals)
  • Emacs Lisp (Animations & Navigation)

Tools

Here are the tools used and a little detail around how they where used in performing “The Stars”:

Clojure: http://clojure.org

Clojure is a LISP language based on the JVM.

Clojure focuses on interactive REPL (Read, Evaluate, Print & Loop) driven development. Which makes it a good choice for interactively coding music. It also turns out functional programming is a good fit for operating over music as data.

Emacs Live: https://github.com/overtone/emacs-live

Emacs live is a Emacs release with packages and defaults that are Live Coding centric. Something I use for both for my work and for my live coding.

To execute our code, we launch a repl instance in our project (NEVER launch inside emacs, since then if emacs crashes the repl and music dies) and connect to it from emacs using cider https://github.com/clojure-emacs/cider.

A simple trick to combine Emacs code and visualizations is to launch an OpenGL window in full screen (see Shadertone) and then put a full screen transparent terminal window running emacs over it.

The tumbling text effect seen at the end of the performance is an emacs animation using Zone Mode which supports writing your own text destructors: http://www.emacswiki.org/emacs/ZoneMode

Overtone: https://github.com/overtone/overtone

Overtone is a Clojure based client to SuperCollider. Supercollider is an environment for real time audio synthesis and algorithmic composition.

Overtone provides us with:

  • Timing (beat generation – example timing code).
  • Building Synths (engineer sound).
  • Running samples (both your own and from Freesound).
  • Live Synth control (changing notes, durations, reverb, etc).
  • Hardware interaction (through midi or OSC).

An example of a synth used in The Stars:

1
2
3
4
5
6
7
8
9
10

(use 'overtone.live)
(defsynth dark-ambience [out-bus 0 amp 1 mul 0.2 room-size 70 rev-time 99 freq 60 ring-mul 55]
  (let [pink (hpf:ar (* (* 0.005 (pink-noise)) (line:kr 0 1 9)) 5)
        src1 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 0)) mul)
        src2 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 1)) mul)
        src3 (ringz (* pink (lf-noise1:kr 0.15)) (+ freq (* ring-mul 2)) mul)
        src (tanh (g-verb (sum [src1 src2 src3]) room-size rev-time))]
    (out out-bus (* amp src))))

(def the-stars (dark-ambience))
Timing

Timing is a complicated issue but so important its worth touching on. You have a choice with Overtone to use Java for timing or Supercollider. I use Supercollider since I have found it to be much more reliable. Everything you need is here (copy and paste), thanks to the hard work of Sam Aaron.

The key concept to take away is there are two types of timing, a beat counter which is forever incrementing and a beat trigger which flips back and forth between 1/0.

1
2
3
4
5
6
7

(require '[cassiopeia.engine.timing :as time])

;;The beat trigger
(:beat time/main-beat) ;=> 0,1,0,1,0,1,0

;;The beat counter
(:count time/main-beat) ;=> 0,1,2,3,4,5,6,7,8,9

The counter is useful for indexing buffers, the trigger is useful in controlling the gate of an envelope (which turns a sound on or off).

In Clojure we can still get access to the beat, in our timing code we send a message using send-trig on every beat. We can hook a Clojure function to callback on this beat:

1
2
3
4
5
6

(on-trigger (:trig-id time/main-beat)
  (fn [beat-no]
    (when (= 0.0 (mod beat-no 32))
      ;;play a sample
      (boom-s)))
  ::on-beat-trigger)

I use this extensively to time graphic transitions with the music.

Buffers

Most of my live coding performance was writing to buffers which are hooked into synths. Buffers are just fixed size arrays but they are stored in Supercollider rather than in Clojure. Here is an example from The Stars where the midi notes are read from a buffer at a rate based on my beat timing signal (a 16th of the main beat here).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

(use 'overtone.live)
(use 'cassiopeia.engine.core)
(require '[cassiopeia.engine.timing :as time])

(defsynth growl [out-bus 0 note-buf 0 beat-bus 0 beat-trg-bus 0 amp 1]
  (let [cnt (in:kr beat-bus)
        trg (in:kr beat-trg-bus)
        note (buf-rd:kr 1 note-buf cnt)
        freq (midicps note)
        vol (> note 0)

        e (env-gen (perc :attack 10 :sustain 1 :release 1) :gate trg)
        src (lpf (mix [(saw (* 0.25 freq))
                       (sin-osc (* 1.01 freq))]))
        src (pitch-shift src 0.4 1 0 0.01)
        src (pan2:ar (* vol amp e src))]
    (out out-bus src)))

(defonce nebular (buffer 96))

(def nebula (growl :note-buf nebula-note-buf :beat-trg-bus (:beat time/beat-16th) :beat-bus (:count time/beat-16th)))

(pattern! nebula-note-buf (degrees [1 3 7] :major :A2))

GLSL + Shadertone: https://github.com/overtone/shadertone

Shaders generate imagery directly on your Graphics Processing Unit rather than going through your CPU. Through a language called GLSL (which is C like) we can express very simple functions which get called on every single pixel generating complex visuals. Here is a simple extract from The Stars that generates all the background small dots:

1
2
3
4
5
6
7
8
9
10
11
12

void main(void){
  vec2 current_pixel_position = mod(gl_FragCoord.xy, vec2(5.0)) - vec2(0.0);
  float distance_squared = dot(current_pixel_position, current_pixel_position);

  vec4 black = vec4(.0, .0, .0, 0.0);
  vec4 white = vec4(1.0, 1.0, 1.0, 1.0);

  //Test the current pixel position and if it should be a circle shade it.
  vec4 circles = (distance_squared < 0.6) ? white : black;

  gl_FragColor = circles;
}

For more examples of whats possible with Shaders checkout Shader Toy

Shadertone is the Clojure library that provides a convenient way of running shaders from Clojure and for feeding in data about our synths. It provides access in your Shader to:

  • Overtone’s Volume (iOvertoneVolume)
  • The frequency spectrum & audio waveform data (Passed as a 2D texture :textures [:overtone-audio])

To synchronize the graphics with the music I created a special Overtone synth which does not generate any sound, it instead feeds information in realtime to my shader.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

(use 'overtone.live)
(require '[shadertone.core :as t])

;;A synth that exposes through taps all the lovely timing information.
(defsynth buffer->tap [beat-buf 0 beat-bus 0 beat-size 16 measure 6]
  (let [cnt (in:kr beat-bus)
        beat (buf-rd:kr 1 beat-buf cnt)
        _  (tap "beat"          60 (a2k beat))
        _  (tap "beat-count"    60 (a2k (mod cnt beat-size)))
        _  (tap "measure-count" 60 (a2k (/ (mod cnt (* measure beat-size)) measure)))])
  (out 0 0))

;;; Used to store our drum beat, 1 for a hit 0 and for a miss
(defonce drum-sequence-buffer (buffer 256))

(def beats (buffer->tap drum-sequence-buffer (:count timing/main-beat)))

;;Open a OpenGL window running our shader
(t/start-fullscreen "resources/shaders/electric.glsl"
                    :user-data {
                    "iBeat"         (atom {:synth beats :tap "beat"})
                    "iBeatCount"    (atom {:synth beats :tap "beat-count"})
                    "iMeasureCount" (atom {:synth beats :tap "measure-count"})})

Inside our shader code:

1
2
3
4

uniform float iBeat;
uniform float iBeatCount;
uniform float iMeasureCount;
...

The other main way of controlling a shader from Clojure is using atoms.

1
2
3
4
5
6
7

(require '[shadertone.core :as t])

(defonce cellular-growth (atom 0.0))

(t/start-fullscreen "resources/shaders/electric.glsl" :user-data {"iCellularGrowth" cellular-growth})

(swap! cellular-growth + 0.01)

Hardware: Monome: http://monome.org

Something you don’t see in the video is that I’m using a 8x16 Monome. For this performance its primary function was a visual aid to show the beat/measure information.

Monome

The hardware is driven by Clojure communicating with the Monome through a serial port: https://github.com/josephwilk/monome-serial/tree/protocols

Live Coding

Live coding music and graphics combines skills in sound engineering, 3d graphics, geometry, physics, musical theory, composition, improvisation & hardware to name a few.

It is difficult, and requires a lot of work and practice.

But of all the code I’ve written over the years this is one of the things I’m most proud of. And i’m only at the beginning of discovering what’s possible.

Comments