Joseph Wilk

Joseph Wilk

Things with code, creativity and computation.

Visuals With Overtone and Shadertone

Exploring techniques for creating live coded performances with Overtone and OpenGL Fragment Shaders. Much learnt from my work performing as Repl Electric. All my shader source code for these performances is open: https://github.com/repl-electric/cassiopeia/tree/master/resources/shaders

shaders

To bring OpenGl to Clojure I use Shadertone written by rogerallen. This utilises LWJGL (Java Light Weight Java Game Library https://www.lwjgl.org).

The Bridge between Clojure and Shaders

A vital feature of Shadertone is a map between Clojure atoms and shader Uniforms. What is a shader Uniform? Well think of it as a read-only global variable in your shader. A Clojure watcher ensures any updates to your Clojure atom persist into your Uniform. A little clunky but all uniforms start with the letter i.

The shader:

1
uniform float iExample;

And in Clojure

1
2
3
4
5
6
(def example-weight (atom 0.5))
(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data {"iExample" example-weight})

;;iExample Uniform will also be updated.
(reset! example-weight 0.2)

Live editing shaders

When a shader file is edited Shadertone is watching the file (using watchtower) and will reload/recompile the changed file. This results in a slight freeze as the new code is run (This might be down to my graphics card). Hence most of the time I prefer alternatives to live editing the shader to create smoother transitions.

Injecting movement

To make static images move we need a continuously changing value.

Shadertone gives us iGlobalTime using the number of seconds since the shader was started:

1
2
3
4
5
6
uniform iGlobalTime

void main(void){
  //Use the continuously changing time signal as the value for a color.  
  gl_FragColor = vec4(sin(iGlobalTime)*0.5+0.5);
}

Putting a continuously changing value through a function like sin/cos is the bread and butter of creating animations with shaders.

Randomness

We often need a cheap and fast way to generate random floats in Shaders. Without persistent state and preservation of a seed it can be difficult. One solution is to use a noise image and the current pixel coordinates as an index into the image for a float value.

Shadertone supports loading custom textures into your shaders:

1
2
(shadetone/start "shaders/example.glsl"
         :textures ["/josephwilk/textures/noise.png"])

The noise texture:

And finally the shader:

1
2
3
//Turning the current pixel coordinates (uv) into a random float. 
vec2 uv = gl_FragCoord.xy / iResolution.xy;
return texture2D(iChannel0, vec2(uv)/256.0, 0.0);

Composing visual effects

I attach a weight to each function or visual phase of the shader. Through this we can select which visual effect is visible or combine multiple effects. Its a bit messy, since I have all my functions in a single shader file. I’ve not explored including of external files with shaders.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
uniform float iCircularWeight;
uniform float iPopulationWeight;

vec4 circular(){...}
vec4 population(){..}

void main(void){
  vec4 circleResult     = vec4(0.0);
  vec4 populationResult = vec4(0.0);
  if(iCircularWeight > 0.0){
    circleResult = circular() * iCircularWeight;
  }
  if(iPopulationWeight > 0.0){
    populationResult = population() * iPopulationWeight;
  }
  gl_FragColor = (populationResult + circleResult);
}

And within Clojure:

1
2
3
4
5
6
(def circular-w (atom 1.0))
(def population-w (atom 1.0))
(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data {"iCircularWeight" circular-w
              "iPopulationWeight" population-w})
(reset! circular-weight 1.0)

Synchronisation

Shadertone uses the seconds since start (iGlobalTime) while Overtone via Supercollider uses the soundcard’s clock. Hence there is no guarantee these two sources will be in sync.

Replacing iGlobalTime is the only option. We create a special synth called data-probes which sole function is to transfer data from the Supercollider world to the Clojure world. Overtone provides a Supercollider to Clojure binding called a tap. We add a tap into our Overtone synth which is polling our global timing signal (this powers all synths and is how we co-ordinate everything).

1
2
3
4
5
6
7
8
9
10
11
(defsynth data-probes [timing-signal-bus 0]
  (let [beat-count (in:kr timing-signal-bus)
        _  (tap "global-beat-count" 60(a2k beat-count))
      (out 0 0))

(def active-data-probes (data-probes (:count time/beat-1th)))

(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data
  ;;An atom wrapping the tap and the running synth instance
   "global-beat-count" {"iGlobalBeatCount" (atom {:synth active-data-probes :tap "global-beat-count"})})

Using our iGlobalBeatCount in our shader now means anything requiring a continuously increasing value flows to our beat.

Shaders & global mutable state

Persistent mutable state between executions is not possible in OpenGL Shaders. Uniforms are read-only.

Lets look at an example. On a drum hit I want the color of a visual to change and persist until the next drum hit. The drum signal is 1 for a hit 0 for silence:

1
[1 0 0 0 1 0 0 0 1 0 0 0]

The current value based on the global clock is passed into the Shader as the iBeat Uniform.

1
2
3
4
5
6
7
8
9
uniform float iBeat;
float color = 0.0;

vec4 function showColor(){
  if(iBeat == 1.0){
    color += 1.0;
  }
  return vec4(color);
}

Will return:

1
2
3
4
5
vec4(1.0);
vec4(0.0);
vec4(0.0);
vec4(0.0);
vec4(1.0);

What we were after is actually:

1
2
3
4
5
vec4(1.0);
vec4(1.0);
vec4(1.0);
vec4(1.0);
vec4(2.0);

My solution is to move to Clojure where mutable state using atoms is simple.

Our timing is guided by Supercollider and a global clock. The value of our kick buffer at anyone time is only known inside the synth and hence inside Supercollider. But if we want to have mutable state we need access to this value in Clojure. So we create a custom synth that taps the value of the kick buffer based on the global clock signal.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(defsynth drum-data-probe [kick-drum-buffer timing-signal-bus 0]
  (let [beat-count (in:kr timing-signal-bus)
    drum-beat (buf-rd:kr 1 kick-drum-buffer beat-count)
    _  (tap "drum-beat" 60 (a2k drum-beat))])
  (out 0 0))

(defonce kick-drum-buffer (buffer 256))

;;Create the synth with the "drum-beat" tap
(def drum-data-probe (drum-data-probe kick-drum-buffer (:count time/beat-1th)))

;;Bind the running synth and the tap
(def kick-atom (atom {:synth drum-data-probe :tap "drum-beat"}))

;;Extract the tap atom
(def kick-tap (get-in (:synth @kick-atom) [:taps (:tap @kick-atom)]))

Now in the Clojure world its simple to watch our tap atom and hence get alerted when it changes value. Overtone is dealing with the magic of updating the atom under the covers, the watcher is a nice implementation independent way of hooking into this. We now know the value of our kick buffer in Clojure. If we use another atom as our accumulator we can update it when the tap atom changes. Finally pushing this new accumulator to the Shader.

1
2
3
4
5
6
7
8
9
(def active-color (atom 0.0))

(add-watch kick-tap :cell-color
  (fn [_ _ old new]
    (when (and (= old 0.0) (= 1.0 new))
    (reset! active-color (mod (+ @active-color 1.0) 100)))))

(shadertone/start-fullscreen "resources/shaders/example.glsl"
  :user-data {:iActiveColor active-color}

Within the shader:

1
2
3
4
5
uniform float iActiveColor;

vec4 function showColor(){
  return vec4(iActiveColor);
}

Thats a lot of work, but I’m very happy with the results in my (end-of-buffer) performance.

Buffer events

Writing to a buffer is a common way of live coding in Overtone. Its very useful to attach some visual effect based on the settings of a buffer.

1
2
3
;;Setting notes to a buffer
(def notes-buf (buffer 256))
(pattern! notes-buf (degrees-seq [:f3 1314]))

We could put a tap into the synth and grab the current note and pass this into the shader. As I’ve mentioned taps are expensive and they are always on while we may not always be using them. This also gets more complicated when say we have 3 instances of the same synth running playing simultaneous to form a chord.

An alternative is to invent an atom which is used as a signal on every buffer write.

1
2
3
4
5
6
;;Could also do this with OSC messages...
(do
  (defonce buffer-change-event-notes-buf (atom 0.0))
  (pattern! notes-buf (degrees-seq [:f3 1314]))
  (swap! buffer-change-event-notes-buf + 1)
)

And adding a watcher

1
2
3
4
5
6
7
8
9
10
11
12
(add-watch
  buffer-change-event-notes-buf
  :buffer-change-event-notes-buf
  (fn [& _]
    (let [n (int (buffer-get notes-buf 0))]
      (case n
        29 (reset! color 0.4)
           (reset! color 0.0)))))

(def buffer-atoms (atom {}))
(pattern! notes-buf)
(reset! buffer-atoms (assoc @buffer-atoms (:id notes-buf) (inc (@buffer-atoms (:id notes-buf)))))

I use this trick in (end-of-buffer) to use the bass note to control the level of distortion of the visuals (source). Its wonderful to focus on the notes and feel the visuals following you automatically.

Midi notes to visuals

I often want to map a midi note to a visual effect. All my notes are mapped to buffers. Much like we did with the drums I can use a tap to get access to the current note being played in a buffer. Skipping over the details, when we have a midi note we send it as an float (to support crazy 42.333 like notes) to the shader via an atom.

We then map it to a nice range value to effect visuals:

1
2
3
4
5
6
7
uniform float iLeadNote;

//midi note: 42 => F#2  
//midi note: 78 => F#5
float noteMapped = smoothstep(42.0, 78.0, iLeadNote);

//noteMapped now spans => 0..1

A cheap way to scale effects based on the height of the note.

Gradual transitions

Often I want a smooth fading it or out of a shader function. Say for example fading to black. Pretty simple, just fire a thread which sleeps and ticks an atom. The atom is fed into the Shader.

1
2
3
4
5
uniform float iColor;

void main(void){
  gl_color = vec4(iColor);
}

Since I use this a lot I created a helper fn in MUD:

1
2
3
(def color (atom 1.0))
;;         atom / target /  rate
(overtime! color   0.0      0.001)

Text

In end-of-buffer I spell the word Repl Electric out of floating lights. We are bound to only a few data structures with fragment Shaders. I used a simple 3x3 matrix mapping each part of a character. Then using this to decided the position of the lights.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
const mat3 LETTER_R        = mat3(1, 1, 1,
                                  1, 1, 0,
                                  1, 0, 1);
const mat3 LETTER_E        = mat3(1, 1, 1,
                                  1, 1, 0,
                                  1, 1, 1);

vec4 letter(mat3 letter, vec2 offset, vec2 uv){
  vec2 point = vec2(0,0);
  vec4 helloPoint = vec4(0,0,0,0);
  vec3 xPos = vec3(0.01, 0.03, 0.05);
  vec3 yPos = vec3(0.05, 0.03, 0.01);

  for(int y=0; y < 3; y++){
    for(int x=0; x < 3; x++){
      if(letter[y][x] == 1){// Show this part of the letter
        point = vec2(xPos[x]+offset.x, offset.y+yPos[y]);
        helloPoint += buildCell(uv, point, STATIC_LETTERS);
      }
    }
  }
  return helloPoint;
}

letter(LETTER_R, 0.2, uv);

And the visual.

Font example

Visuals effected by frequencies

Shadertone provides a 2x512 array with the frequency spectrum (FFT) and audio waveform data. It does this by loading the data into a 2D Texture. The audio data is taken from tapping the main Overtone audio bus.

1
2
;;Tell Shadertone to fill iChannel0 with audio data
(shadetone/start "shaders/example.glsl" :textures [:overtone-audio])

It’s always a challenge to utilise this without creating something jerky or causing motion sickness. Hence I tend to use the waveform or FFT as a distorter rather than a base for animations.

It also helps to concentrate on specific ranges of frequencies of the waveform data to create a stronger connection between a synth and the visuals.

1
2
3
4
5
6
7
8
9
10
float sound = texture2D(iChannel0, vec2(max(uv.x,0.9),.75)).x;

//uv.xy => current x,y coordinates of pixel.

//First argument is an index into the 512 values of the waveform.
//By limiting the first argument we can ignore certain ranges.

//Second argument selects:
//    0.25 => FFT
//    0.75 => audio waveform

Here is an example where I use the audio waveform to distort the scale & jaggedness of a series of circle shapes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
const float tau = 6.28318530717958647692;
vec3 wave = vec3(0.0);
float width = 4.0/500;
for (int i=0; i < 60; i++){
  float sound = texture2D(iChannel0, vec2(uv.x,.75)).x;

  float a = 0.1*float(i)*tau/float(n);
  vec3 phase = smoothstep(-1.0,.5, vec3(cos(a), cos(a-tau/3.0), cos(a-tau*2.0/3.0)));
  wave += phase * smoothstep(width, 0.0, abs(uv.y - ((sound*0.9)+0.2)));

  //This shift of uv.x means our index into the sound data also 
  //moves along, examining a different part of the audio wave. 
  uv.x += 0.4/float(n);
  uv.y -= 0.05;
}
wave *= 10.0/float(n);
return vec4(wave,1);

And the resulting visual:

FFT example

Final thoughts on Live coding visuals

Through Clojure’s binding of atoms with Fragment shaders we have the power to live code visuals and music. Though it comes at a cost of complexity having to wrap lots of functions in order to have powerfully connected visuals. Fragment shaders are extremely terse, and can be pushed to replicate many advanced effects but they are also performance intense, and often taking a non-shader route will be much more performant.

Stability

My knowledge of LWJGL is small, but crashes in the fragment shaders often occur leaving the JVM wedged. This has happened to me quite a lot practicing, but never in a performance. Its worth reflecting that something (be it fixable) leaves a risk of a freeze in a performance.

Combining State & Shaders

I’ve started to explore what a shader application might look like if it was a server and provided a state machine so the live coding language does have this complexity. In turn producing a freer and more spontaneous interaction. This project is Shaderview and steals all the good ideas of Shadertone while adding some new features like vertex shader art. I’ll be writing up more about Shaderview soon.

Comments