The Chaos Machine - Synthesizing Temperature Measurements into Sound
tl;dr
Want to learn something about arduino, midi audio or sound synthesis? Using python, R and many other tools. Or you just have some spare minutes to watch and listen to and read about a project where live data is translated to sound? This article shows how the Chaos Machine produces sound like this:
Introduction
This is an introductory blog post with a strong focus on documentation and some of the relevant topics if you want to construct such a machine. The cool thing is that all the software used is free. The only money spent was on the hardware (a circuit of a microcontroller and some thermometers), which is not very expensive. All the software (microcontroller, programming, data management and audio-related) is completely free. Furthermore, all the code used in this project is open source.
I will give an overview over the Chaos Machine code repository. This repository is the result of the artist and my friend Axel Crettenand having the idea of his project called CHOEUR AQUATIQUE (the Underwater Choir) and hiring me to develop the code.
As we live in different cities and traveling is difficult due to Covid-19 we decided it’s best to make a public repository with the instructions for Axel to reconstruct the machine himself. And with the possibility that somebody else is interested I was more motivated to write a good documentation. :) This article is to briefly introduce you to the project.
To get an idea how the machine sounds so far you can have a listen to the generated mp3 file in this folder (or scroll down to the audio player at the bottom of this article). This file is the synthesization of the midi file you can also find there.
Technical description of the chaos machine
Basically, the machine translates temperature measurements into sound in real time. The arduino code sketch uploaded to the microcontroller reads the measured temperatures of multiple thermometers and then sends them to the computer. On the PC a python program (see here) runs a main loop where the sent temperature data is again read in. The temperature differences are then translated into midi notes of a chosen scale. The midi nodes are then sent to fluidsynth which finally synthesizes the midi notes to sound. This will be referred to as the chaos machine.
Main steps in the chaos machine
This is an interactive diagram1
- hover (mouse-over) to see description
- click on clickable nodes (not all) to view source code file
The main interfaces are the serial connection between python and the arduino on the one hand. On the other hand, the midi format can either be synthesized directly with a midi port or by rendering the midi to audio files. This is done by fluidsynth using an sf2 soundfont that stores audio information for midi notes.
Bigger picture
Watch here if you’re interested in a deeper understanding of the available tools in the project and arduino / midi in general.
The execution of the code produces output files. These are then processed for visualization of the notes and audio file output. Furthermore, there are some auxiliary functions. The following diagram depicts how these programs can be used and how the elements roughly interact with each other.
Further short descriptions
Live plotter
Perhaps it’s best to start with the data sent by the microcontroller. It’s really easy to program a live plotter in R with the package arduinor. This yields an app shown here:
showing how I touched 4 of the 5 thermometers briefly after starting the measurements.
Temperatures to midi
Translation scheme
Once the temperatures are measured and sent to the PC, python computes all the pairwise differences Δ and translates them to midi notes in a given scale. For the C major scale this yields
where the function maps to the midi notes of the white piano keys. This is explained in more detail here.
Chaos machine data
To get an idea how the translation scheme works, have a look at this plot:
It shows the live data that is generated during the main loop of the chaos machine. The differences
Δ of the temperatures T received from the microcontroller are
transformed to midi notes according to the above scheme.
The data for the plot is produced during the execution of the chaos machine and saved to csv files after
it has finished. Only look into these csv files if you’re interested in
more detail what kind of data structures are used to control the machine. If you
want to try out the machine, have a look at the chaos machine
package2.
Midi to audio (live)
Once the midi notes are sent, they can be synthesized to audio with fluidsynth and a soundfont file. The soundfont specifies which sample is played for which midi note. An example how to do that with python can be seen here. This yields in a sequence of rising notes of the sound font. This soundfont of choir voices was used. You can enjoy the result in the following audio file:
Consider this as a minimal example how to send live midi data with python to fluidsynth and after the session translate the recorded live data to a midi file and render it to the audio file that you were about to listen.
Output files
The main python programm produces
- csv output files (as already mentioned) of
- the temperatures from the serial connection
- the calculated temperature differences
- the derived midi notes, as well as
- a midi file
- which is then transformed to audio files by fluidsynth using the specified sf2 soundfont
- a musescore sheet music file
Midi files
The piano roll visualizations of the midi notes can also be derived from the midi file that’s generated after the chaos machine has finished. In the algorithm a new note is only started (o) if it differs from the previous note, otherwise the previous is not stopped (+).
Midi files to audio files
The midi files can be rendered to audio files also using fluidsynth and a soundfont.
You can listen to the result here:
Musescore
Musescore is a music notation software. The midi file can be automatically translated to sheet music in the musescore mscz file format by running
mscore3 live_record.mid -o output_file.msczThe mscz file can then be animated to this beauty3. The sound in this video was generated with their proper soundfont. Channel 10 is traditionally used for percussion. :)
If you’re interested how you can produce and manipulate graphs like this with graphviz code try out yourself with the interactive ObservableHQ notebook HERE.↩︎
The package also provides a demo mode option. Thus, you can try it out without having the microcontroller setup but instead using recorded temperature data also provided by the package.↩︎
Also checkout the animation of the midi data on the musescore webpage by clicking play (make sure to also choose the funky piano roll animation by clicking on the piano key symbol!)↩︎
