Mixing Computer Science and Audio Art using Arduino and Max/MSP
In this post, I present a performance art piece for Max/MSP and N64 controller, that samples audio and video from The Legend of Zelda: Majora’s Mask OST.
Computers have changed almost every aspect of our lives in the last 50 years, and audio is a space that has scene huge revelations in its craft. From digital synthesizers to MIDI sequencers to digital audio workstations (DAWs) such as Logic Pro and Ableton Live, audio engineering has been in the background of some of our favorite sounds in music.
One of the lesser known tools in the audio engineers’ toolbox is Max/MSP. MSP stands for Music Signal Processor — this low-level code puts an original signal through a series of different filters such as reverb, distortion, etc., programmed by the user. Max is a higher level logical graphical programming language that allows engineers to code different routines into a performance. Max/MSP is not meant to be used as a DAW such as Logic or Ableton (though it can be) — rather it shines most when used for performance pieces. Max/MSP allows a multitude of inputs, from simple graphical buttons, to custom instruments and controllers.
Observing how computer science and art combine to produce something spectacular is simply amazing for me. I love watching how a transfer learning algorithm such as neural style can transform a regular image into something by Van Gogh. I love seeing tech-infused museum exhibitions, such as Arduino powered mirrors. I love seeing genetic algorithms such as the reinforcement learning algorithm that plays the original Super Mario Bros. on NES. There are so many possibilities that have yet to be explored in this space.
Considering myself a fairly decent programmer, I decided to build a Max patch (they’re called ‘patches’ in reference to patch cables that would link early synthesizer modules together) that integrate with one of my favorite pieces of art. I am a big fan of video games, specifically of the Nintendo 64 era, as it produced many high quality games with legendary soundtracks. I decided to use The Legend of Zelda: Majora’s Mask as the base of the piece, and develop mini patches to express various moods for different environments found within the game.
For inputs, I thought it would be best to keep it retro. I have an old N64 controller in which I have a pre-built interface to my computer. The controller hooks into the computer through an Arduino. The Arduino itself is running code that reads the input of the controller, and packetizes it before sending it through USB to the computer. The computer is running a Processing script that reads the packet and interprets the inputs of the N64 controller.
Getting these inputs to be registered into Max/MSP is a bit trickier. At first, I attempted to see if I could use Max objects such as serial. Unfortunately, the integrity of the resulting signal was not too good (sending bits over wires can be very tricky, especially when working at a low level). A lot of data was either not sent, or appeared in different spots than expected. This was quickly becoming a headache, as it was impossible to discern which controller button was pressed or if any button was pressed at all. Thankfully, I found a Processing script that mapped the controller buttons to keyboard presses. With this, I used the key object in Max and then selected the appropriate object based on the key’s Unicode value. From there, buttons on the N64 controller could be mapped to control different parameters in the Max patch, such as playback speed, reverb, changing which audio file is in a buffer, etc.
Now that I had the inputs configured, I went on to design what the outputs would look like.
The Majora’s Mask soundtrack is wonderfully orchestrated. Music is a big part of the Zelda series, and songs are composed as to add an extra layer to the experience. For example, if we are in a beach/water environment, the background music plays a soothing beach melody complete with steel drums and high reverb. If there is an approaching enemy, the music will fade to a darker, suspense-filled melody to get your blood pumping in anticipation of a battle.
Majora’s Mask achieves such a successful synchronicity in its arrangements, that it would be almost impossible to improve upon. So instead of improving it, I decided to bastardize it by designing entirely different Max sub-patches for each environment we found ourselves in. For example, in the beach/water environment, I decided to crank up the reverb and add mild distortion. Also added was the ability to use the N64 controller to modify the playback speed and volume.
A big part of Majora’s Mask and it’s direct prequel, Ocarina of Time, is the ability to play a magical instrument known as the Ocarina of Time. In the game, the ocarina is played through button presses that are transferred into distinct musical pitches. The ocarina also serves as an integral game mechanic that allows the player to warp to far away lands, manipulate time of day, or trigger other characters to break out into dance. In the games, it is fun to even play the ocarina as a real instrument and create melodies on the fly. I wanted to capture something similar to this in my Max patch, and thus made a few of the buttons correspond to the in-game pitches.
Once I had created several Max sub-patches for different environments in the game, I worked on creating the performance aspect of the piece. I decided on pre-recording in-game video from the game, and later match the music to their associated environments. The result can be sonically pleasing at times, and downright quirky at others.
Check out the show below.
