Experimental studies on music visualization - BFA senior project

Bach Kantate

Sculpting Music is a project that I started during my last quarter as an undergrad and it was the primary piece shown at my senior BFA show. This is the project I have continued as a graduate student.

This project bridges the two worlds of traditional music and computer graphics by mapping one medium to the other. For example, music contains many different structures like melodies, harmonies, rhythms, patterns, chord progressions and much more.


In a music theory class, students dissect pieces, analyze their chord progressions, and decipher the brilliance of some of the greatest composers like Beethoven or Mozart. These structures and patterns were the musical basis for the Sculpting Music project, and I wanted to visually communicate these musical intricacies.

 
spectrogram
In the first phase of the project, numercial data is extracted from an audio waveform. This is done through the use of a spectrogram, which is essentially a visual fingerprint of the musical waveform. I extracted amplitude peak values at specified frequency ranges. These numbers are then imported into Alias' Maya. Maya has a very comprehensive scripting language called MEL. More details about my implementation of MEL are available on the MEL scripting page.
 
Bach Kantate
 
The first piece is based on a Kantate by Bach entitled Wachet auf, ruft uns die Stimme. The illustration above was shown as a framed picture in addition to the movie. The staff notation indicates where the captured image was taken in relation to the music. In this piece, the sphere represents the string section and the cube represents the trumpet part. I had to make some timing adjustments in the MEL script to accommodate when the trumpet part starts and when it stops. The MEL script dictated where an object would be placed in the 3D environment.
 
Rain Storm Omen
 
The second piece is based on a piece that I composed entitled Rain Storm Omen. This slowly developing piece contains many different moments of tension and release. The tension or ease is exhibited by the dissonance in the chord.

To illustrate this point better, imagine playing a C major on the piano. The chord sounds very stable, relaxed, and doesn't feel like it is going to lead to anything else. Chords like these are often the first and last chords one would hear in a piece of music. A dissonant chord would be exemplified if one just played as many keys as they could on a piano. It would sound very jarring, as some notes would audibly "clash" with one another.

The tornado in this piece twists from the left to the right based on whether there is tension in the chord or not. The size increased based on the growing intensity of the music.
 
A Dance with Her
 
The final piece was also a composition of mine entitled A Dance with Her. The piece incorporates some of my trumpet playing with a computer string accompaniment. In this work, I tried a slightly different variation on the Bach Kantate. The strings are still represented by the sphere and the trumpet is paired with the cube. The choice for these shapes was driven by the nature of the sound in the instruments. More info can be found here on the mapping of sound. In this work, the size, bending, and stretching of the shapes was dictated by the numerical data taken from the audio.