Posted by Like many people, I have different parts of my life that don’t generally intersect. I always enjoy it when they do, however. So far, I’ve blogged about the data analysis & visualization part of my life. Another big part of my life is music. Being both a musician and a data vizzer, I’ve often thought about ways to visualize music.
“Music viz”
The issue with doing data viz on music is there are just too many dimensions (the dreaded curse of dimensionality). One common way to visualize a sound is to map the sound based on how loud it is at different frequencies:
This is cool, but it doesn’t account for change over time, which is essential when thinking about music. You could try to overlay the marks:
But this gets confusing quickly. The above shows just nine moments in time. If you look at even 10 seconds of a song, you get something like this:
While this is an interesting graphic, it’s not very insightful.
A moment of insight
Earlier this week, I had a spark of inspiration. What if you used color to encode the decibel level for each frequency? This would create a sort of one-dimensional choropleth or heatmap:
Then, you could make a one-dimensional heatmap for each moment and build up an interesting graphic!
How I actually did this
Okay, I’ve cheated a little bit so far, showing some of my end results to help illustrate my thought process. At this point I did some research into different types of audio files. I started looking at the MP3 format, but this turned out to be highly complicated and proprietary. WAV files turned out to be the most basic way of digitally encoding music. It then took only a moment to find a wonderful R package (tuneR) and a very helpful blog by a Biological Oceanography PhD student (thanks Hansen Johnson!).
I was happy for an excuse to play in R, as I’ve started trying to learn R for data science purposes. I’m still primarily a Tableau user, so I decided to process the WAV files in R and output to CSV, in order to visualize in Tableau. I know and appreciate ggplot2, but this seemed more like what my probable workflow will be if and when I use R for my day job.
This not being a post on R programming specifics, I’ll just say that you can see my code here.
Meet the spectrogram
It turned out that what I’d come up with already existed, and is called a spectrogram. Mostly it’s used for science, but while researching this blog post I found out that you can do this easily in Audacity as well as other free downloadable software. There’s one cool website worth calling out specifically: academo does a live, streaming, online version of what I’m about to do. All of these are pretty cool already, but it still felt like a worthy and fun exercise to do it myself.
The CSVs I started creating were huge, so I decided to look at 10 second samples (from 1:00 to 1:10 for each song). I picked a few random songs from my “favorites” list at the Free Music Archive:
- Mani Malaikat by Arrington de Dionyso
- Midori by Beat Culture
- Work by Bebe Fang
- Four, Floss, Five, Six by Blue Ducks
- Bad Banana Bread by CSC Funk Band
Here’s a first pass at what came out:
Pretty cool! But I realized I was losing information due to there being such dense data (1,766,400 marks in this view alone), so I converted both axes to be discrete. This made the headers impossible to read, but ended up giving cleaner images:
Normally I would try to be more consistent in my color choices, but this seemed to be well within the ‘pop viz’ world, perhaps giving some more flexibility in that regard. Instead I tried to pick colors that brought out interesting textures for each song.
In addition to the pure aesthetic pleasure of creating and viewing these, I did find it interesting to compare the different rhythmic patterns and cycles in the different songs, as well as what frequencies were used.
Plus, now I’m that much closer to making algorithmically-produced electronic noise art! Merzbow better watch out…
That’ll do it for me today. Until next time, have fun out there!
Jeff
One comment