Fun with R: Visualizing WAV Files via R and Tableau

Posted by

Like many people, I have different parts of my life that don’t generally intersect.  I always enjoy it when they do, however.  So far, I’ve blogged about the data analysis & visualization part of my life.  Another big part of my life is music.  Being both a musician and a data vizzer, I’ve often thought about ways to visualize music.

“Music viz”
The issue with doing data viz on music is there are just too many dimensions (the dreaded curse of dimensionality).  One common way to visualize a sound is to map the sound based on how loud it is at different frequencies:

Frequency plot

This is cool, but it doesn’t account for change over time, which is essential when thinking about music.  You could try to overlay the marks:

Time spagetti

But this gets confusing quickly.  The above shows just nine moments in time.  If you look at even 10 seconds of a song, you get something like this:

Super spagetti.png

While this is an interesting graphic, it’s not very insightful.

A moment of insight
Earlier this week, I had a spark of inspiration.  What if you used color to encode the decibel level for each frequency?  This would create a sort of one-dimensional choropleth or heatmap:

freq to heat

Then, you could make a one-dimensional heatmap for each moment and build up an interesting graphic!

How I actually did this
Okay, I’ve cheated a little bit so far, showing some of my end results to help illustrate my thought process.  At this point I did some research into different types of audio files.  I started looking at the MP3 format, but this turned out to be highly complicated and proprietary.  WAV files turned out to be the most basic way of digitally encoding music.  It then took only a moment to find a wonderful R package (tuneR) and a very helpful blog by a Biological Oceanography PhD student (thanks Hansen Johnson!).

I was happy for an excuse to play in R, as I’ve started trying to learn R for data science purposes.  I’m still primarily a Tableau user, so I decided to process the WAV files in R and output to CSV, in order to visualize in Tableau.  I know and appreciate ggplot2, but this seemed more like what my probable workflow will be if and when I use R for my day job.

This not being a post on R programming specifics, I’ll just say that you can see my code here.

Meet the spectrogram
It turned out that what I’d come up with already existed, and is called a spectrogram.  Mostly it’s used for science, but while researching this blog post I found out that you can do this easily in Audacity as well as other free downloadable software.  There’s one cool website worth calling out specifically: academo does a live, streaming, online version of what I’m about to do.  All of these are pretty cool already, but it still felt like a worthy and fun exercise to do it myself.

The CSVs I started creating were huge, so I decided to look at 10 second samples (from 1:00 to 1:10 for each song).  I picked a few random songs from my “favorites” list at the Free Music Archive:

Here’s a first pass at what came out:Arrington de Dionyso Mani Malaikat - continuous.png

Pretty cool!  But I realized I was losing information due to there being such dense data (1,766,400 marks in this view alone), so I converted both axes to be discrete.  This made the headers impossible to read, but ended up giving cleaner images:

Arrington de Dionyso Mani MalaikatBeat Culture MidoriBebe Fang WorkBlue Ducks Four, Floss, Five, SixCSC Funk Band Bad Banana Bread

Normally I would try to be more consistent in my color choices, but this seemed to be well within the ‘pop viz’ world, perhaps giving some more flexibility in that regard.  Instead I tried to pick colors that brought out interesting textures for each song.

In addition to the pure aesthetic pleasure of creating and viewing these, I did find it interesting to compare the different rhythmic patterns and cycles in the different songs, as well as what frequencies were used.

Dashboard 1

Plus, now I’m that much closer to making algorithmically-produced electronic noise art!  Merzbow better watch out…

That’ll do it for me today.  Until next time, have fun out there!

Jeff

One comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.