Author Archives: Yitian Liao

Data Project-sound and color


My inspiration of this project became from Neil Harbssion, who could not see colors. His world is black and white, but now he is an artist. Harbssion describes him as a cyborg. Dangling over his forehead is an antenna that curves up and over from the back is his skull. The device, which he calls “eyeborg”, helps him connect him and the color by detect the light frequencies of color hues and translate it into sound frequencies.

Harbission’s artwork burs the boundaries between sight and sound. He can listen to color, and also see sounds.

Here is the how Harbssion sees Biber’s Baby and Beethoven’s Fur Elise.

These made me wonder if normal people without an eyeborg could also participate into this fun with the help of software technology.

We have many phone games that are able to transform the tones, beats, and melody into colors, such as Cosmic DJ, Synesthetic, InSong, which means there is a technology to establish my assumption.

I found Harbssion’s match sheet between sounds and colors from his cyborg company and hope if there is someway I can match the color and sound frequency myself.

Harrbsion's mapping

In the beginning when I asked around my friend, nobody seemed have this kind of experience. They said the assumption sounds doable, but they do not how to do it. Finally I reached my friend’s friend, who is an artist with CS skills, she said:

“His (Neil Harbssion’s) mapping of tone to color is actually really complex, and he uses frequency to detect the nuances of colors, which is a very laborious and intense process. If you want to do it from a scientific approach of parsing frequency to color, it is not an easy process and you won’t be able to finish in time.”

However, she said, I can also fake it by assigning the color with very basic tone.

I tried several music software and finally came up with one called Mixed in Key 6.

This is a DJ software that simplifies a DJ technique called harmonic mixing. It analyzes MP3 files and determines the musical key of every file.

The basic key of each song from Mixed in Key is showing as a specific number with letter, such as 12B as E major.

mix in key 6


My datasets includes, Billboard Hot 100 from 2014 as representation of Western pop music, Melon top 100 from 2014 as representation of Korean pop music, Li Yundi’s Tokyo Concert as representation of classical music, and the sound tracks from Jersey Boys as representation of the Broadway music. By using Mixed in Key 6, I can generate the basic key into letter and match back to basic key in musical notes.

dataset sample1

After getting all the basic keys of my music, I use the tableau Public to generate the date into columns and pies.

If we distribute the number of songs with specific notes, we can see that the 4 groups of music have their own “favorable” notes.

BillboardJersey BoyClassical Jersey Boy

(from left to right: billboard, korean, classical, and jersey boy )

For billboard, it will be C minor, D major, D flat Major, D flat minor, and G major.

For Korean music, it will be A minor.

For classical music, it will be C minor and some B-flat minor, D flat, and F sharp major.

For Broadway music, it will be A major, B-flat major, E major, and F minor.

We can have a simple conclusion that note C and A seems apparent more frequent than others.

In the beginning, I used specific color to draw the pie; the color distribution looks very complex and uneven.

Billboard Korean music Classical

Then I started thinking that the match of Harbssion’s mapping has many color in similar tones, and it may be because the color has similar hue tone also reflect to the keys that sound similar for people to hear.

I modified the pies and use the same shade of the color to indicate keys also share similar hues from the map. The distributions look much better now. (from left to right: billboard, korean, classical, and jersey boy )

Billboard Korean musicClassical Jersey Boy

Looking at the second color pie, it seems all four groups have more even distribution in music tones. And because some color is more brighter, it looks more obvious, such as the orange, representing F#. There is no trend showing that people will be more in favor of one color tone. However, by looking at the pie, we can see that classical music use more F than other genres, and broadway music uses more F#. Theses four genres share similar distribution of note A (green) and D (magenta). And classical music may have more common with broadway music.


There are some limitations of this data project:

  1. Mixed in Key 6 is not a software with high accuracy to determine the basic key of a music
  2. It may be to broad or rough to only look at the basic key of a music and may mislead the interpretation of a song.
  3. Whether the dataset is representative enough

To determine whether is a real trend of music that people like to hear, further research of this project is, indeed, needed.

And it will be on my project proposal.

sonicafication of data?

Neil Harbisson


When I see “data visualization”, I always think about numbers and charts. We make the data become more understandable for people to read and easier for them to get the information we want to provide. During the weekend, I found a video from TED, and it made me think if we can have “data audiblization”.

The story is to use extra sensory by combining the idea of technology and human behavior to extend our sense when we see, feel, hear and even say. For Neil Harbisson, hearing is not a problem, it is seeing. He can see everything but colors. Neil is living in a black and white world, but he is an artist and painter. Instead of using eyes to distinguish and learn colors, he uses his ears.

For him, red is F, yellow is G, green is A…

Neil found he could not see color when he was 11. At the beginning, he refused to wear colors but he realized that it is hard to reject colors in everyday life. He went to art school and colors became mysteries and invisible elements he wanted to persist.

In 2003, he got an electronic eye. This thing is attached to his head, which is like an antenna loops over his head and attached to the end is a little camera. The little camera is what looks at and recognizes colors. A chip installed in the back of Neil’s head detects the frequency of colors and he hears colors through bone conduction.

When Neil goes to an art gallery, he hears the paintings like he goes to a concert. When he goes to a supermarket, he feels like he is in a nightclub.

His proceed of beauty is different from others: he hears the face. Someone looks beautiful but sounds terrible. Faces have sound portraits to him. Prince Charles “sounds” similar to Nicole Kidman when we compare their eyes. Two people who you probably never relate now have some sort of connections.

It is not just colors become sounds; sounds can be translated into colors in Neil’s mind. He can paint people’s voices. When music gets translated into colors, it will be easier to compare different artists. It is more visualizable to distinguish similar colors than to distinguish similar rhythms.

Mozart’s piece used many G, which is yellow. Justin Bieber’s songs have many E and G, which are pink and yellow. Artists share many similarities when they compose.

Justin Bieber's "Baby,"

Neil improved his devices to catch more colors than human eyes do. Now to him, a good day or bad day is based on different sounds.

The development of technologies and our daily life behaviors expand our database regarding humanities from traditional statistical numbers to even social network hashtags. Data interpretation also shifts from simple chars to fancy motional drop lines. Before this video, I never thought about to tranlsate human face into sounds. Now I’m thinking that professor Manvoich probably can add “sounds” as a component to his selfiecity project. Meanwhile, when sounds translate back to colors, I look at Justin Bieber’s song very differently now: they are pink.

The Chinese Twitter:

The development of techonolgy has helped to upload traditional archives to digital platforms that people can use to share information easily and also prevent the risk of physical archive damage. The wide use of social networks in our daily lives also expands the definition of archive and causes our traces or “self-identity“ online to become a new kind of archive.

If you want to study a person, besides going through her/his resume, you can also check her/his Internet accounts. In the academic world, information from social media has already been used as a database. Lev Manovich uses the ten thousands random pictures to construct a database for his project called SELFIECITY ( Selfie pictures are selected from five cities and were analyzed by face analysis software. In this case, Instagram has participated as an archive.

Facebook, Twitter, and Instagram now are the three main social media platforms people use. In 2008, China banned Facebook, but Chinese created Renren as alternative. Twitter was banned the following year, and the Chinese launched Weibo. Luckily, China has not done anything to Instagram yet. Renren and Weibo are adopted from the originals but have altered to fit Chinese market and Chinese users. I’m not a frequent user of Renren, so I can only talk about Weibo.

The major difference of Weibo compared to Twitter considering their function as an archive is you can trace more information about this user on Weibo.

This is what a post will look like:

User A is whom you subscribed to, user B is whom user A subscribed to. You can tell where the post is originally from, and you can also tell how your subscriber found this post. The “forward” is likely the “retweet” in Twitter.

However, you can add comments while you retweet. You can also simply comment under the post by clicking “comment”, which will not be shown on your timeline. You can add @username to remind the author or any people about the tweet.

Weibo shows what kind of platform/which device system you are using to go onto weibo. It is beside the “time” part, after the “via”. It is determined by the weibo phone app.


If more than one people you subscribed retweeted this post, you can also see their “retweet” at the bottom.

When you click “forward” on user A’s retweet, you can see all the retweets made from user A. You can write down your own comment but only 85 characters remain due to pervious comments were being made. You have to share the length. There will be a double slash to separate the two opinions. Weibo considers this as part of your post. So you can delete or edit the previous opinions if you want. One thing about weibo is its privacy. You don’t have to make every tweet public. You can choose to tweet it as visible only to your own friends group. It also applies when you retweet. You can even retweet it via private message. It is like Facebook.


You can go to the original post and check all the forwards (retweet) and comments made directly to the original post. And if you decide to forward directly from the original post, you have 140 characters to comment! Yay!




The “forward” (retweet) and comments allow for more ‘reposted’, engaged, threaded conversation. Twitter users can also share the tweet with her/his followers by retweeting. However, the tweet to be shared can’t be modified.

On weibo, user can do more than that. She or he can add comments or opinions to the retweeted tweet. The opinion is limited to 140 Chinese or 280 western characters.

This is how you “Weibo”.

Screen Shot 2014-09-18 at 12.20.36 AM

You can see different potions to insert rich media like images, videos, music, emoticons and polls when you click “more”.

You can see different portions to insert rich media like images, videos, music, emoticons and polls when you click “more”.

Weibo in Chinese means “micro-blogging”, with its hierarchical  comments to the original tweet, it seems quite easy to follow and participate in conversation. Weibo has adopted many features from Facebook to fit into the Twitter platform. These features such as the emoticon which people can use to tweet to total online strangers is a representation of the “online-identity”, which can be very different from the one in real life. Comments made when retweeting or those which are read after the previous tweet is also a process of self-consciousness and self-censorship. If you use Weibo as database to study an Internet phenomenon, the numbers of forwards and comments can show you directly how hot and trendy the post is.

One of the criticisms about Weibo is that it is not simple enough. The conversation may be fun but information can be overwhelming. You cannot directly quote from one comment to reply, which makes it hard to follow (but now Weibo learnt from Twitter again and had the feature of conversation that allows you see the information line). If the original poster deletes her/his post, you can no longer see it, which unlike on Twitter when you RT the tweet, it is on your timeline and no one can take it off.


However, simply being a database or archive, Weibo may have done enough.

Here are some English weibo accounts from celebrities if you want to explore weibo further:

David Beckham,  Stephon MarburyAlicia Keys.


Happy weiboing!