Hacking for science and creating synesthesia

This past weekend I willingly deprived myself of sleep in order to participate in Science Hack Day in San Francisco.

What is a Hack Day?

Well, a hack is a quick solution to a problem. Not necessarily pretty, but probably clever.

A Hack Day is usually a 48 hour event where people with ideas get together and make something cool! Wikipedia lists a bunch of hack days around the world. It seems to be a fairly new phenomenon. Science Hack Day was first held in London last year. Since then people have gotten excited and made things with science! (exclamation points are very much part of the culture) in Mexico City, Cincinnati and last weekend in San Francisco for the second time. Upcoming events include Nairobi and Cape Town.

The idea behind science hack day is that there is so much data out there, and so many things to be found, that people can do science or use science in innovative ways through hacking.

Last year’s winningest hack was the particle wind chime. Matt Bellis, a physicist and post-doc at Stanford, led the team. He writes:

The idea was to allow users to take properties of the particles that we observe in our detector (energy, distance from the interaction region, type of detector it is interacting with, etc.) and map that onto sonic characteristics (volume, timbre (instrument), pitch, duration, etc.). In this way, the user can explore the data themselves and find mappings which either make sense to him or her, or are simply more aesthetically pleasing.

I admit that I was not expecting to be of much use at a hack day. I have no experience in software development or coding and very little ability to use tools and build stuff. But the website said “If you’re a coder, designer, scientist, hacker or just an enthusiastic person with good ideas, Science Hack Day is for you.” I wasn’t sure that I had good ideas, but I could do enthusiasm.

Emails from Ariel Waldmen, the founder of Science Hack Day SF, repeatedly emphasized that I didn’t even need to have an idea before coming. And that eavesdropping on other people was completely acceptable. So I went.

I wandered around the morning of the first day, introduced myself to people and tried to think about how I could be useful. Everyone was very friendly, very enthusiastic and full of ideas.

After inspirational talks I found a couple of young women who won me over with their NCBI ROFL shirts. We were going to create a walk-in, interactive model of a cell, but didn’t think of a satisfactory way to accomplish that over lunch. Instead we turned to the group behind us and decided to help with their idea: create a synesthesia machine to override our everyday senses.

Synesthesia is a mixing of the senses. Synesthetes might report seeing green when they read the number 4, feeling a strong impression of a particular shape when they eat ice cream  or tasting raspberries every time they listen to the third movement of Beethoven’s Moonlight Sonata. Scientists used to believe that this was just a strong memory or that people were making up the sensations. But recent studies show a neurological basis. This study found additional activity in the brain of synesthetes corresponding to their sensations.

The project was based on an idea by this guy:

I asked our fearless leader to tell me about his idea to create a synesthesia machine. He responded via email:

There’s a strong evolutionary pressure to optimize our perception of the world to emphasize information that makes a difference to our survival. This means that we necessarily under-appreciate a large amount of the information in the world, that is, we ignore much of reality. The goal was to short-circuit the pre-processing systems in our visual cortex by sending visual input into an inappropriate sense, the sense of touch. The hope is that the resultant synesthesia would allow us to appreciate a different slice of reality to the quotidian.

Hacking this idea did take us most of the time. I stayed up until 3:30 a.m. and I loved it.

Lil modeling the placement of speakers.

We used a webcam to capture an image and translate that to a 12 pixel square. Each pixel in the square would register the image as either white or black. That would translate to an “on” or “off” code transmitted to an Arduino –“an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software.” The Arduino was wired to small, cheap speakers sewn on the inside of a full head mask. So the visual input was translated to vibration (and sound) on your face!

Initially, we thought we could design a vest lined with the speakers. I was ready to try and modify a thrift-store purchase into a tight fitting garment. That probably would have been beyond my sewing ability. We decided that a mask would be a more visually striking design.

Lil found a pattern online of a bondage mask. Yes, I know … but we needed something that would keep the speakers in contact with your face. I didn’t find a good pattern for spiderman masks, so bondage mask pattern it was.

Other members of the team worked on coding. I have very little idea what goes into software programming, so I don’t know what they did, but they did it well.

We also needed to wire the Arduino to the speakers, something that required soldering and some electronic-building knowledge. I remember building a tiny robot using a kit when I was younger, so I did get to try my hand at soldering. Though I am not an expert, my skills in that area improved under the tutelage of another member of our group (Parker, who was also my old friend from college).

I think was was most remarkable was the way each of us in the group was instrumental in getting to the finished product.

The final product was…amazingly creepy. We call it Syneseizure. It may possibly be the next superhero or supervillain.

Check out our project’s blog for more photos, a link to the code we used, video of the first public test and video of our presentation at the end of the hack day. Our hack won the People’s Choice award!

The other hacks were incredible. From a globe with a laser inside that tracked the movement of the International Space Station, to a drinkable DNA extraction (with Bacardi 151 and strawberries!), there was some seriously cool things going on.

I had a great time. Thanks so much to the organizers who worked to keep us hacking and fed! They did so much. Here is a time lapse video of the 8 hours of set up at the space. And some more photos of the day.

More reading:

An article from Scientific American about synesthesia.

A website about synesthesia for kids, from the University of Washington.

The American Synesthesia Association