Okay, Schlieren photography is still pretty awesome. I’m actually still quite fascinated with our ability to photograph sound waves. I also love that I can do it myself for less than $10. But, this project announcement from MIT is pretty wicked, too. Scientists from Microsoft, MIT, and Adobe have developed a way to use similar science to recover the audio from images taken using high speed cameras and even a prosumer level DSLR.
By placing a camera behind a window of soundproof glass and focusing the camera on an object in a room on the other side of the glass, researchers have successfully reproduced sound from vibrations bouncing off the object. Here’s how Abe Davis’s Research explains it:
“When sound hits an object it causes that object to vibrate, the motion of this vibration creates a subtle visual signal that’s usually invisible to the naked eye. In our work, we show how using only a video of the object and a suitable processing algorithm we can extract these minute vibrations and partially recover the sounds that produced them letting us turn everyday objects into visual microphones.”
To be clear, no audio microphones were used to record any of the data. An algorithm processes data taken from the images, all of which were captured using cameras capable of shooting thousands of frames per second (not by coincidence, that is faster than the frequency of an audio signal). However, they were also able to replicate the results using an ordinary consumer level DSLR, though, admittedly, the results are not nearly as clear as with the high speed images. Take a look:
They discovered a loophole of sorts in the DSLR’s rolling shutter which essentially allows them to recover sound at frequencies 3 times greater than the frame rate of the video. (For those not in the know, a rolling shutter captures images one row of pixels at a time by rapidly scanning the scene horizontally or vertically rather than just recording the entire scene at the same time.)
To put their audio creations to the test, the team employed Shazam (an app which recognizes a song and outputs the title and artist) to see if it could actually recognize a song they reproduced by photographing a pair of earbuds connected to a laptop.
And just how tiny are these vibrations? According to the scientists, some of the motions they recorded were one tenth of a micrometer. In lamen’s terms that is about five thousandths of a pixel if you are looking at it in close-up view.
As for it’s purpose other than just being cool, the potential uses of such a tool seem obvious, but I like to see it the way UC Berkley professor, Alexei Efros, so eloquently puts it:
“I’m sure there will be applications that nobody will expect. I think the hallmark of good science is when you do something just because it’s cool and then somebody turns around and uses it for something you never imagined. It’s really nice to have this type of creative stuff.”
The science side of this story is pretty interesting, but I’d be especially excited to see what kind of artistic applications can be made of it were the project ever to become available for such uses. What do you think creatives, what sort of things are possible for us with this technology?
