Abe Davis: New video technology that reveals an object's hidden properties

http://www.ted.com/talks/abe_davis_new_video_technology_that_reveals_an_object_s_hidden_properties

As a PhD student at MIT, Abe Davis talked about his recent research on analyzing tiny vibrations of objects. In the beginning of the speech, he showed two videos that look completely still. We may think they are two images. Normally people have to observe motions through touch directly. By using high resolution cameras, software can detect these subtle vibrations from a distance. This technology inspired Abe Davis' imagination. He had a crazy idea that he wanted to recover sound from a silent video by analyzing the vibrations of every object in video. He then dedicated his efforts with his collaborators to this challenging project. Finally, they figured out a solution. The mechanism in background is when sound hits an object, it causes tiny vibrations of the object's surface. They developed a digital image processing algorithm to extract these subtle motions of every pixel in every frame of a video, they can then partially recover the sound that produced these subtle motions. They also pushed the limit by changing the hardware they used. In his case is camera, this technique can also be applied to regular camera, produced a sound with tolerable distortion.

Toward the end of the talk, he showed a recent work they did, where they used similar vibrations in video to learn about the structural and material properties of objects. Then he demonstrated how their new algorithm works with a silent video of a bush moving in the wind. After converting the file with their new software into a photo-alike file, he can click and drag the bush with his cursor. His mouse created a realistic "virtual wind" toward the bush. Their software predicted how the bush would move when it was blown. Abe Davis believes this kind of interactivity to virtual objects can be applied to many fields, such as video games, 3D model, etc... In the future, we may be able to use it to detect whether an old bridge can hold up if we drove car across it. Finally, he believes what he found out until now is just a beginning. What more exciting is, we can build new lens to see the world.