Compositing a character into a real-world environment captured on video is easy these days, but having said character realistically interact with objects in that environment generally isn't so easy – or, it wasn't. Researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have now developed a technique that enables just that, by constructing an interactive model of a real-world object from as little as five seconds of video.
The system, which its creators call "Interactive Dynamic Video" (IDV), works by analyzing the tiny vibrations of an object in a video. These vibrations are almost imperceptible to the naked eye, but by studying the frequencies at which an object vibrates, the researchers are able to create a model of the various ways the object can move, and predict how it will react in a range of situations far beyond than what's shown in the video. The result is a simulation of the object that users can interact with dynamically using a mouse, pushing and pulling the thing in real time.
"This technique lets us capture the physical behavior of objects, which gives us a way to play with them in virtual space," explains Abe Davis, the PhD student publishing the work for his final dissertation. "By making videos interactive, we can predict how objects will respond to unknown forces and explore new ways to engage with videos."
As opposed to labor-intensive, somewhat rigid 3D modeling, the resulting IDV can provide a faster, simpler and cheaper way to model the physics of an animated object. Davis demonstrates how the system can bridge the gap between animated and live action objects and characters, with a video that makes it look like he's shaking the branches of a bush with the power of his mind.
Augmented reality could be made more realistic as well, with Davis citing the example of the creatures in Pokémon Go interacting with the real world more naturally through the user's smartphone camera. Outside of entertainment, IDV could be used in engineering, to test the structural integrity of buildings and bridges in earthquakes and strong winds.
"The ability to put real-world objects into virtual models is valuable for not just the obvious entertainment applications, but also for being able to test the stress in a safe virtual environment, in a way that doesn't harm the real-world counterpart," says Davis.
Davis demonstrates the IDV system in the video below, and the results are really impressive. He explains in more detail on the project website, Interactive Dynamic Video, with other videos including the Pokémon Go example.
Source: MIT
