At CES in January, Microsoft Research teased its IllumiRoom concept, which involves projecting an image around a TV screen to enhance video games with additional visuals. Unfortunately, the company didn't offer much info beyond a short video that briefly showed it in action. But the team behind the project recently showed up at the CHI 2013 conference in Paris with some more in-depth details about how Illumiroom will not only expand the game screen, but completely alter the appearance of your living room.
The team's initial prototype consists of a widescreen InFocus IN126ST projector and a Kinect for Windows sensor mounted above and behind the user, though they hope to create a future version that sits on a coffee table between the player and the television. The device is able to calibrate itself to any room, with the Kinect detecting the colors and layout in front of it, while the projector uses that data to properly align the image displayed around the TV.
We've seen how this can extend the game environment beyond the boundaries of a TV screen, but that's just the simplest use for it. The system can also set the visuals to appear differently from those on the TV, so only certain elements (edges of buildings, bullets, explosions, power-ups, etc.) are shown for a more a distinctive style. It's even able to project only onto the wall behind the television, leaving out any surrounding furniture or objects and giving the appearance that the game extends beyond the wall.
IllumiRoom can also alter the appearance of the room and objects contained within to match the aesthetics of the game by mapping visuals to their geometry. By overlaying objects in the room with corresponding bright colors and dark outlines, for example, it can give everything a cartoonish look, or it could cancel the colors out almost entirely to give the room a black-and-white appearance.
The system can even project the exact same features of the room and then distort them to match the gameplay – making the room ripple or shake with each gunshot, for instance. It also allows virtual objects to interact with the room so that balls can bounce out of the screen onto the floor or falling snow gathers on the ground. Developers can essentially mix and match various effects to create more immersive games.
This expanded view allows for increased visuals with movies as well, either turning your whole wall into a projector screen or adding effects outside of the TV. The catch is it only works with footage shot using a custom dual camera rig that captures narrow and wide fields of view simultaneously.
It's important to note that this is all just a proof-of-concept model (i.e. not necessarily a future consumer product), and carries a number of limitations with it, like the fact that the projection effect appears muted when the lights are on. So while these new details arrive tantalizingly close to Microsoft's reveal of the next Xbox on May 21, it will probably be some time before the technology is available in a consumer device.
The video below demonstrates more examples of what's possible with IllumiRoom using some amazing unaltered footage of its visual effects.
Source: Microsoft Research, Brett R. Jones
