At CES 2010 in January Microsoft announced that its controller-less accessory for the Xbox 360, dubbed Project Natal, will ship by the end of 2010. Unveiled in June 2009, Project Natal is the Redmond company’s attempt to out-Wii the Wii. Instead of a hand-held controller, wireless or otherwise, Project Natal uses a 3D sensing unit on top of your TV to read your gestures, recognize your face or other objects, and even respond to your voice. Project Natal is among the latest examples of devices that are controlled by so-called “natural user interfaces”.
The goal of a natural user interface (NUI) is to eliminate the awkward and often complex artificial controls between a user and their device. Each generation of UI tries to bring the user experience closer to this ideal - from desktop mouse to multi-touch screens.Although it first announced Project Natal at the E3 Expo video game trade show in June 2009, Microsoft has been working to develop other NUI technologies as well. In addition to the high-profile Project Natal, other examples include Microsoft Surface, Windows Touch, and Ford SYNC, the voice-controlled, in-car entertainment and communications system that Microsoft developed jointly with Ford. But it is Project Natal that brings us the closest to the sci-fi interfaces seen in movies like Minority Report and Star Trek.
The key aspect of Project Natal is its ability to recognize gestures, but the system also includes the ability to respond to voice commands and facial or object recognition. These last two technologies are well established at this point. Despite some consistency issues and a literal learning curve, voice recognition is commonly found in cell phones and speech-to-text typing applications. Facial recognition, often viewed with trepidation for its big-brother security implications, is now widely available in consumer point-and-shoot cameras.
At CES 2010 and at E3 in 2009, Microsoft demonstrated several games on the Project Natal platform including Ricochet, a handball-type game where your avatar hits balls against a wall, and Burnout, a driving game where you move your hands as if you’re holding an invisible steering wheel while your foot controls the virtual accelerator pedal.
Microsoft hopes that Project Natal’s advanced technology brings together gesture, voice, and facial recognition to create a UI that is more approachable for people who might be less comfortable with technology. The technology is essentially hidden, adapting to the person using it instead of the other way around. One of Microsoft’s goals is for Project Natal to broaden the Xbox 360's audience beyond its hardcore gamer base.
The Project Natal hardware itself is a flat horizontal sensor about 9in. (23cm) wide that mounts on top of your television set. Inside lurks a microphone plus a depth sensor that includes am infrared projector combined with a CMOS sensor camera. This array allows Project Natal sensor to see in 3D under nearly any ambient light conditions. The unit has an adjustable sensor range, plus the ability to automatically calibrate itself based on the user’s movement, game play, and the physical environment (for example, the presence of furniture and so on).
The Project Natal 3D sensor works somewhat like a laser rangefinder to provide detailed three-dimensional information about the gaming environment. Unlike a simple camera, the Project Natal sensor generates a “point cloud” based on the surface of objects and people in its range. The Project Natal software uses this 3D data to interpret certain parts of the point cloud as a person, using advanced human tracking algorithms to further identify which parts are the hands, feet, head, and so on.
All of this computer vision processing requires some processing horsepower, and the Project Natal sensor was originally designed to have its own on-board CPU. However, earlier this year Microsoft dropped the chip from the controller presumably to reduce costs. This is in line with recent reports that 3DV Systems (the original developers of the Project Natal technology) first demonstrated the system to Nintendo, who declined to pursue it because of the high price.
The removal of the on-board CPU means that the Xbox 360 will have to use 10 to 15% of its CPU processing power to handle computing chores for the Project Natal sensor. What impact this will have on Xbox 360 game play remains to be seen, but it seems likely that not all Xbox games will be able to support the CPU demands of Project Natal. Indeed, Microsoft has said that Project Natal will focus primarily on “brand new experiences”, meaning games that are designed specifically for it using gesture and voice for control.
The release of Project Natal at the end of 2010 will mark the ten-year anniversary of the Xbox console, which arrived on store shelves for the 2000 holiday season. Over the last decade, UI technology has continued to advance in nearly all areas of consumer electronics. Touch-screen technology has exploded in dozens of different types of products, voice recognition continues (though struggling somewhat on the sidelines), and early brain-to-device interfaces have begun to show some progress. Project Natal, with its gesture, voice, and facial recognition, is bringing UI technology closer to that ideal combination of natural inputs to create a truly natural user interface.
Though Project Natal is scheduled to be released during the fourth quarter of 2010, Microsoft has not announced pricing nor the final list of game titles that will be available.