Virtual Reality

Augmented reality robot fleshes out virtual characters

The "virtual humanoid" prototype will allow you to physically interact with a computer-generated character seen through a head-mounted display
The "virtual humanoid" prototype will allow you to physically interact with a computer-generated character seen through a head-mounted display
View 10 Images
The "virtual humanoid" prototype will allow you to physically interact with a computer-generated character seen through a head-mounted display
1/10
The "virtual humanoid" prototype will allow you to physically interact with a computer-generated character seen through a head-mounted display
A concept sketch showing the various components of the virtual humanoid
2/10
A concept sketch showing the various components of the virtual humanoid
The computer-generated character image is super-imposed over the robot using green screen technology similar to that used in TV and movies
3/10
The computer-generated character image is super-imposed over the robot using green screen technology similar to that used in TV and movies
A prototype called U-Tsu-Shi-O-Mi developed in 2006 was too costly to commercialize
4/10
A prototype called U-Tsu-Shi-O-Mi developed in 2006 was too costly to commercialize
A smaller version of the prototype will sell for between US$4,800 and $5,300
5/10
A smaller version of the prototype will sell for between US$4,800 and $5,300
Strain gauges in the robot's arms detect movement, allowing users to physically interact with it
6/10
Strain gauges in the robot's arms detect movement, allowing users to physically interact with it
The digital character's movements were based on those of the robot, allowing them to synch up
7/10
The digital character's movements were based on those of the robot, allowing them to synch up
The original prototype had heavy padding, which meant that the digital character was often surrounded by a strange halo effect
8/10
The original prototype had heavy padding, which meant that the digital character was often surrounded by a strange halo effect
A thinner version of the prototype lessened the halo effect
9/10
A thinner version of the prototype lessened the halo effect
The character can be seen from all angles and will carry on simple conversations through speech recognition and speech synthesis
10/10
The character can be seen from all angles and will carry on simple conversations through speech recognition and speech synthesis

Different Dimension Inc., a Japanese start-up company, will begin marketing an Augmented Reality robot doll that can take on the appearance of any virtual character. Essentially, a computer-generated image is projected in real-time onto a robot (covered in green screen material) that is viewable through a head-mounted display (HMD). It's a spin-off of a bizarre research project led by Michihiko Shoji at the Yokohama National University and the Japan Science and Technology Agency, originally developed in 2006.

The so-called "virtual humanoid" is an example of "mixed reality" – a technology which takes augmented reality to the next level by giving virtual objects a physical counterpart in the real world. Video images from the user's perspective – taken from the camera mounted to the HMD – are combined with a computer-generated character that moves in synch with the robot. The user can then touch and physically interact with the virtual character through the robot.

The original prototype, called U-Tsu-Shi-O-Mi, presented several challenges. It was a costly life-sized robot, and its green padding could be seen around the edges of thinner virtual characters. Additionally, the system only played back recorded voices, so it could not actually converse with the user. The new version has been shrunk to 60 percent of its original size (a height of 13.7 inches / 35 cm), and is only the top half of the body. It features a slimmer profile, which matches up better with the virtual characters, and the researchers have paid special attention to the texture of the green screen material, so that it feels more natural to the touch.

A prototype called U-Tsu-Shi-O-Mi developed in 2006 was too costly to commercialize
A prototype called U-Tsu-Shi-O-Mi developed in 2006 was too costly to commercialize

The new prototype can tilt and turn its neck and move its arms at the shoulders, elbows, and wrists. The servo motors which power each joint contain position sensors which read the joint angles – data which is fed to the software program controlling the virtual character – thereby synchronizing their movements. It's similar to the system adopted by the Qumarion, a puppet which can be posed to animate virtual characters.

The actual software used is called MMDAgent, which creates an interactive 3D anime-style character that can carry on conversations using speech recognition and a synthesized voice.

The research team is exploring possible applications for the technology, including education, training, and video games – ignoring the naughtier uses that might immediately spring to mind. Interested parties will be able to pre-order the virtual humanoid in March. It will be priced between ¥450,000 - ¥500,000 (US$4,800 - $5,300). The company, which will soon launch its website, expects to sell 120 units the first year, followed by 180 and then 270 units in the following years.

You can see the original U-Tsu-Shi-O-Mi and the MMDAgent software in action in the following videos.

Source: Wingover (Japanese) via MyNavi (Japanese)

U-Tsu-Shi-O-Mi

【MMDAgent】初音ミクとおしゃべりできるソフトをつくってみた

2 comments
yrag
That all seems like fine research I suppose, and who knows where it may lead, but it seems to me, that if a user doesn't require true physical interaction, working out the software to overlay 'social' interactive characters/images within a head-mounted display (perhaps with force feedback haptic gloves) would be much cheaper, more practical, easier (relatively) and nearly if not totally as effective.
kapil
Now that is some technology. Good Job. I am seriously 1 BIG fan of Augmented Reality. After watching this I have realized in coming days we will see much more and much better things than this. My first experience of Augmented Reality was because of AdStuck, in India. A big thanks to them as well.