What’s better than as robot bartender that can pour you a beer? How about a robot waiter that can see you need a refill and comes over to pour you another one. Hema S. Koppula, a Cornell graduate student in computer science, and Ashutosh Saxena, an assistant professor of computer science are working at Cornell’s Personal Robotics Lab on just such a robot. Using a PR-2 robot, they've programmed it to not only carry out everyday tasks, but to anticipate human behavior and adjust its actions.

Robots are the neat freaks of the technology world. They like things to be tidy, orderly and predictable, meaning they work best in places like laboratories and factories where everything can be controlled and where it’s easy to predict what’s going to happen next. When a robot moves out of its comfort zone into our imperfect world, it can run into difficulties. Even something as seemingly simple as noticing that someone’s glass is empty and topping it up requires a lot of observation and planning on the robot’s part.

Unfortunately, people may unintentionally hinder the robot by moving their glass as the robot goes to top it up. This could get very messy, so the robot needs to anticipate possible human actions and adjust accordingly. If it sees someone reaching for the cup, the robot has to know when to stop trying to pour.

Robot's view and anticipation of a person approaching a fridge

The Cornell anticipatory robot avoids embarrassing spills and other accidents by using its Microsoft Kinect scanner to build up a 3D map of the objects present and then calculating how they might be used based on the action currently being performed by the person.

The robot manages this by means of a database of 120 3D videos of people performing everyday household tasks, from which it reduces the person's movements to symbolic skeletons. It then classifies these skeletons into subactivities, such as reaching, pouring and carrying, for example, while associating different objects with different actions.

The robot is also able to put various subactivities together in different combinations to form models of larger activities that it can use to anticipate the movements of people in different situations. The models it builds are general enough to take into account the fact that different people will perform the same activity slightly differently.

When faced with objects and someone performing an activity, the robot will generate a number of different possible continuations, before calculating which is the most likely. These predictions are continually updated and refined as the action continues.

However, it's predictive capabilities are still limited and depend on how far in the future it has to participate. In tests carried out by the researchers, the anticipatory robot is able to correctly predict actions 82 percent of the time when looking one second ahead, 71 percent for three seconds, and 57 percent for 10 seconds. So there's still some time before robot waiters show up at your local restaurant.

“Even though humans are predictable, they are only predictable part of the time,” Saxena said. “The future would be to figure out how the robot plans its action. Right now we are almost hard-coding the responses, but there should be a way for the robot to learn how to respond.”

The anticipatory robot project’s findings will be presented at the International Conference of Machine Learning, June 18 to 21 in Atlanta, Georgia and the Robotics: Science and Systems conference June 24 to 28 in Berlin.

The video below shows the anticipatory robot in action.

View gallery - 4 images