Software brings eye contact to Skype, with a little help from Kinect

Software brings eye contact to...
Software being developed at ETH Zurich isolates the foreground image and tilts the face to bring eye contact to video calling
Software being developed at ETH Zurich isolates the foreground image and tilts the face to bring eye contact to video calling
View 3 Images
Steps in processing the image
Steps in processing the image
Software being developed at ETH Zurich isolates the foreground image and tilts the face to bring eye contact to video calling
Software being developed at ETH Zurich isolates the foreground image and tilts the face to bring eye contact to video calling
Before and after images of processed faces
Before and after images of processed faces
View gallery - 3 images

Skype has been around for ten years now. Once a science fiction dream, the video calling service has 300 million users making two billion minutes of video calls a day. The only problem is, most of them can't look each other in the eye. Claudia Kuster, a doctoral student at the Computer Graphics Laboratory ETH Zurich, and her team are developing a way to bring eye contact to Skype and similar video services with software that alters the caller's on-screen image to give the illusion that they’re looking straight at the camera.

Making a video call from your computer may be incredibly convenient, but it still lacks that Gerry Anderson quality because it always seems as if the other person it staring at your chest or over your shoulder instead of making eye contact. That’s because the webcam is usually sitting over the screen, meaning it's transmitting an image of you looking at the screen, not the camera.

It is possible to restore eye contact, but currently this involves complex and expensive systems using mirrors or multiple cameras and special software. Kuster's aim is to develop a solution that uses standard, or at least, simpler equipment and operates in real time.

Before and after images of processed faces
Before and after images of processed faces

The new system, which is still under development, currently uses a Kinect to act as a depth sensor. Other than that the computer is fairly standard. “The software can be user adjusted in just a few simple steps and is very robust,” says Kuster.

The clever bit isn’t how it takes an image, but in what it does with it. On current gaze-correction systems, the entire image is tilted. This requires a lot of computing power and is prone to generating large gaps in the image. If you’ve ever used one of those insert-a-background camera applications without a green screen, the patchy results will be familiar

Instead of shifting the entire image, Kuster’s system only deals with the foreground. The Kinect builds up a depth map of the image and isolates the person in it with a border area around it that is used to blend the person into the background. The software then plots a coordinate mask onto the person’s features and calibrates it. The system can then tilt the face to the correct angle to make it look as if eye contact has been made.

Steps in processing the image
Steps in processing the image

The next step is to extract the face from the image using a state-of-the-art face tracker. This computes 66 feature points over the face and uses these to alter the tilted image so that it is consistent and synchronizes with the person’s movements. The image is then pasted back onto the person’s image and blended in, so that things like hairlines and shadows match.

As it does so, the software adjusts for lighting, matches colors, handles two faces at once and deals with a variety of face types and hairstyles. It can even compensate for things like cups moving into the picture, though with less success. However, it can’t handle spectacles at this stage, which is a pity.

Kuster and her team hope to develop software that will work with standard cameras not only in computers, but also tablets and smartphones. The eventual goal is to turn it into a Skype plug-in.

Source: ETH

View gallery - 3 images
Truly amazing software engineering, my hat goes off to the dedicated folks behind this.... however.... A solution I always imagined would be for your face etc to be continually scanned by 2 or more monitor mounted lasers to form a continiually updating live map of your head and shoulders etc.. then it can be re-rendered in any orientation you desire.... including looking right at you.. and it won't need to be creatively distorted at all... it will be as is. This will even work in pitch darkness.. the colours will need to be scanned too
Art Toegemann
I wondered if this would go the way of the shift lens, from original architectural photography. Then I realized it would probably go digital.
Digital still distorts architecture.
Ken Aspeslagh
This seems backwards. When using a webcam, people are usually looking down, at their screen which is below the camera, not up... eh?
Why not place the camera behind a one-way transparent screen?
Devin du Plessis
This is nonsense you can't change the angle from which you see a nose (for example) from 2D image minipulation. You can skew it a bit and make it seem different but if the camera sees your nose from the bottom, the output will still see it from the bottom even if the angle seems to change slightly. The diagram really shows what I'm talking about.
a simpler fix would be to move the user's thumbnail video feed in Skype's display as close as possible to the webcam position . This way your focus is nearer in proximity to the "eyes" of the person you're chatting with. I generally avoid using Skype fullscreen.
Aditya Maurya
This is fake! even though it looks like you are maintaining eye contact you are not!
Lets wait until flat glass camera is invented and can be embedded into touch screens. This way you'll have a real eye-to-eye contact on webcams. :-)
Aliasgar Babat
Eye contact with Skype? Wow!! I have been using Skype for years now in addition to RHUB, WebEx etc. but I have never heard about such thing. Will try out the same.