Ordinarily, if you want to create a lifelike three-dimensional digital model of someone's face, a 3D scanner and/or multiple cameras are required. Now, however, scientists from Pittsburgh's Carnegie Mellon University have created a system that lets a single smartphone do the job.
The technology was developed by Assoc. Prof. Simon Lucey, working with master's students Shubham Agrawal and Anuj Pahuja.
It begins by using a smartphone to shoot 15 to 20 seconds of video of the subject's head, with the videographer moving around the person – from one side, to the front, to the other side – as they do so. Lucey's team utilized an iPhone X on the slow-motion setting, in order to gather as much visual data as possible due to the higher frame rate.
An existing technique known as visual simultaneous localization and mapping (SLAM) is then used to determine the basic geometry of the face. It does so by triangulating points on the face's surface to calculate both its shape, and the position of the smartphone camera relative to it.
Next, deep-learning-based algorithms are used to identify the subject's facial profile, along with the relative locations of landmarks such as their eyes, ears and nose. This still leaves some informational gaps, however, which traditional "mesh fitting" computer vision techniques are then used to fill.
The entire process takes 30 to 40 minutes to complete, and can be done entirely on a smartphone. It results in 3D models that are claimed to be highly detailed (down to a sub-millimeter level) and realistic – more so than previously-developed similar setups.
It is hoped that the system might ultimately find use in applications such as the creation of gaming avatars, biometric identification, or in the field of medicine.
The team's work was presented early last month, at the IEEE Winter Conference on Applications of Computer Vision in Colorado.
Source: Carnegie Mellon University
