With the arrival of consumer virtual reality gear to the market, many tech evangelists – with Facebook-owned VR company Oculus at their helm – insist that VR is the next major computing platform. What hardware shortcomings need to be overcome in order to get there?
It would take a crystal ball to know for certain, but Oculus Chief Scientist Michael Abrash is the next best thing. During the keynote event at the Oculus Connect 3 conference last week, Abrash offered a rare look behind the curtain and identified several major hardware hurdles that must be overcome before VR acquires the capacity for truly groundbreaking applications.
App content is being left up to the software developers, but as an example of a "goal" environment, Abrash described a platform that would be powerful enough for coworking with remote colleagues in either a virtual destination or a simulation of a real-life one. Colleagues are represented by avatars that can emote, speak, gesture and interact in real-time. You can see real-life things like your hands and keyboard superimposed into your virtual surroundings, in a blend of space Abrash calls augmented virtual reality.
Achieving a platform powerful enough to support these activities will require collective advances in display, graphics, optics, audio, haptic and ergonomic technology, Abrash said.
Top priority: Eye tracking for foveated rendering
Improvements in eye tracking capability are at top priority. At present, eye tracking technology is good enough to power an avatar's features, and that's about it. But if eye tracking is made perfect, it will enable foveated rendering, a kind of graphics processing in which only the small area you're looking directly at is rendered in full resolution.
Foveated rendering takes advantage of the human eye's natural limitations. At any given time, your eye is only capable of fully rendering a 3-degree area, so populating unnecessary pixels that can only be seen from the corner of your eye is a waste of processing power. Therefore, foveated rendering allows displays and processors to do less work. This starts a positive chain reaction: smaller, more ergonomic headsets, better displays, more obtainable wireless data transmission, and so forth. It would also open the door for possibilities like being able to navigate in VR with your eyes alone.
Abrash said that failure to achieve flawless eye tracking is the biggest single risk factor to his predictions. And it won't be easy. Pupils move and change shape constantly; they even have a slight gelatinous wobble after sharp movements. Eye shape, eyelids, and bone structure also vary dramatically from person to person. Eye tracking also requires a collection of cameras and illuminators to fit inside the headset.
It isn't known whether current eye tracking problems will be solved with more data and engineering, or if completely new technology will be required. Still, Abrash falls on the side of confidence, and seems to think eye tracking – and foveated rendering – will become core VR assets in the next five years.
The easy stuff: Display, audio
Other areas pose less of an obstacle; Abrash seemed rather confident that display and audio technology will improve as a matter of course.
The display, he says, will become more comfortable, realistic and convincing. He expects resolution to edge out 4K x 4K, up from the 1200 x 1800 resolution available now. It's less easy to extrapolate on how much field of view will be able to increase, which will require some kind of optics breakthrough. In turn, field of view affects the level of pixel density that can be achieved (which also depends on whether or not foveated rendering comes to fruition). Unfortunately, there is no hope for holographic, light field, or varifocal displays anytime soon.
In the realm of audio, researchers are making a study of how sound diffracts around the head and ears, so that virtual sound will sound more like the real thing and will be traceable to a source. Abrash says we're on the way to making virtual sounds (and by extension, virtual rooms) much more convincing.
Headgear and controller build
It's no surprise that headsets will be made continually lighter, more comfortable and with better weight distribution, but of course, shape and looks will be determined by how the core technologies continue to develop.
Abrash cited the need for a wireless headset – it would be uncomfortable, if not impossible to spend long hours in augmented VR with a tether – but also said that there is no current wireless technology capable of bearing the data load that VR requires.
He was much less equivocal on the topic of hand controllers. The newly-released Oculus Touch controllers will be an industry standard for a long time, he believes, labeling Touch-like controllers "the mouse of VR." He certainly doesn't think we'll be using our actual hands in VR within the next five years for anything beyond lightweight, gesture-based interface control.
Convincing virtual humans are still far off
While Abrash has ambitious expectations for many areas of VR, he makes no claim that the rendering of humans will soar to ultra-realistic levels in the near future.
As humans, we are hardwired to be extremely sensitive to the slightest face and hand movements, and VR won't be up to snuff in the next five years. Abrash does expect improvements in avatar hand representation, facial and emotion reconstruction, as well as body tracking. In a few years, rendering will be on the "other side of the uncanny valley, but not yet convincingly human." For those of us with science-fiction induced nightmares about VR, or who are just resistant to the idea of replacing human contact with screen time, this could be a good thing.
Now & later
Regardless of the above hurdles, both Abrash and Facebook CEO Mark Zuckerberg asserted that socializing and coworking are coming soon to the virtual reality pipeline. We are already being entertained and educated by VR; at last week's Oculus conference, we learned about the steps being taken to make it a social platform.
Time will tell whether or not VR becomes an everyday computing tool, but by breaking down and itemizing the hardware insufficiencies that must be overcome in order to get there, it actually seems more achievable than not. After all, some early virtual reality machines were so heavy they needed to be suspended from the ceiling, and required an omnidirectional treadmill for maneuvering. The gap is certainly closing, and few areas of technology are as ripe for growth – and are as well-supported with funds and interest – as virtual reality.
You can watch Abrash's full speech for yourself below.