After the launch of Firefox Reality, Mozilla's new virtual reality browser, we spoke to Mozilla's Head of Mixed Reality Program Strategy Andre Vrignaud about the future of the browser, VR, mixed reality and 3D digital content. Our conversation took us from the specifics of Firefox Reality, including the prospect of PlayStation VR support, to the future of VR technology, Mozilla's role in the industry, and VR's prospects of entering the mainstream – and why the likes of Boeing and Airbus might just help it to get there.
Note: Minor edits have been made for ease of reading.
How did Firefox Reality came about?
We're always looking ahead and trying to figure out what technologies, what capabilities are people going to be looking for and needing in the future and 3D was one we saw several years ago. So we did a lot of work on the first standards of WebVR, we helped engineer and design those and actually release the first implementations and that allowed people to start creating 3D on the web.
That work gave us a lot of visibility with partners and we began to start looking ahead and seeing what was going to be coming down the road in the terms of stand alone AR and VR headsets. We heard a lot of positive energy from partners, we also saw potentially a really large consumer opportunity. That's really part of our goals serving those consumers being able to get in front of people and being able to act as a trusted user agent for them wherever they are.
Above: The view from inside Firefox Reality
So, we started working with those partners, we started hearing what they wanted to see out of a browser. We also started working on WebXR which is sort of a next version of WebVR but it's basically WebVR which is all around creating 3D on the web. WebXR is taking into account new AR devices like Magic Leap, ODGs, or HoloLens and taking those concepts and bringing them into the standard. That work led to a prototype WebXR browser – you can find it in the iOS store today.
But that allowed people to start playing with, you know, what would it mean if I was creating 3D objects or creating 3D that actually interacted with the real world. To be clear that's not a real released browser. WebXR continues to be a standard that we're working on with the other browsers to get to a final state so we can implement it in quote "real browsers." But the fact that we have working implementations caused even more AR providers to sort of reach out to us and that entire kind of pull from partners and the roadmaps we were given really highlighted to us the need for a new type of browser that was focused on 3D on the web, both VR and AR, and that's where Firefox Reality came from.
So we started building it, released the first version now across Google Daydream, HTC Viveport for the HTC Focus and then the Oculus Go and are hoping to release it on other platforms in the future, especially some of the AR platforms coming down the road.
And when did that work begin?
Ah, several years ago, so it depends how you count it. The browser Firefox Reality itself was about almost 2 years ago, but about 3 and a half or 4 years ago was when we started working on the WebVR standard which sort of led down that whole path.
How would you describe its current feature set as it stands right now – the things that set it apart from a regular web browser?
So I'd say it's a great MVP [minimum viable product]. There are certainly features that we know and want to have. It has WebVR support, it's going to have the early WebXR support. We have the ability to give you the environments, personalize your space. The differentiation that we aimed for based upon our early user feedback fell into two buckets. One was around difficulty in typing. Keyboards in VR tend to be challenging, it's usually like a laser pointer or a little stick and you kind of bonk on the keyboard and such – it's a challenge. So we invested a lot of time in the basics that you have to have, the keyboard that you type and poke on.
But then we realized there was an opportunity to enable voice and if you could just actually talk to the browser it would accelerate things so much more. So one of the differentiators that we did do for the release was voice search so you can just talk to the browser, click a button and do a quick search for like, computer backpacks, and it just takes you immediately to the search term and it works amazingly well.
Above: Voice detection can take the pain out of typing while wearing a VR headset
Some people are already using it as a little bit of a shortcut for navigation, too. You know, if I want to go to Netflix I just search Netflix and it appears as a search result that I can click on, that saves me from the typing. The other area that people mentioned was a challenge was WebVR [being] a new standard so the content that's out there is just beginning to flow. So you know, how do you find the best WebVR content, the best immersive experiences? How do you find the really good, say, 3D videos that you want to go and experience?
To answer that we wanted to go and help people do it with a content, we basically invested in a content feed. It's a homepage for Firefox Reality and it's curated and the intent is for us is to be curating and highlighting to people the best WebVR content on the web on a regular basis or an act of conversations with partners who want to include themselves on there and it's a way for you to quickly find that content.
Above: Firefox Reality features a feed to highlight VR content
We are aware there are features that we just weren't able to get in for 1.0 that we absolutely planned to. Bookmarks is a simple example. It's just a matter of time, [and] as I mentioned we have several partners, you know, pulling pretty strongly wanting to release this. So we were trying to find the sweet spot of getting out into market, hearing from actual users.
We did user tests. We went to a company called Blink UX here in Seattle and we did a bunch of user testing to get real-world feedback. That helped guide us toward this first version. But there's nothing like getting in front of hundreds of thousands of people who all are passionate and giving you input to actually understand and help guide your future roadmap. So we wanted to get out and start getting that feedback and now we're kind of working hard on our future version where you'll be seeing things like bookmarks, the ability to sync content between browsers and more robust theater controls, more robust 360 support – that sort of thing.
Is it just search or are there other voice commands as well?
Right now it's just search, but that was just for the MVP. We're looking at potentially expanding that and having the ability for you to do dictation. So kind of Siri-like. You know – "Hey Siri, here is a string of text" and be able to insert that in, as well as voice commands like go forward, go back, go home. It's kind of interesting though because once we enabled voice search we found that seemed to cover – I'm going to make up a stat here – 70 or 80 percent of the use cases that users actually wanted to accelerate with, with search, with voice I mean.
Above: Firefox Reality's voice search in action
So, we're not quite sure what, you know whether, I personally wonder how useful is it to be able to talk to your browser and say go forward, go back versus just clicking the button because you have the pointer and you're looking and navigating at it. I don't pretend to know the answer, it's something we're going to do user testing on and see if it's actually valuable. I think dictation might be a little more useful. Because I can squint and see places where people might want to, you know, leave a review on Amazon and they'd like to be able to just talk and insert a large string of text. So I think that might be an area we can also explore.
Why build on mobile browser technology?
It's built on GeckoView for Android. All the devices we run on now are basically mobile Android variants, so that made it a lot easier for us to get a head start in actually bringing it out there. But yeah, that's basically the core reason. And we want to bring it to future devices. I can't really talk about which ones, because a combination of partners not really ready to talk about it and us not being quite sure how we're going to prioritize it so I don't want to accidentally excite people, but some of those are built on different OSs [and] some of those are on different platforms so there's additional work we need to do to be able to bring Firefox Reality to them.
I have to ask about Playstation VR…
I worked in the game industry for near 20 years (more on the Xbox side) but as you can imagine they are very cagey and won't want to talk about anything. I can give you the probably-not-terribly-useful answer of we'd love to be on there. I mean certainly. But you know, there's a lot of steps to kind of get there beforehand.
Above: Sony's PlayStation VR headset
And so I think the honest answer is if I was Sony I would probably be waiting to see a lot of demand from their consumer base for the 3D web and then they would probably be looking to investigate what offerings are available to them at which point, you know, they definitely know how to find me.
What is holding back virtual and mixed reality from going truly mass market?
I wouldn't even say holding back, I think it's just where we are [in] time and space. You have to remember WebVR is only a couple of years old so the ability to even just create 3D content is barely a few years old. So there's a lot of work going on in enabling tooling and technologies that actually make it easy to create that content. Unity, for example, has done a great job of being able to allow you to create content in their engine and then export it to the web. We've done work with them to help them to help do that with the web assembly work we've done in the past as well as for WebVR export tools that we have available in their store.
So we're kind of trying to see the tooling market so people can start creating content. There's also the simple fact that the WebXR standard, the one that is kind of encompassing the early WebVR work plus AR going forward, isn't done yet. We're hoping to have that standardized in the coming months. Technically a developer can't even build something. They can build it on the prototype browser like the one we have in the iOS store, but they can't actually ship it anywhere and actually start testing with users, so we kind of have to build that platform to be able to do it.
Above: Cheaper headsets like Oculus Go could help VR go truly mainstream
And then there's just the user base which is just, you know, a chicken and egg problem. Oculus has been investing really hard on releasing pretty high-performance quality devices at a low price point like the Oculus Go. They've mentioned that they have future devices. I'm not sure what on their roadmap is public or not, so I can't really talk for them. You can imagine they're going to continue pushing on that and I think the bet is that at some point those are going to become compelling enough – the price point, the value to the consumer – that it starts taking off.
Where do you think the most compelling non-game content is going to come from? Big entertainment companies or is it going to come from independent developers or even individuals?
I think there's going to be a significant chunk that's going to be on enterprise, honestly. That probably won't be as interesting to my mom who might buy an Oculus Go for Christmas but, you know, one of the key scenarios we hear over and over is from … I'll call it Boeing or Airbus: the huge manufacturers who are looking for the ability to have augmented displays and augmented apps to help accelerate construction on the assembly line or all sorts of other scenarios.
When you talk to them they are very clear that they see the power of the technology and you know, I've seen stats (I'm not going to quote one now because I forget the actual numbers and I don't want to throw you down the wrong path) – think significant double digit improvements in efficiency when you actually have information right in front of you. They want that and they're absolutely charging ahead.
What they don't want to do though is have to build a native app on a different engine for every different platform, release a version of their app into every store, deal with the content policies and review policies of all those different stores and manage all that. They want to effectively create a single web app that they can push to a website and have any device just access it and be able to do whatever the capability is of that application at the click of a URL.
Above: Mozilla Hubs lets you share a mixed reality space to meet with or without a headset
So I think that's an incredibly powerful scenario and I think we're going to see a lot of work there. We believe in that as well. I don't know if you're aware of our Hubs work? Hubs is by Mozilla but it's basically a VR kind of chatroom social environment. You could probably map it mentally to things like VRChat or Rec Room except the difference is it's all based on the web.
You and I could be having this conversation right now in a Hubs room where I would just create a hub and I would send you a link or I would send you to a website called hub.link with a 4 digit code. You would type it in on any device – it could be your desktop, it could be your phone, it could be your headset – and then we'd suddenly be in a VR world being able to talk, communicate, whiteboard, import objects from the web, videos, 3D models and actually have an interaction with them.
The magic there is it's all a website, it's all literally a web page that's hosted somewhere and it's all just a click away. It's a simple hyperlink that you share. So I think that's where we're going to see the majority of the content coming is when people start finding that kind of instant collaboration. In the short term what we're seeing a lot of is these headsets being used as kind of tablet replacements: so media consumption devices.
Above: 360-degree video like this can work much better with a VR headset
I think that probably leads to the reason you're seeing a lot of 360 video, 180 video, and that sort of content is because you can sit down, put on basically a large screen in front of you and just consume it. We're hearing a lot of feedback from Asia; China in particular.
China seems to be about 6 to 9 months ahead of the US in terms of use cases. People love the headsets there because it gives you a large screen when potentially you live in say a small apartment where you don't have the room to have a large screen. It gives you privacy so you can watch the content you want even if you're with friends or family in the apartment, you have the ability to kind of immerse yourself in a larger experience in that small area.
And partners have told us about scenarios they see where users are basically wandering around during the day with their mobile phone and they're caching or queuing up content that they want to view later into kind of a bookmark area and then they get home and then they just simply put on their headset and they go watch through all of that content on that large screen. And that's kind of a scenario that we're looking to enable as well.
What do you find most exciting about the future of VR and mixed reality?
Wow, so many things. I mentioned I've been in the game industry for 20-plus years. Kind of parallel to that has been AR or VR efforts that people have been trying to kick off for 20 years or so. In fact, our VP of emerging technologies, Sean White, has been actually working on this space for 20 plus years himself directly. And we've kind of seen VR sort of come forward and not quite take off. The price points weren't there, it was too expensive, [and] the technologies weren't there.
With the advent of the whole mobile phone generation and the mass production of the gyroscopes and the positioning sensors and the quality high resolution screens, all of that stuff that you need for mobile phones. It kind of kick-started the real VR generation and now AR, and it feels like we're not going to reset this time.
It feels like we may be going through a slight trough of disillusionment where people are like, well it didn't explode like we were hoping, but when you look at the rumored announcements from a variety of hardware providers and you see people moving ahead and you see what you can do today.
Especially in something like an Oculus Go at a US$200 price point. You can actually squint and say okay, I think 3D is going to be a really key part in people's lives in future. And for me personally, I think from the Mozilla perspective, the ability for us to act as a trusted intermediator for the VR and AR web of the future, is huge. Because when you have these devices and these incredibly robust sensor packs they're collecting insane amounts of information and shipping it off somewhere.
You know, a simple example is an AR glass. I'm not even going to mention any names, you don't need to. Just imagine AR glasses that can sense the world around you. You know some companies' vision of the future is that all of those glasses will send back all of that data to some central place and they can literally map this amazingly detailed view of the world. But then you go, well what does that mean if I'm looking around my room and I see, you know, things that should be private?
Above: Microsoft's HoloLens augmented reality headset
You probably saw the Roomba mapping, the robot vacuum mapping of the floor plans, you know, and they were sharing that back and the explosion of what happened there? It's a thousand times worse when you can map the real world and see real world objects and understand how you interact with it and listen and such.
Bringing all that back, I get really excited about the ability for Mozilla to try to act as a trusted arbiter and help the industry think through privacy issues, think through security issues, think through the way that this data can flow and give the user control over it. That's really exciting in this new space.