The trouble with learning a foreign language is that to become fluent – or even just to be passably coherent in a reasonable timeframe – you need to be immersed in it. You need to live in a country where that language predominates. But cost or opportunity often make that infeasible. San Francisco startup Learn Immersive wants to create the next best thing. Its two-man team has built a virtual reality platform that transports you to real-world environments and helps you understand them in their native language.
For co-founder Tony Diepenbrock, the thrust of the idea comes from frustration. "I studied French for 12 years, but when I tried to speak it in the country, often-times foreigners would respond to me in English," he tells Gizmag. The test-obsessed American schooling system doesn't work for learning languages, he feels. "You need to immerse yourself in situations where you need to figure out what to say."
That's what Learn Immersive is all about. "We're trying to simulate these scenarios, and [to] make anything and everything interactive," Diepenbrock continues. "You can click on any object around you in VR and receive the translation. We can create puzzles for you to figure out."
The hook is something they call Virtual Field Trips. "What's important is to ignite students' passion to learn," Diepenbrock explains. "That passion follows you for life. For me, that passionate was ignited by field trips. Field trips were always the best part of any class for me. They made the homework, tests, and lectures matter. They gave it context." Learn Immersive simulates these field trips.
The company plans to scan – or have others scan – environments such as a Chinatown grocery store, Ghirardelli Square in San Francisco or a clothing store in Puerto Rico. You will then be able to experience these environments through a virtual reality headset such as the Oculus Rift, right from your home. You'll be able to see translations of text and vocabulary of objects, and they're looking at working with language teachers to provide voice-overs and create quizzes.
"Down the road, once we have improved 3D scanners/models," says Diepenbrock, "we'll be able to create Virtual Field Trips at the Eiffel Tower, Golden Gate Bridge, Great Wall of China, and other cultural icons around the world."
These Virtual Field Trips are largely created by software. You feed in 2D or 3D footage and it automatically identifies objects and text, then generates sentences about the scene. These can then, of course, be translated into the language in question. It's not perfect, but Learn Immersive has plans – and tech – to make it better.
"We spent the last eight months working on building a text recognition engine for the real world," Diepenbrock reveals. "Companies can submit any image to us, and we'll extract the text for them."
They're deep learning researchers, concerned with ways to make computers do the hard work of describing images. Deep learning software gets better not only as its algorithms improve but also as you feed it more data. Diepenbrock rifles off names of companies in the field, including Clarifai, MetaMind, ImageNet.
At Stanford, Fei-Fei Li and her team created software that can recognize objects in photos and videos and describe entire scenes almost as well as people. The work was done in conjunction with Google's Research team, which famously created an artificial neural network in 2012 that trained itself to recognize cats from examining 10 million stills of YouTube videos.
"Computers can now generate entire sentences based on just an image," says Diepenbrock. "This is nothing short of incredible."
Diepenbrock and fellow Learn Immersive co-founder Charles Moyes thought this technology could be applied to helping people learn a language. The timing was almost a perfect storm. Microsoft's Kinect and Google's Project Tango initiatives, along with others by smaller companies, are making real-time 3D scanning of environments affordable and mainstream. "Soon, every mobile phone will be able to create these 3D scans," suggests Diepenbrock.
That has implications in virtual reality, which is itself breaking into the mainstream thanks to the Oculus Rift, Google Cardboard, Samsung Gear VR, and Sony's Project Morpheus. As with 3D scanning, VR is working its way into the mobile phone market, and nearly everyone owns a mobile phone.
Diepenbrock sees a future where anyone can scan their environment with their phone, choose the correct suggestions that the Learn Immersive deep learning technology spits out, and share it with anyone, anywhere.
Back in the present, though, they're limiting who can create a Virtual Field Trip. The first few trips will showcase the technology and be released free, possibly around May 2015. "Shortly after, we will accept 3D scans from curators who own scanners like the Kinect – but it won't be open to everyone [until later]," Diepenbrock says.
Creators will eventually – "in maybe a year or two" – be able to sell their Virtual Field Trips via an online marketplace rather like the App Store. And Diepenbrock hopes that Learn Immersive will lower the barrier to entry for learning a second language and bridging gaps between cultures.
"I am now trying to learn Spanish, because my girlfriend is from Puerto Rico, and we traveled to Spain last summer," he explains. "I want to visit Spain again, but this time, I will be able to speak with the people who live there."
Source: Learn Immersive
