Synchronizing data between a computer and a mobile phone has generally required connecting the two devices via a USB cable. For simple tasks many people even resort to manually typing text from one device onto another. Apple's iCloud is designed to take the hassle out of this task by automatically syncing data between your various devices over Wi-Fi, but MIT graduate student Tsung-Hsiang Chang and Google employee Yang Li have developed a system called Deep Shot that makes it possible to transfer simple computing tasks between devices simply by taking a photo of the computer screen with a smartphone's camera.
Designed to work with Web applications, Deep Shot exploits the fact many Web apps use a standard format called the uniform resource identifier (URI) to describe their current state. URI's are those long links that contain extra information, such as the start and end points and geographical coordinates in Google Maps, for example.
While these links can be copied and pasted and emailed, Deep Shot simplifies things by sending the URI between two devices over Wi-Fi via software installed on both the phone and all the computers with which the phone will interact.
The camera comes into play when uploading data to the phone by identifying the application open on the screen using existing computer vision algorithms. It is also used to identify the specific computer the camera is trained on - work or home, for example - when downloading data from the phone to a computer. The system will also resize the application window to match the framing of the photo.
Because URIs use a standardized set of codes, the system can also transfer data between different applications - from one map application running on a computer to another installed on a mobile phone, for example.
While it is easy to extract information from some Web applications, such as Google Maps, others can be more difficult but the developers say the system should work with any application that reveals its state through URIs with minimal additional coding. In theory, it could also work with off-the-shelf software with some minimal modifications to their code by the software developers. Currently the system works with several common Web applications, including Google Maps and Yelp.
Because Deep Shot was developed when Chang was doing an internship at Google, the search giant owns the rights to it. Google hasn't made the system publicly available yet but Chang expects it to and says he'll be among the first to install Deep Shot when it does.
Source: MIT
