Capturing the complexities of the human hand is a difficult task – just ask any artist or animator. Now, engineers at Cornell University and the University of Wisconsin-Madison have developed a new wearable system that uses thermal sensors to accurately predict hand positions, with potential applications in VR, robotics and translating sign language.
The device is dubbed FingerTrak, and it’s basically a bracelet adorned with four small thermal cameras, each about the size of a pea. From their respective positions, the cameras snap images of the contours of the wearer’s wrist. That’s enough for a specially-designed algorithm to accurately reconstruct the entire hand, including the positions of each finger.
“This was a major discovery by our team – that by looking at your wrist contours, the technology could reconstruct in 3D, with keen accuracy, where your fingers are,” says Cheng Zhang, an author of the study. “It’s the first system to reconstruct your full hand posture based on the contours of the wrist.”
The FingerTrak system uses machine learning to predict the position of 20 finger joints, based on the wrist contours. These hand poses can then be recreated on a virtual model, or even a robot hand. In tests, the device was able to accurately reproduce actions like opening a book, writing with a pen, drinking, and using a phone.
There’s no shortage of devices designed to try to track your hand movements, using various technologies. Some watch your fingers using depth-sensing cameras or infrared sensors, some use motion-sensing gloves, and others use electromagnetic sensors on your fingertips. But almost all of them are a bit too bulky for practical use.
The FingerTrak is designed to be more lightweight, and while it’s still not the most comfortable-looking piece of tech, it does look to be on the right track (excuse the pun).
With a bit more refining, the team says that the FingerTrak system could have a range of uses. It could allow virtual reality players to track their own hand movements in-game, it could make for remote-controlled robots that directly mimic a human operator’s motions, it could help translate sign language into text or speech, or it could help monitor health issues that affect motor skills.
“How we move our hands and fingers often tells about our health condition,” says Yin Li, an author of the study. "A device like this might be used to better understand how the elderly use their hands in daily life, helping to detect early signs of diseases like Parkinson’s and Alzheimer’s.”
The research was published in Proceedings of the Association for Computing Machinery on Interactive, Mobile, Wearable and Ubiquitous Technologies.
The FingerTrak can be seen in action in the video below.
Source: Cornell University