Musician Jason Barnes had part of his arm amputated in 2012, but with the help of a team at Georgia Tech led by prosthetics researcher Gil Weinberg he has found new ways to keep honing his music skills. In 2014, the team developed a novel drumming prosthetic that allowed Barnes to keep pounding away, and they've now come up with an even more advanced prosthetic that is capable of the finer control required for the keys of a piano.
Barnes had his right arm amputated just below the elbow following an electrocution accident at work five years ago. While his hand and most of his forearm were lost, he retained some of the muscles in his residual limb responsible for controlling his fingers, which does allow him some degree of finger movement using an everyday prosthesis.
This device uses electromyogram sensors, which are commonly found in both traditional and new, advanced kinds of prostheses. These types of sensors rely on electrodes to pick up electrical signals from the muscles, but according to Weinberg, who is the director of Center for Music Technology at Georgia Tech, they have their shortcomings.
"EMG sensors aren't very accurate," he says. "They can detect a muscle movement, but the signal is too noisy to infer which finger the person wants to move. We tried to improve the pattern detection from EMG for Jason but couldn't get finger-by-finger control."
To allow for better dexterity, Weinberg teamed up with other Georgia Tech researchers and sought out an ultrasound probe to attach to the arm muscles instead. This is the same kind of ultrasound used to observe babies in the womb, and the muscle patterns they observed in this way enabled them to distinguish between movements intended for each individual finger. By feeding these unique patterns into a machine learning algorithm, the system was able to predict which finger Barnes was trying to move, and even how much force he planned to use.
Combining the ultrasound sensors and algorithms with a robotic arm, Weinberg was able to fit Barnes with a prosthetic that allowed control over each individual finger, allowing him to play the piano for the first time since his accident. He even performed the Star Wars theme song as an homage to Luke Skywalker's bionic hand in The Empire Strikes Back.
"If this type of arm can work on music, something as subtle and expressive as playing the piano, this technology can also be used for many other types of fine motor activities such as bathing, grooming and feeding," said Weinberg. "I also envision able-bodied persons being able to remotely control robotic arms and hands by simply moving their fingers."
You can see Barnes and his bionic hand at work in the video below.
Source: Georgia Tech
