It's perhaps not much of a stretch to imagine a computer program that could visually identify which pieces of music were being performed in silent piano-playing videos. A new AI system goes further, however, by digitally reproducing the sound of that piano-playing in a realistic fashion.
Known as Audeo, the technology was designed by a team at the University of Washington. It incorporates artificially intelligent software that was "trained" on approximately 172,000 frames of overhead video of pianist Paul Barton playing music by classical composers such as Mozart and Bach.
When analyzing a silent video, the resulting system starts out by observing which keys are being pressed in what order, determining the individual notes and their arrangement. That said, it also perceives how hard each key is hit and how long it's held down – this lets it figure out the intensity of each note, and the length of time that it persists beneath the sound of subsequently played notes. It also takes into the account the distinctive acoustic characteristics of pianos.
That data is then converted into a format that can be understood by an existing digital synthesizer. When that synthesizer plays back the music file, it reportedly sounds very much like the original piano music, as opposed to simply sounding like an 8-bit ringtone.
In a test of Audeo, it was tasked with reproducing piano music based on muted videos of Barton playing pieces of music other than those on which the system had been trained. When music-recognition apps such as SoundHound analyzed those reproductions, they were able to recognize the piece of music with an accuracy of around 86 percent. By contrast, when the apps analyzed the original piano audio in the same videos, their recognition accuracy climbed to 93 percent. That gap should decrease as the technology is developed further.
"We hope that our study enables novel ways to interact with music," says Asst. Prof. Eli Shlizerman, senior author of the study. "For example, one future application is that Audeo can be extended to a virtual piano with a camera recording just a person’s hands. Also, by placing a camera on top of a real piano, Audeo could potentially assist in new ways of teaching students how to play."
You can hear some of Audeo's handiwork via the link below.
Source: University of Washington