In another advancement in the field of brain-computer interfaces (BCI), a new implant-based system has enabled a paralyzed person to not only talk, but also 'sing' simple melodies through a computer – with practically no delay.
The tech developed by researchers at University of California, Davis (UC Davis) was trialed with a study participant who suffers from amyotrophic lateral sclerosis (ALS). It essentially captured raw neural signals through four microelectrode arrays surgically implanted into the region of the brain responsible for physically producing speech. In combination with low-latency processing and an AI-driven decoding model, the participant's speech was synthesized in real time through a speaker.
To be clear, this means the system isn't trying to read the participant's thoughts, bur rather translating the brain signals produced when he tries to use his muscles to speak.
The system also sounds like the participant, thanks to a voice cloning algorithm trained on audio samples captured before they developed ALS.
The entire process, from acquiring the raw neural signals to generating speech samples, occurs within 10 milliseconds, enabling near-instantaneous speech.
The BCI also recognized when the participant was attempting to sing, identified one of three intended pitches, and modulated his voice to synthesize vocal melodies.
This bit, demonstrated in a video supplied by the researchers, appears rudimentary, but it feels wrong to use that word to describe such a remarkable development in enabling nuanced communication among paralyzed people who may have felt they'd never express themselves naturally again.
Sergey Stavisky, senior author of the paper on this tech that's set to appear in Nature, explained that this is a major step in that direction. "With instantaneous voice synthesis, neuroprosthesis users will be able to be more included in a conversation," he said. "For example, they can interrupt, and people are less likely to interrupt them accidentally."
If this work sounds familiar, it's because it's similar to tech we saw in April from University of California Berkeley and University of California San Francisco. Both systems gather neural signals using brain implants from the motor cortex, and leverage AI-powered systems trained on data captured from the participant attempting to speak words displayed on a screen.
What's also cool about the UC Davis tech is that it reproduced the participant's attempts to interject with ‘aah’, ‘ooh,’ and ‘hmm.’ It was even able to identify whether he was saying a sentence as a question or a statement, and when he was stressing certain words. The team said it also successfully reproduced made-up words outside of the AI decoder's training data. All of this makes for far more expressive synthesized speech than previous systems.
Technologies like these could transform the lives of paralyzed people, and it's incredible to see these incremental advancements up close.
The UC Davis researchers note that their study only involved a single participant, and their subsequent work will see them attempt to replicate these results with more subjects exhibiting speech loss from other conditions.
"This is the holy grail in speech BCIs," Christian Herff, a computational neuroscientist at Maastricht University in the Netherlands, who was not involved in the study, commented in Nature. "This is now real, spontaneous, continuous speech."
Source: UC Davis Health
