Brain-to-Speech Tech Good Enough for Everyday Use Debuts in a Man with ALS
scientificamerican.com
scientificamerican.com
The Scientific American article was written about our lab's first major paper, "An Accurate and Rapidly Calibrating Speech Neuroprosthesis" published 14 August 2024 in the New England Journal of Medicine [0].
This video nicely summarizes the project [1].
Some additional coverage can be found in the New York Times[2] and Bloomberg[3].
Our lab's website can be found here (written in elm) [4]. The software platform that this BCI uses can be found here [5].
[0] https://www.nejm.org/doi/full/10.1056/NEJMoa2314132
[1] https://www.youtube.com/watch?v=thPhBDVSxz0
[2] https://www.nytimes.com/2024/08/14/health/als-ai-brain-impla...
[3] https://www.bloomberg.com/news/articles/2024-08-14/brain-tec...
[4] https://neuroprosthetics.science/
[5] https://iopscience.iop.org/article/10.1088/1741-2552/ad3b3a
It looks like there are three components: 1.) the device itself 2.) a microphone 3.) eye tracking (??)
Is that right? If so, to what degree does a person’s ability to make at least utterances into the microphone matter to the overall system’s performance? Are you heading in the direction of purely using the device itself to determine what patients are intending to say?
A brain computer interface is a device that records brain signals, and allows people with paralysis to control objects in their environment using their thoughts. Here, we demonstrate how the BCI can help restore communication to a man with ALS using only neural signals recorded from the speech-related motor areas of the brain (ventral precentral gyrus). As he tries to speak, our decoding system translates the neural activity into words on a screen. Hence, to answer your question, we are only using brain signals to determine what he's trying to say.
The microphone is not involved in interpreting his speech at all (and in fact, speech-to-text algorithms applied to the audio signals are nonsensical). One can hear him speak in the supplementary videos in the NEJM paper, and in the video produced by UC Davis.
The eye-tracker is used to help control the GUI, much like an able-bodied person would use a mouse.
What are some current limitations of this technology?
Where do you think the trajectory will lead in 5 years in terms of capability?
Are there long term health and safety risks that have been discovered through the various implementations of this technology since 2004?
The progress described in the NEJM paper is happening as part of an ongoing clinical trial called BrainGate2. The clinical trial has been ongoing for over 20 years. The interim safety analysis was recently published about the first 17 years of the clinical trial [0].
The thought behind the question: how similar is everyone's process to construct the word "happy"?
Big congrats to the team at UC Davis.
I'm wondering a bit about this claim, though, and hoping someone more familiar with the field can shed some light on it:
> The device predicts the wrong word less than 3 percent of the time, an error rate on par with nondisabled speakers reading a paragraph aloud.
Does this mean that in a paragraph of 100 words, nondisabled readers are expected to get 3 words wrong? That would be a lot higher than I would expect, but is what it sounds like the quoted sentence means.
Or is it more like if a hundred nondisabled people read a standard paragraph out loud, three of them would trip over (at least) one word in it? This seems closer, but still like a fairly high error rate for something they're calling "perfect."
It is not connected to the "part" where ones inner voice lives, because nobody has a clue where that is situated.
The system maps a sequence of phonemes decoded from neural signals into words on a screen. The words are selected from a 125,000 word dictionary, and he can add custom words to the list.
So, to answer your question, if he were to add "OMG" and "STFU" to the dictionary, then he could say them (if he wanted to).