Hacker News Reader

Comment by solarkraft | Hacker News Reader

I don't fully understand.

> As each participant recited hundreds of sentences, the electrodes recorded the firing patterns of neurons in the motor cortex. The researchers associated those patterns with the subtle movements of the patient’s lips, tongue, larynx and jaw that occur during natural speech. The team then translated those movements into spoken sentences.

So this means the audio example is speech synthesized from data gemerated while the person was actually reading out loud, right?

Why does the text under the headline claim "no muscle movement needed", which would imply audible speech synthesized from mere thoughts?

View on HN

Former neuroscientist working on similar stuff here.

While the flashy part is the “speech synthesis”, the science breakthrough is actually better frames as an machine learning problem.

Imagine you record someone moving their hand across a canvas. The hand movement becomes the input, the drawing the output. The ML problem they solved is to reconstruct the output (or at least something that resembles it) from the input. In the study’s case, that’s efferent motor signals.

There is a long history of mapping these kind of signals in the sensorimotor homunculus to the respective muscles they control downstream (and some really cool stuff like prosthetic limbs can already be controlled with it), but speaking is a notoriously hard motor task and requires a lot of muscles to work in unison in very precise ways. When you implant these multi-electrode arrays, you get a few hundred more or less random single neurons, astrocytes, and local field potentials from the nether in between. Nonsense, noisy data. Being able to map this back to the result they produce in the body is technically as complex as astonishing!

There's a picture of the electrode array on the Nature site, and it looks about 3cm x 3cm, with about 16 x 16 sensors. So 256 inputs to the analysis system.

I know nothing about these kinds of electrodes, but how sensitive do they have to be? Sub-microvolt? And how fast? Sub-millisecond?

Finally, on the results themselves. If I read it right, the sensors had been previously implanted into epilepsy patients, and were "re-purposed" for this study.

So I assume they patients were able to speak? If so, it's trickier to demonstrate advantages for those who have lost the ability to speak.

Participants spoke sentences while their neural activity was recorded. The acoustic speech was associated with motor movement which was in turn associated with learned neural patterns. As such the final result is a multi-stage model which can translate from patient specific (at a minimum in the sense of electrode configuration) neural activity into audio.

All neural activity (from what I saw in the paper) is neural activity while the individual is reading sentences out loud.

Yes, they are synthesizing speech from thought while the person mimicked speaking moving their mouth but making no sound.

> Although synthesis performance for mimed speech was inferior to the performance for audible speech—which is probably due to absence of phonation signals during miming—this demonstrates that it is possible to decode important spectral features of speech that were never audibly uttered (P < 1 × 10−11 compared to chance, n = 58; Wilcoxon signedrank test) and that the decoder did not rely on auditory feedback

Okay. I now understand what's actually significant about this - and conclude that "no muscle movement needed" is a lie.

Reply on news.ycombinator.com