Silent speech with ultrasound | Hacker News Reader

Silent speech with ultrasound

alephneuro.com

109 pointschrwn35 comments

This sounds promising -- would love to see this become more practical.

With wearable devices becoming more common - I am anticipating a wave of "sensors" that can be as simple as small band-aid patches that wirelessly send data to your smart device. Those sensors could also open up human-coputer-interface innovations like these.

---

In similar space ...

There was a post about a thought-to-text project from MIT no less --

"AlterEgo"

10 months ago: https://news.ycombinator.com/item?id=45174125 8 years ago: https://news.ycombinator.com/item?id=16780357

When I saw the demo posted last year it left me with an uneasy feeling -- gut feel said it was more marketing than a real working technology demo. Nothing seems to have come out of that lab since then -- strengthemning my suspicions.

The use of ultrasound is interesting! The first step (identifying speech) is called subvocalization recognition, and early research (Chuck Jorgensen, et. al (2023) from NASA) use EMG[1].

[1]: https://scispace.com/pdf/sub-auditory-speech-recognition-bas...

s/2023/2003/

I remember a story decades ago about "subvocal" speech, similar to this: https://spacenews.com/nasa-develops-system-to-computerize-si...

> The two biggest hardware challenges are reducing the size and weight of the ultrasound probe and replacing ultrasound gel with a more practical coupling material, such as hydrogel. We think both are solvable, making it possible for the probe to eventually become a lightweight wearable or adhesive patch.

Not sure I'd want to put an adhesive patch on my neck every morning so I can silently talk to an LLM in the cubicle farm. I hope this is not our future.

Very cool tech though and surprisingly good results for so little training.

I think time might be better spent improving a lip reading model (no adhesive required), assuming we're unable to read brainwaves directly.

Keeping a camera focused on the lips could get awkward. A hands free throat patch may be more ergonomic.

It won’t necessarily be adhesive. It could be attached mechanically - we’re still exploring the form factor.

The idea is that you won’t need cubicles anymore. Your hands will always be free because you won’t need a keyboard or mouse. You’ll be able to control the digital world using only your tongue.

The future is bright: you’ll talk to your Jarvis while driving through the Norwegian fjords.

Really impressive. Seems like this could help people with health problems like ALS or Parkinson's, if you can discern clear intent from their tongue muscles. Or just help people who are intubated in the hospital.

That could be immensely helpful for people who cannot speak due to vocal chord problems.

It could also be the ultimate, always-on remote control for everything around, with a near-zero error rate.

Exactly - a universal input device.

It could help not only people with vocal cord problems, but also essentially anyone who is paralyzed from the neck down.

I am bullish in this tech , but there is something missing to become this to private like headphones .another thing is it still has to do some physical movement which is not going to too much fast like direct brain to text .

I am researching in this field right now, MY ideas are if there is minimum amount of muscle movement will be the best .

Very cool! While this is a different type of inverse problem, it reminds me of a radiolab story [0] about a device to help blind people “see” using electrical impulses on a metal strip under the tongue.

[0] https://radiolab.org/podcast/seeing-tongues

I remember reading about something like this tech in one of Asimov's Robot novels. Thought about trying to build something like this, but don't really have the skills. Awesome they did build this! I guess if they make it easy to train on your speech (just combine it with speech to text), it's a product waiting to happen!

Thank you for the warm words!

Model already generalizes on out-of-distribution people, so you actually don't need to train a model for each person

Sony presented a simpler solution to this at CHI https://dl.acm.org/doi/10.1145/3772318.3791397 - check video in that. Also it seems every Cornell student is working on this. Just search on YouTube Cornell silent speech

Interesting. This seems similar to (though using different methods/techniques) what q.ai (that was sold to Apple for $2b) is developing.

See this hn thread about it: https://news.ycombinator.com/item?id=46816228

I'm very eager for subvocalization tech to take off - this, AlterEgo, etc.

I have never liked talking aloud to Siri or similar, but I could see using "voice" as an interface for so much more if I could speak silently to my device.

Voice is still a very underexplored modality. Right now, it’s used mostly for dictation, but it could also be used for editing.

The funny part is that the tongue itself could act as a cursor or perform gestures, which sets this approach apart from other silent-speech methods.

I wonder if, like with lip reading, they switch from American English to a different language that's not so peculiar they would have and much less error rate

reminds me of the handwriting recognition language Graffiti from palm/handspring days.

It ended up altering my handwriting even after I stopped using it.

I suppose Spanish or German would be easier to recognize than English, due to the more distinct sounds, but Chinese would be much harder, due to the tones.

Tongue movement detection cannot track only six phonemes: - b - p - m - f - v - h

That’s a remarkably small number compared with other methods. English could easily be optimized to achieve a sub-5% WER with a very small model.

I'm having trouble imagining how one would use this. How do you speak without making sounds?

You move your mouth and tongue as if you are gonna make the sounds, just don't exhale the air. They have a demonstration video in the blog post. As somebody else have already pointed out, it's a great tool for interacting with always-on camera devices like smart assistants, makes it a bit less weird.

Oh I see. So no sound but not with mouth closed

One cool fact: with silent speech, you can “speak” not only while exhaling, but also while inhaling

there is a perception that ultrasound is not as dangerous as xrays (radiation etc). As we find more use-cases for it we need to be aware that the damage caused by blasting organs with energy waves is not negligible.

is this true for medical use cases where it’s primarily for imaging? i.e. echocardiogram or pregnancy scans

I had thought of this, so nice to see it done!

Straight out of Life Artificial...

These guys are on a roll!

I find it annoying to try and talk completely silently. I'm always whispering at least a bit. I can't imagine doing this for the entirety of a call.

When I was collecting data, I spoke silently for more than two hours.

After the session, I tried to use Wispr Flow but forgot that I was still speaking silently, so it didn’t work.

When I finally said it out loud, it felt like such a hassle that I essentially sold myself on the idea of silent speech.

Wonderful tech, and video example. I think there may also be a special forces application, but I don't know enough about how well their current solution works.

In the office, a non-contact video solution (lip reading) is likely to be far more popular, but a lot depends on which is more accurate.

Reply on news.ycombinator.com