Hacker News Reader

Comment by abdullahkhalids | Hacker News Reader

For anyone looking to build on top of this. I have tried a few different STT systems, and they accurately capture what I am saying. Unfortunately, they don't support the reasonable workflow

I want to open an office document, for example, and start talking. And I want the software to continuously type what I am saying at the cursor with minimal latency. The continuous part is crucial. Many software will paste whatever I said after I have stopped recording, but that is not useful.

View on HN

Totally understandable, but I’ve found that software that transcribes everything after I finish recording actually works better for me. I’ve tried both kinds, and systems that continuously type what I’m saying distract me from completing my thought. I end up reading what’s being typed and noticing transcription mistakes instead of focusing on what I’m trying to say.

I often prefer to dictate everything in my head about a particular thing for 5–10 minutes and then go through it afterward. I find that much more useful because it doesn’t break my thought process the way continuous transcription does.

I can understand both modes. I mostly use transcription as input for my AI assistant and there I find it very useful to be able to check my input and just repeat myself in case something wasn’t fully captured. When using Apple’s transcription feature built into iOS and macOS, I also really like being able to edit everything right while the dictation is still active.

Dream would be a combination of AI models that are smart like a human transcriber. One can just tell the model what mistakes it has made (e.g. "No, you wrote XYZ but I meant WXY"), and it is intelligent enough to realize when it is being instructed and when it needs to transcribe exactly what I say.

You can fairly easily modify [Handy](https://handy.computer) to do this if you want

I’m planning on having it as a first class feature of the app too just too many other issues to work on first

I’m really glad to hear this!

A while ago, I auditioned about 10 different STT apps on my Mac, with this realtime/streaming transcription as a goal. I failed to find that feature in an app I was happy with, but settled on Handy as the best option otherwise. So if Handy adds this, it will be perfect!

Can you give some pointers around this? I'd gladly help with a PR for this, but if you have anything docs/ideas around this it would be helpful.

I’m on a train right now but off the top of my head the audio pipeline may have to be modified slightly to emit partial text segments as they come in from the transcription engine. And then calling the appropriate paste method the user has in their settings.

It may be easier than expected in some way since we already emit events for the live overlay, so it could be as small as a function call, but I don’t know the code path well enough from memory and what complexities it has. Probably with the Tauri context and a bit of other mess we have as this bit of code has gone through a lot of pain

You know English doesn't work like that. The word you're saying only becomes clear with the surrounding context. Eg, 'there' vs 'their'.

It may be interesting to have it immediately insert the words, even if they are wrong, and when a sentence is finished, replace what has been written with the final corrected sentence.

Google released this awkwardly named app called edge eloquent recently that does exactly that.

In fact, it cleans up the entire paragraph that you just said, and even if you have meandering thoughts, it cleans those up too.

Actually, this above statement was fully dictated with iOS and it added all the punctuation automatically, so I think that iOS is also doing some of this natively. In fact, I’m on the iOS 27 beta and it seems to be doing an even better job of correcting itself and correcting earlier words and adding punctuation too.

This sounds fantastic, but I'm utterly surprised that Google, of all companies, only releases this for macOS and iOS, but not Android.

It’s an awkward app and the whole interaction just feels weird and kind of slapdash. I think it is meant to be a prototype.

But in this day and age it’s easy enough to at least write the iOS and Android versions. But maybe not dealing with the play store.

I tried this on my Mac soon after launch and it was consuming a significant amount of processor cycles even just sitting idle in the menu bar. (From memory, ~20% of an M1 Max.)

It may have been an early issue but with no obvious way to interact and report the issue and, eh, Google’s general attitude around customer satisfaction, I just gave up and deleted it again.

In fact, a few apps I have tried do exactly this inside the app themselves. There is a live textual field that displays whatever the model thinks, and this display constantly goes back and fixes earlier words.

So I know it's possible. I just want to integrate this with the paste-at-cursor feature that these apps have. I imagine the app would have to create a virtual keyboard and use backspace or arrow keys to go back and change things.

Model should be able to understand where logical sentence ends, to stop buffering, and optionally rewrite some of the test that has already been output.

IIRC that is exactly how Dragon Naturally speaking did it decades ago.

But this is still possible to do if you track the whole run of text. You could replace all of it each time so it LOOKS like it’s streaming but earlier words also change. I’m hoping the streaming models do this eventually.

I believe the built-in iOS dictation already does this.

What would be the benefit of this, besides from looking cool?

In iOS this means you can edit the text as it’s being transcribed. For example, I want to dictate a todo list and after each item I can hit enter to go to the next line.

Do the parts before you hit enter still get updated if later context indicates you said something else?

More accuracy. Like others have said, homonyms (their, they're, there) is easier to determine once you have more context. So then you may need to go back a couple words and update them.

Same with punctuation, you could determine that a comma belonged in a certain place once you have enough words.

Oh certainly, I was asking what the benefit would be to the opposite!

Handy already supports streaming transcription models, and you can see the words in the small Handy pop-up while you are talking.

So in general this definitely works. Handy is just missing the feature to insert these streamed words into the app where the cursor is.

I suspect the hard bit is that it sometimes needs to back up and redo, and that's an interface they haven't got figured out. I'm fairly sure I remember Dragon Naturally Speaking doing it in Word years ago though, so the interfaces should be there.

> The continuous part is crucial. Many software will paste whatever I said after I have stopped recording, but that is not useful.

It really depends on how one uses transcription.

For example, I really value being able to open different windows, and look at graphs, or scroll some data while I'm dictating, because it can help me with providing some support information for what I'm saying.

Some apps can even take into account things you copy or look at as part of the transcription's context to improve the results [0].

[0]: https://superwhisper.com/docs/common-issues/context#types-of...

This is pretty cool. The dream would be that all editors (including brower tabs) have an api interface which allows arbitrary LLMs to modify the editor content. So you can indeed look at different windows as you talk, but the editor keeps getting updated live, so you can go back and see where you already at.

This is what I attempted with https://github.com/electronstudio/low_latency_dictation

However the accuracy of the real time models is poor, so I did a second pass with a higher accuracy model before committing the text.

That was one of the main motivations for: https://rift-transcription.vercel.app

I think I prototyped the most fluid real-time streaming transcription experience. For example, you can interrupt/intersperse the realtime transcription with keyboard input.

If you click "Replay..." in the demo above, it shows an actual transcription use case (Black represents typed text, the colored text was dictated):

- Say "Left"

- Type `ium:` (because no model has been able to transcribe my username; special characters are hard)

- Continue dictating...

Apple Dictation does this, or something similar, in my experience. Some apps (e.g. terminals in my experience) buffer the entire transcript but in most apps it's identical to typing as you speak. Have you tried it?

Agreed. It's something I've found annoying about a few systems.

It looks like the rust bindings have streaming examples so hopefully there is a nice solution here.

You used to be able to do this with dragon naturally speaking (don’t remember if that was it’s exact name) 10 ish years ago

whisper.cpp has realtime capability. Been using it for 2 years at this point

Reply on news.ycombinator.com