Speech Recognition and TTS in less than 500kb | Hacker News Reader

Speech Recognition and TTS in less than 500kb

github.com

570 pointspetewarden92 comments

I made a little python wrapper around it to serve an HTTP endpoint that’s OpenAI/elevenlabs compatible https://github.com/clayrosenthal/bootlegger

Quick link to the video where he demos it: https://www.youtube.com/watch?v=kMliOFYBiz4

Thanks for that ... impressive!

Amazing that this works. As an aside, and I appreciate this is just a demo, if the use case is to get a device to join a WiFi network - would a single or double line lcd with 3 buttons not be cheaper than 520KB?

the target rp2350 is a sub-$1 chip. a 16x2 LCD module is over $1. but more importantly, you might have this much ram sitting around unused on whatever you're building anyway.

Is that Microsoft Sam? :)

(Also, I know it's besides the point but this might be the most painful way to connect to Wifi physically possible. "Make normal everyday tasks slow, tedious and painful" is a bit of an odd choice for a product demo.)

Say, speaking of Sam, what were the memory requirements for SAM (Software Automatic Mouth) on C64. I guess they were not more than 64K? Although, the bulk here is probably for the speech recognition, not the TTS. (And this one does sound a little nicer :)

Browser demo of a reversed SAM:

https://discordier.github.io/sam/index.html

There are two Sams here, the Microsoft one and the C64 one. I don't believe there's any connection between the two other than the name.

According to [1], the weight of a modern runnable version is around 39k.

The ratio of how good it sounds compared to how much computing power it uses is ridiculous. The C64 has ballpark 3 orders of magnitude less CPU throughput as an RP2350, and the codebase uses an impressive array of tricks to do actual formant synthesis (barely) and a pretty refined form of Elovitz text to phoneme conversion. One of my favorite tricks is its up and down bouncy pitch, which is not random, but based on the opposite contour as the first formant. It's simplistic but enough to make it not sound like a robotic monotone.

I've been playing around with this some myself and SAM is an inspiration, along with other landmark systems like MITalk (predecessor to DECtalk), SP0256, and other. I believe it's possible to use modern techniques to get pretty good sounding speech in, say, 64k and 10% of the throughput of a RP2350. It's really cool to see projects like OP, especially under permissive license.

[1]: https://simulationcorner.net/index.php?page=sam

This is awesome. I am trying to build a full scale ASR system within 20-25MB. Now that we have Claude code to run experiments, I have started running some experiments. Promising results so far. First realization is that you can capture the nuances of speech in just 3300 embedding vectors(786d). This sequence can be decoded with a small CTC system to get text. Next experiments are on reducing the 768 dimension space into a 64D space. Thats also show some promising results. Hooking up my system so that the agent blogs the results everyday[1]. So my research "claw" setup does the experiments and posts results which I check in the morning and adjust the experiment direction as needed. Its not fully automated yet, but almost there.

[1] https://blog.trulm.com/posts/speech-as-independent-parts/

I think Google's Conformer paper is SOTA at the <30M model size, where I think they put an incredible amount of flops into a 10M param model to reach around 2% lsc clean (the whole model and RNN decoder were trained domain specific to librispeech here).

I think my small Talon models are next, around 3% lsc clean at ~28M (greedy CTC decoding, no external encoder, no LM, not trained in a domain specific way). I reached around 6.5% at 10M.

I've been working on some new baselines I want to release soon as public artifacts. This article is inspiring me to try pushing the param size down a bit. I suspect we can do large vocabulary end to end in the <5M range.

Stt/tts systems always seem to me so promising, but I pretty much never use voice to interface with a computer. Sometimes instead of typing on my phone, I use a voice dictation. I would be keen to use voice to control Claude code, but I've always felt that the way I speak is different from the way I write good prompts.

Fishing for anecdotes here, does anyone have any good tts/stt experiences?

I'm founder of ottex.ai, I use stt pretty much all the time when work with AI and quite often for communications to draft emails and chat messages.

I started ottex half a year ago after I tested gemini 2.5 flash native audio support. I was blown away by the quality of transcripts and decided to built an app to use it myself.

Currently the default model in the app is Gemini 3 flash, but you can connect to 9 providers and God knows how many models to play with.

I would suggest you to try this models for ai prompting:

- Gemini 3 / 3.5 flash - Soniox rtt v5 - Mistral transcribe v2 - assembly 3.5 pro

One of my side projects is a tool that lets you control your entire system with STT. It's built on Whisper and supports hot swapping custom profiles, so you can add easy commands for any software.

I intend to use it to work on low stakes vibe coding projects while I'm doing other stuff. Todays LLMs are a lot better at interpreting rambling dictation with mid-message corrections.

There are a few paid programs out there that do the same, but they made my vibe slop sense tingle and are not aimed at development.

I do a ton of coding (codex) with a tts/stt wrapper. During walks, cycling, in the car. Not every task is suited to this style of interaction, but many are. Long form codex replies are condensed, code blocks are suppressed all in the name of making it work for tts feedback. So it works best on well defined projects with guardrails, where you know the agent can perform well.

During cycling! Do you have a phone mount on your bike that you use while biking or is it all in-ear?

... I cannot think of an activity less suitable for coding (except scuba diving)

I would die In minutes

Think long winding, quiet, dedicated cycling roads in forested areas and natural parks. Not busy roads shared with cars and lorries.

I have my phone in my pocket, no screen interaction is required. I use a headset (Shokz OpenComm2) with wind muff (when cycling). I made an Android app that listens for codex turn-complete or intermediate updates and plays them back to me. My answer is transcribed and pasted back to the relevant codex (tmux) session on the server (which I can select by voice) a tiny layer helps with things like /new, /plan, answer selection, etc.

I'd love to hear more about it!

I'm still wondering if this is something the big AI companies will come out with tomorrow, if it has business potential (it's extremely niche I think) or if I should put it on GitHub and build a community around it. This is what the Android app looks like: https://imgur.com/a/qy8B2gt https://imgur.com/a/C0eTPyn

This is extremely dangerous and you should stop doing it.

https://etsc.eu/tiny-proportion-of-drivers-understand-danger...

Talking on the phone is extremely dangerous? What are your thoughts on talking with a passenger? Is that something that people ought to stop doing? Drive in silence. 100% focus?

Seems like an overreaction.

It's not my thoughts, it's simple facts.

All the studies show it's not an over-reaction, you're 400% more likely to have an accident. It is extremely dangerous and it's not an over-reaction, the more people do this, the more people die. It's simple maths. If you're doing extended programming sessions, you're not paying attention to the road like you should be.

And no, talking to a passenger is not as dangerous as it's a different cognitive load.

It'll take you 20 seconds to google this, please do and stop putting everyone else in jeopardy.

In what way is talking on the phone a different cognitive load that talking to a passenger?

After googling the 8th or 10th result had a link to the study I assume you are referencing.

I just don't agree with you. I believe that there is risk when driving inherently although relatively low and a 400% increase from 0.0001 to 0.0004 does not really concern me.

I have been in one accident in 15 years and I was not at fault and it was unavoidable.

Many people engage in all sorts of distracted driving, texting, eating, driving while tired etc. which I would wager are more detrimental than speaking on the phone.

Having your one bone to pick being speaking on the phone just seems absurd and an odd message to preach when nearly everyone does it.

One key factor you’re missing is attention.

The difference between talking to a passenger versus using a hands-free phone is an interesting one. While they intuitively feel similar, it has been shown that speaking on the phone results in significantly worse performance. This is because a passenger shares awareness of the driving situation and surroundings. Their conversation naturally adapts to traffic conditions, often pausing or shifting topics to focus on the road, which can help with awareness.

Texting and eating are far more dangerous as they divert eyes from the road and introduce motor interference. Fatigue also severely degrades both attention and cognitive function.

Coding while driving does seem unnecessarily risky though. Unlike other non-motor distractions, it demands more cognitive engagement without necessarily moving your hands or eyes. It is far more demanding than a standard phone conversation, which has already been shown to impair driving.

You will make errors that you won’t remember because your brain is occupied with the code rather than monitoring the environment.

Your ability to recognize objects and react slows down. If a child runs into the road, that split-second delay could mean the difference between stopping or killing them. The same applies to sudden obstacles like kangaroos in my case. The delay in reaction could turn a near-miss into a fatal crash.

Most of the risk lies in the complex and unexpected situations. It's really not worth it, just listen to some music or an audiobook and enjoy your ride. Do coding later.

If you have any other psychology or cognitive-related questions to it I can try and answer. I'm not an expert in this specific area but we did some work on it at university.

I can't honestly think of a case where this would be remotely useful. This goes somehow beyond vibe coding to vibe interaction, where the only feedback comes via the AI. I'd love to see a concrete example of this working practice.

I've been building out the Android app that works as the wrapper between codex and me this way. ADB debugging access over WireGuard. I ask to add a local wake phrase detection for the phrase "Hey codex" to start dictation. Codex tells me it thinks Vosk or OpenWakeWord might be a good fit, I voice select Vosk. Codex starts working. Codex deploys (updates) the app on my phone and restarts it and I hear the task complete response. I say "Hey Codex" to test and ask for some tuning/improvements, or move onto the next task. All while keeping my phone in my pocket.

But there are countless projects I work on this way. Eg, I got an email from person X, it says it encountered a bug when doing x and y. Codex reads the email (using nvidia/gliner-PII to hide PII data) and investigates the reason for the bug and proposes a solution. I ask it to implement the fix on my dev server and increase test coverage. I enjoy my walk and after say 7 minutes get a an overview of the fix and can decide to deploy to production.

That's fantastic. I have long, winding trails near me also and one of these days I also want to start prompting a coding agent on my bike with a headset. Do you recommend any particular type of headset?

Edit: never mind, I see you already suggested the Shokz OpenComm2 in another comment. Thanks!

This is so alien to me. Why not plug into the machine matrix when out in the great outdoors enjoying sublime nature? Why not!

It's not a replacement for being outdoors, connecting with nature. It replaces indoor desk bound office work.

For STT, wispr flow has a generous free tier. For TTS, I have Claude read out loud what it just finished as a stop hook, so I know which claude finished up.

Industry leading Interactive Voice Response systems have become very good at filling in ambiguous information from context, and modulating pronunciation to Ape emotional information.

However, being able to interact with these natural language systems in uncontrolled settings is still a fools errand. For STT, there is also regional dialect, slang, and individual differences.

Witnessing blind users hit unrecognizable reading-speeds on old Gordon 8 TTS systems was surprising. I learned people adapt to imperfect systems pretty quickly. =3

I've always dreamed of having the ability to just talk to my computer (in the right circumstances) so I actually worked in the field for many years. The main reason I never use speech recognition today is because I have zero interest of sending recordings of my voice to the servers of some global corporations.

Running speech recognition and TTS locally is quite feasible, as projects like this one show.

If you want a local and open source option (MacOS only at the moment though), I've been happily using Keyscribe for dictation, which is built on Moonshine I believe.

https://rsperko.github.io/keyscribe/

> local and open source option (MacOS only)

ouch - this is ironic in an extreme, given Apple's OS layers and anti-GPL efforts

next, personal computers that come with a secret OS that can read all RAM by remote command?

On some days, half of my promts to Copilot at work are spoken to a local whisper medium model running on an Intel ARC GPU.

I (well CC and I) wrote a tts/stt pipeline for the CLI of CC. It's a lot more, immersive I guess, when I open my dev environment and it gives me a verbal walk through of what's going on.

(this inspired some more demo-y stuff I have where claude can manipulate the mouse and audit things it's built visually in conjunction with that). I'm sure this has already been wrapped up into some MCP framework, but it was fun to build it super early on and it just sort of works for me.

I don't use this in my day job, but it does feel very futuristic when I pull up my home lab.

I use the Hex[1] app on MacOs for near instant transcription with Parakeet V3. This is how I speak to Code agents at least 80% of the time. The idea is - I tend to be lazy if I need to type lots of details, so speaking lets me get into details that I otherwise wouldn’t and this helps give more useful context to the agent. This often tends to be an unstructured brain dump so I sometimes ask the agent to repeat back what it understood, so I can make sure, and this also likely helps it stay on track.

[1] hex https://github.com/kitlangton/Hex

I used to use Handy but as of a few months ago it had stuttering and other issues so I switched to Hex. Even for relatively long minute long dictation, transcription is near instant and accuracy is more than sufficient, especially when talking to AI since it that can “read between the lines “.

Using voice with code agents is a huge unlock and I’m surprised to see some people I recommend it to, still resist it.

As for TTS I found it fun to make a voice plugin for Claude Code that uses PocketTTS, so it can give brief updates whenever it finishes a turn:

https://pchalasani.github.io/claude-code-tools/plugins-detai...

What I want is a 1940s style “taking dictation” where the words I say go through a step where the goal is to create the text Im imagining. So if I say “… very significant, actually just significant…” what Claude Code receives is “…significant…”.

I built this myself with whisper -> “secretary” prompt -> Claude Code, but having the first two steps be interactive is really what I would want.

I use dictation to drive Claude code frequently, and it’s never had a problem with stream of consciousness and retroactive correction. Maybe try just direct voice and see if you notice any difference versus pre-cleaning?

Claude Code's speech recognition works so well for me, I was blown away the first time I tried it. I wish I knew what model they were using (I assume it's not in house since they've never talked about it).

I acknowledge this may just mean I haven't tried enough modern voice recognition systems. But I've used Whisper and I don't think it works nearly as well for real-time speech.

(I still don't tend to use voice mode in Claude Code because I find typing more comfortable.)

This reply helped me think more about what I want. I do my thinking when crafting the prompt, and I want to be able to do several passes. So the first draft I like to dictate and be able to live edit with my voice, but then I want to do a subsequent pass at the keyboard after the full prompt is written.

https://handy.computer is the goto. It can even add an LLM pass (optionally) so if you say "remind me to buy three, actually, four, eggs" it'll type "remind me to buy four eggs".

Do you have any accuracy benchmarks?

I’ve worked in this space. TTS in a small footprint isn’t the hard part —- it’s doing it accurately that’s hard.

Although for the use cases OP is targeting, lower accuracy may be good enough!

> I’ve worked in this space. TTS in a small footprint isn’t the hard part —- it’s doing it accurately that’s hard.

This actually holds for everything in AI.

Very true!

If you look at this chart here it seems the tiny model has a WER of ~12%… not sure about the micro model:

https://github.com/moonshine-ai/moonshine#when-should-you-ch...

That's the error rate for STT, not TTS. TTS is generally easier than STT because you only need to produce one valid pronunciation and don't need to handle variation within and between individuals.

Wow, it seems like this might beat out flite for very-low-memory TTS? I ended up abandoning a project of mine because I couldn't get high enough quality or low enough memory usage out of flite, so I'm very excited to try this out.

Flite for comparison: https://github.com/festvox/flite

For TTS I wonder how this compares to nanotts[1] with the en-GB voice, which is sort of unreasonably good.

[1] https://github.com/gmn/nanotts

I installed the command line version using uv

    uv init
    uv add moonshine-voice
    uv run moonshine-voice mic --language en

super nice to be able to run it to test it like this

good job on a clear readme.md tbh

`uvx moonshine-voice mic --language en` That is even simpler.

So at that tiny 500kb size I imagine it could be compiled to web assembly, and run entirely in the browser right?

Couldn’t find a link, is that hard to do?

500k memory but not sure about disk.

Should be very doable. I ship a small CNN in a browser extension via onnxruntime-web and the model weights were never the bottleneck, the runtime was. The wasm backend adds a few MB of runtime before your first inference, so a 500kb model with a lean hand-rolled wasm build would actually beat most "tiny" browser ML deployments in total download.

One gotcha if anyone wants this in a Chrome extension: MV3 requires 'wasm-unsafe-eval' in the CSP for any wasm at all, which surprised me the first time a build that worked fine as a web page died silently as an extension.

Yeah, I also found that for ultra low footprint models ORT is a big portion of the total payload, because it contains logic for general ONNX graph operations. In my case I found that ORT alone was 3.4MB over the wire, so I swapped it out for a tiny wasm that was 850x smaller and only contained the operations I needed: https://blog.lukesalamone.com/posts/creating-tiny-semantic-s...

did you skip simd just because the model's tiny? naive conv perf is honestly the only reason i haven't done exactly this for the cnn

Yeah, the model is small enough that inference is already basically instant for my usecase (only 6 transformer layers for the blog search).

Very cool. I've done TTS on a 32K Arduino but it was pretty croaky. https://youtu.be/ErGDboTpwM0

This looks like an extreme point for AI-based TTS, as formant/tract modeling synths tend to be more accurate if you want TTS in a tiny amount of compute, but sound distinctly robotic.

TTS (neural diphone synth @ 16 kHz) ~1.8 MiB voice pack

This is in the realm of Microsoft Sam.

Presumably it's not, but the TTS voice in the video sounds to me more like formant synthesis than diphone - it reminds me of my DECtalk.

The project credits does mention espeak (which is formant based) as well as various other TTS projects, although it sounds like they are only using the pronunciation part of espeak, not the voice synthesis.

https://github.com/moonshine-ai/moonshine#acknowledgements

It certainly sounds similar, but seems more nimble with phonetic pronunciation in the demo.

Having it run on a pico would be pretty impressive =3

http://cmuflite.org/

https://github.com/festvox/flite

> Having it run on a pico would be pretty impressive

Yes, although relative to the DECTalk DTC01, a Pi Pico is a beast !

Pico : dual core ARM @ 133 MHz, 2MB flash, 264K RAM

DECTalk: 68000 @ 10 MHz + TMS 32010 @ 20 MHz (5 MIPS), 256K ROM, 64K RAM

this is good to see. i also trained a stt under 500kb for sub dollar chips. it had about 20 words that it could understand(like start, stop, left, right, go, up etc) and then the spell mode where you could say the word spell and then say the individual english alphabets and close with spell. it was super fun to work on. these tend to be extremely unstable though, like confusion between p and t (at least for my accent). will have to try this one now.

I remember someone training smart kettle to use its speaker as microphone

IIRC the Alexa enabled voice remotes also used a similarly small model though perhaps not this small

Could you get people to use the NATO phonetic alphabet for the spelling part? I suppose a challenge is that many people don't know the whole thing, even if they're aware it exists.

NATO phonetic is to be understandable over a noisy radio channel, if you want just distinct sounds then Talon Voice users settled on shorter ones easier to use all the time:

    air a
    bat b
    cap c
    drum d
    each e
    fine f
    gust g
    harp h
    sit i
    jury j
    crunch k
    look l
    made m
    near n
    odd o
    pit p
    quench q
    red r
    sun s
    trap t
    urge u
    vest v
    whale w
    plex x
    yank y
    zip z

Interesting, that some words don't start with the letter they represent.

“Plex” ?

a) I went to a Talon Voice cheat sheet and copied them out because I didn’t remember them. Is plex not right?

b) Talon Voice is a configurable voice engine, the commands are something you setup and people use common shared starter packs and then edit their own, so there isn’t a formal standard, plex might be different in some setups?

c) they just have to be sounds you can vocalise which are distinct from each other, and it helps if they are one syllable when using them often; there’s no reason they have to be dictionary words or mean anything.

There's presumably a lot of benefit to having stable and widespread standards in this area (so people can re-use their skills!). So hopefully this is now stable at least within this community.

But all of the other words begin with the letter they represent. Plex ends with it. It sticks out quite a lot.

One could make "Pit" be "t", with that logic.

This property is also not true for "sit" or "crunch".

I've a local dictation workflow for coding, and one thing I've learned is that transcription accuracy is only half or even less than half the problem now. The other half is latency. Once the delay gets low enough that you stop noticing it, voice input starts feeling much more natural. It'll be interesting to see where this lands compared to Whisper-based setups for continuous dictation

This is really impressive.

If I get time, I would like to try compiling it to WASM. This would allow me to swap my robot poet’s native browser voice synthesis for it. Not sure if it is worth it, but it will be fun to play around with.

Edit: typo

[0] https://muffinman.io/bard/

It looks great, thank you! I'll see if I can use it for my in browser AI assistant project's ( https://aidekin.com ) voice part. It's currently using Nemotron-3.5-ASR and supertonic-3 but overall it requires 1.2gb download.

The voice activity detection alone here is compelling - very useful for doing things like highlighting a speaker who's transmitting in realtime. At that rate the impact on perf will be so minimal that you could easily run it in the browser across devices.

Given the tiny size of this, I wonder about possible future integration with esphome compatible hardware

https://esphome.io/

I suppose, but for home automation, esps are best for getting the audio to something more powerful. If this lets a raspberry pi do voice recognition really fast, that alone is worth it.

Compare

https://github.com/ggml-org/whisper.cpp/

This makes me want to have a server room with 5 of these around my house and control everything that house in LabRats

Voice is one of the most latency-sensitive modalities in AI. Moonshine is doing awesome stuff

Will it be able to understand my English with an Indian accent?

Is the dataset open

wow now that's a really tiny tts model. is there a comparison to https://github.com/kyutai-labs/pocket-tts ?

Great work!

ngl, it looks incredible

very nice I love it

Thank you for this. I love your work on Curb Your Enthusiasm.

What work?

Probably a joke that the author looks like someone on the show? I'm puzzled as well.

Possibly Cousin Andy? https://curb-your-enthusiasm.fandom.com/wiki/Andy_David

Played by the great Richard Kind, who my wife swears she saw on the Highline in NYC.

Reply on news.ycombinator.com