Hacker News Reader

> I found mycroft, Jarvis and a few others, but either got bogged down in dependencies or configuration.

More recently, there is also Rhasspy (https://rhasspy.readthedocs.io) and voice2json (https://voice2json.org). I'm the author of both, if you have questions.

> Why is everything done in "the cloud."

Besides being a way of collecting data and ultimately making money, it avoids some of the "bogged down in dependencies or configuration" problem. My voice projects need to run entirely offline on a variety of hardware and operating systems. If each client was just a little app piping audio data to a cloud service, it would be way easier to write and maintain.

> So the only way to semi-accurately do voice recognition is to source algorithms that re-train off of millions of people?

Nope. You can absolutely tune a speech model locally on your own samples and get great accuracy. The trouble comes with open-ended speech: people expect the voice assistant to recognize that new artist or movie they heard about yesterday. That doesn't work without upkeep somewhere.

Rhasspy/voice2json are intended for pre-defined voice commands using a template language. You can get almost perfect accuracy with this approach, even with millions of possible commands. Re-training only takes a minute, so personal upkeep isn't bad.

> Even if it meant I needed to download a 230GB data set, I'd gladly do it, if it could remotely help in getting away from these data silos.

It's a lot less than that; at most 1-2GB for a given language, usually a few 100MB: https://github.com/synesthesiam/voice2json-profiles/

Very cool projects. Thanks for sharing (and creating them!).

@synesthesiam your work is wonderful and I am willing to test both ASAP.

It kind of fills a gap left from snips.ai, will check and compare with what's on the market now.

Super recent (less than a year for both projects, but may be wrong) and apparently well documented. Hats off

That is what they are working on, but that needs high-quality training data: https://commonvoice.mozilla.org/en

This is just another way of gathering that data. (If consented to.)

Mozilla DeepSpeech is trained on about 2000 hours of audio that is mostly spoken by American males. It has little ability to handle noise or accents and has a 5.97% Word Error Rate on LibriVox (which is noiseless, plain spoken english).

Meanwhile, Google, Microsoft & IBM have tons of fresh audio coming in constantly to use in augmenting their models.

Baidu was able to build a competitive English Speech to Text model with 5000 hours of quality audio to train against.

Mozilla did create Common Voice to address this serious data gap, but it has only collected 1492hrs of validated English audio: https://commonvoice.mozilla.org/en/datasets

There's tens of millions of hours of video content out there that has been subtitled pretty well, and I'd wager a lot of it is under usable licenses for Mozilla. Has that been considered?

If you know such sources, file an issue, and better yet, download the video content yourself and publish a dataset.

But note that raw video content is not training data. It has to be segmented to be in short enough parts for training (few seconds), the subtitles have to be aligned to match what's said precisely, and one needs to balance the data, e.g. when 90% of speakers are men and 10% are women, you have a problem.

Gotcha, all excellent points.

> why haven't we been able to advance on-device processing?

Probably because getting the payloads shipped for server side processing provides a constant stream of training material updates? That's my cynical take, at least.

But there's got to be more. Paraphrasing H.L. Mencken[0], for every complex problem there is an answer that is straightforward, easy to accept - and wrong. I remember how in the late naughties Nokia had an early version of on-device speech recognition: long-press the trigger key and you could call people by uttering out a saved "voice tag". IIRC the feature picked the right entry roughly 1/3 of the time.

For what comes next I'm relying on hearsay, but from what I heard at the time, the feature was originally developed at MIT. Nokia then financed the team to optimise their code and underlying detection model to be ported to ARM and to fit the constrained memory/CPU envelope. Because Nokia definitely had collaboration with MIT at the time, this is at least plausible.

If people are expected to use speech-to-text in real life, it has to work within very strict boundaries. Low latency and high accuracy are table stakes. At least some level of contextual awareness would be nice. As long as predictive text input routinely provides us with meme-worthy failures, I won't expect anything better from (fundamentally noisier) speech inputs. And if server-side processing is the only way to get performance from dismal to somewhat functional, practical applications don't have much of a choice. Plus, you don't have to ship your model to end user devices.

For what it's worth, I dislike voice interfaces. But when they do work, I dislike them less than Byzantine and user-hostile phone menu systems. I guess that qualifies as progress.

0: https://quoteinvestigator.com/2016/07/17/solution/

1: https://nokia-e71.helpdoc.net/en/nokia-e71-user-guide/phone/...

Try Nuance Dragon options (no affiliation, just a good tool). They have software that works offline, without cloud. They have a very competitive voice recognition quality for the supported languages. There are some products that are explicitly cloud-based (like mobile ones), but there are also desktop versions that do all the processing, inferring and even learning of your voice and new dictionaries on your local machine. You will need a somewhat powerful PC, but nothing crazy.

(PS but to be fair, it's a dictation tool with some extra commands, even if a very good one. It is not really a voice assistant. But I'm not sure if that is harder or easier to do. There are open-source offline assistant tools too that work quite well already, because once you have a set of pre-determined phrases/formulas, they are much easier to parse.)

Speech-to-text is relatively easy (not to discount the decades of work that went into it, lots of which was done by Dragon), but extracting intent and sending the user to the correct place is what is really difficult. IMO it is essentially an unsolved problem.

Let's say you have two companies A and B. A does on-device only and B does in the cloud. Even if their systems begin being just as accurate, B has an ever growing stream of new training data with which to improve their system. A does not. Over time B will become better and better while A does not. Worse, by downloading the model onto devices the model can be easily copied, cloned, reverse engineered, interrogated and otherwise used by competitors.

Kalliope[1] is a no code, but Python-based, voice assistant. I like it because you can switch out voice backends and it is designed to be modular with node-like constructs to build signals and responses.

I stuck it in a container and can control Home Assistant with it, run scripts, etc.

[1] https://github.com/kalliope-project/kalliope

Even Apple, with its focus on privacy and local AI, needs an internet connection for most Siri queries.

The HomePod has a relatively powerful CPU for what is doing, I'd assume there's lots of headroom to do local intent recognition on device, but Apple likes to get new training data constantly too, so they haven't really tried.

Drives me nuts. "Hey Siri, Pause" shouldn't require an internet connection. I often have bad or no connection (eg podcasts while hiking). Voice commands must be ubiquitous. If I have to fail-revert back to touch, what's the point?