DeepSpeech 0.6 | Hacker News Reader

DeepSpeech 0.6

hacks.mozilla.org

278 pointspionerkotik66 comments

I do not understand how to use Deepspeech even in the most simple use case.

1. I want to teach it ten words. How do I do this?

2. I want to speak into my microphone (available as a Pulseaudio device) and recognise the words and output the words as a text stream on stdout. How do I do this?

This is the documentation:

https://deepspeech.readthedocs.io/en/v0.6.0/Python-Examples.... https://deepspeech.readthedocs.io/en/v0.6.0/Python-API.html

It does not answer the questions I have.

https://deepspeech.readthedocs.io/en/v0.6.0/DeepSpeech.html

The introduction page is full of incomprehensible jargon.

> 1. I want to teach it ten words. How do I do this?

I'm assuming you mean "my target vocabulary is only ten words". In that case, you have a couple of options:

1. Just use it as is and see how it does.

2. Create a language model containing just those 10 words so that the engine is constrained to that vocabulary, which should improve accuracy.

3. Train a (smaller) model from scratch, which requires (audio, transcript) pairs. This is basically research work, there's no recipe you can just follow. The advantage is that if you can make it work, it can be a much smaller model than a model trained for general English dictation.

> 2. I want to speak into my microphone (available as a Pulseaudio device) and recognise the words and output the words as a text stream on stdout. How do I do this?

We have a few examples with microphone input, but they're all based on voice activity detection: https://github.com/mozilla/DeepSpeech/tree/master/examples

I should write an example that is just continuous streaming and output without any voice activity detection.

I'd be happy to discuss further on Discourse: https://discourse.mozilla.org/c/deep-speech

>Just use it as is and see how it does.

How? The docs seem to be suffering from the same problem as the git docs: they assume the reader is already a domain expert, which would seem to run counter to the stated goal of being simple (and ubiquitous).

I tried to figure out how to "just use" deepspeech and failed.

There's a basic example here transcribing from an audio file: https://github.com/mozilla/DeepSpeech/tree/v0.6.0/#project-d...

There are also examples transcribing from the microphone as I mentioned above: https://github.com/mozilla/DeepSpeech/tree/master/examples/

I also have a small GUI example here [0], click once to start recording, once again to stop and show transcript. It receives the same command line arguments as the ones shown in the README link above (namely --model, --lm and --trie).

The README starts with a very basic installation and usage guide, I don't see how that's assuming any expertise.

[0] https://github.com/reuben/ds-qttest

> There are also examples

Fyi, the other 4 example links there are 404. Also in the root README. The linked README's are missing, or diversely suffixed and mislinked.

You are right, and I expect Reuben and team will fix this and are grateful for the heads up.

I was able to find the right links though, for example the WAV transcriber is at : https://github.com/mozilla/DeepSpeech/blob/v0.6.0/examples/v...

(Nod. I may have edited that out from under you - sorry. And thanks.)

Ugh, sorry. I fixed it on master. We'll probably cut a v0.6.1 soon to publish small fixes like these.

Thanks for all your work - it's appreciated.

I think you're being a little harsh.

1. Train it using the training dataset but change all word labels that aren't in your 10 word list to "OTHER_WORD" or whatever. There's probably not much point doing this. Alternatively I guess you can restrict the beam search to only look for your 10 words. Yes this is really complicated - it's a speech recognition engine - they are really complicated!

2. The API docs look pretty clear to me:

> createStream()[source]

> Create a new streaming inference state. The streaming state returned by this function can then be passed to feedAudioContent() and finishStream().

> feedAudioContent(args, kwargs)[source]

> Feed audio samples to an ongoing streaming inference.

> intermediateDecode(args, kwargs)[source]

> Compute the intermediate decoding of an ongoing streaming inference. This is an expensive process as the decoder implementation isn’t currently capable of streaming, so it always starts from the beginning of the audio.

Huh well that's a flaw the neglected to mention! I guess you can't really do what you want yet.

> Huh well that's a flaw the neglected to mention! I guess you can't really do what you want yet.

The documentation is outdated. That is no longer true, intermediateDecode is cheap. Thanks for noticing, I'll fix it.

So, funny story: I wanted #2 and did it myself. I was similarly frustrated with the lack of documentation.

It's been a little while since I got it running, but I basically got a siri clone working. If you want to test it out, I can try to answer questions / whatever problems pop up.

The code is here: https://github.com/shawwn/DeepSpeech/commit/01f5cf8d39c356ae...

As far as I know, you can simply run speech_to_text.sh. It will connect to your microphone and start dumping out transcribed audio to stdout.

It wasn't super easy, but once you spend a little time with the code you can sort of figure out ways to get it to do what you want.

EDIT: By the way, people will try to convince you that you're nuts and that the documentation is crystal clear and so on. Know this: It's not just you. I had the exact same experience. It's a recurring theme in AI programming.

The only thing to do is to either find someone else who feels similarly, or roll up your sleeves and dive into the code.

>By the way, people will try to convince you that you're nuts and that the documentation is crystal clear and so on

No it really is simple: the original implementation had a base of prefabulated amulition grammar, this was surmounted by a malleable logarithmic function in such a way that the two main spurving vocabularies were in a direct line with the panametric grammar fields. The latter simply consists of marzlevanes fitted to the ambifacient morpheme waneshaft to eliminate side utterances. This main winding is of the normal lotus-o-deltoid type, just placed in panendermic semi-boloid stators. Basically every second conductor is connected by a nonreversible tremmie pipe to the differential grammeters.

All these words are in: https://en.wikipedia.org/wiki/Turboencabulator

> It achieves a 7.5% word error rate on the LibriSpeech test clean benchmark

Anyone have a comparison for how good/bad that is compared to other solutions, and what it means for practical usage, if that can be guessed at from a single number?

You can find a leaderboard here: https://paperswithcode.com/sota/speech-recognition-on-libris...

It seems like sota is 2.20% word error rate

I like reading through https://github.com/syhw/wer_are_we sometimes

Note that the leaders there are not "near real time" streaming type implementations like this is. They have the benefit of having full context (they act on the full input not a stream of current state). 7.5% is really good for this type of low latency stream based system.

LibriSpeech test clean is without background noise, with a good microphone, and most likely biased to native speakers (it is 50/50 male/female at least). In summary, it's not what an ASR engine will encounter. However, the dataset is used as a common method to benchmark ASR engines and thus that number is valuable. The SpecAugment paper by Google for example claims a 2.5% word error rate [1] on that corpus (using only LibriSpeech for training data).

[1]: https://arxiv.org/pdf/1904.08779.pdf

It’s worth mentioning that while DeepSpeech 0.6 supports SpecAugment, the released model was not trained with SpecAugment switched on.

Author here. I would add to what the sibling comments have mentioned by saying that SotA results should be taken with a grain of salt. Our engine is capable of streaming (processing the audio as it's being recorded), which is not doable with architectures that have bidirectional decoders or attention mechanisms that require the whole encoder input ahead of time.

For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds. This is why, if you run a standard test set like LibriSpeech on, say, a commercial offering from Google, it will perform considerably worse than state of the art according to Google papers.

This repository [0] has a benchmark of some commercial offerings. Our model beats all of those on Librispeech clean and other (except for Speechmatics on Librispeech clean), as well as on Common Voice. But note that the Common Voice corpus used in that benchmark is very old.

In sum, I would compare this against solutions that go for the same space: fast, client-side ASR, rather than state of the art.

[0] https://github.com/Franck-Dernoncourt/ASR_benchmark#benchmar...

Hi! I'm from the RWTH team (https://paperswithcode.com/sota/speech-recognition-on-libris...). Our best system (2.3% WER) (trained only on the Librispeech data, i.e. much less data than your model) currently is a hybrid HMM/NN model, and you are right, the acoustic model uses a BLSTM. However, we have shown in other work (https://www-i6.informatik.rwth-aachen.de/publications/downlo...) that you can expect similar performance in an online system, or not much worse (there is also many related work, which often is maybe 5-10% relative worse than the offline system).

I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right? As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants. They should give you all the same properties (online capable, low latency, simple, end-to-end), but much better WER performance. Or is this simply because there currently is no simple ready-to-use implementation for those? Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon.

Hi!

> I guess you prefer an "end-to-end" model over a hybrid HMM/NN model, for simplicity, right?

Simplicity and ease of targeting other languages, yes. We're a small team.

> As far as I remember, you use CTC, right? I always wondered why you have chosen CTC, and not some better model, like RNN-T, RNA, or some of the streaming attention variants.

We started DeepSpeech in 2016, before these recent developments for end-to-end ASR were mainstream/SotA.

> Or is this simply because there currently is no simple ready-to-use implementation for those?

Implementing the model architecture for training is only part of the problem for us. We have a hand-crafted inference graph to allow for small and efficient client code and inference models, and the more complex the architecture is, the trickier it gets to make sure it all works on all platforms, including TFLite, with quantization, etc.

We're investigating alternative architectures as well as mixed CTC/RNN LMs to deal with language model size, but no final decisions made yet.

> Note that we published some TF code recently for some streaming attention variants, and plan to publish some RNN-T/RNA code soon.

Nice! Can you share a link to the streaming attention code?

I really hope you adopt the latest models particularly streaming attention variants. I think you should validate with users the assumption that latency is more important than WER.

IMHO the WER is more important than latency improvements in the millisecond range. The most frustrating thing is having to dictate over and over and the transcription is incorrect each time.

Consider that the time to a correct transcription is the latency plus error correction. If error correction is manual it will be orders of magnitude slower, so optimize for WER.

I’m terms of competition, Siri has latency in the 5+ second range due to the network call especially in area with poor data rates. I think a client side model like yours will easily win in this category. If you’re already ahead here, why not focus on WER next?

Another great capability is to generate alternative transcriptions for words with low confidence values to allow for quick error correction. Do you offer something like this today?

Also, consider the long term view that new models are constantly being released and refined. It’d be best to have an architecture that allows quick replacement without a lot of hand tuning, or where the tuning can be automated to a greater extent.

Thanks for all the hard work you have put in so far @reubenmorais

+1000 to @mostlyjason's comment - Great latency figures mean nothing if the word error rate is high, since it dents confidence in the output (so why use DeepSpeech?) and (as the parent comment notes) necessitates manual error correction.

I would love to see a future release focus on optimizing WER for these reasons.

> Nice! Can you share a link to the streaming attention code?

It uses our TensorFlow framework Returnn (https://github.com/rwth-i6/returnn). We currently only have some configs/code online for hard attention variants, or segmental models. They can be found here: https://github.com/rwth-i6/returnn-experiments/tree/master/2...

Note that the configs are maybe not so cleaned up, as this is very much research. This is for a paper we submitted to ICASSP. We did not publish the paper yet elsewhere, but I can send you a copy by mail (just contact me: albzey@gmail.com).

Also, as this is research focused, the encoder here is also a BLSTM, because we wanted to compare this work to other global soft attention models, and have the comparison mostly focused on the streaming attention modeling aspect. And also there is some lookahead, which is currently unlimited. So it would need a few modifications to really be used for online streaming. I'm also not sure whether this is the best model, or whether some of the many other variants (MoChA, RNN-T, etc) are maybe better.

Edit: I forgot, we also have some local windowed attention variants, which can also be applied for streaming: https://github.com/rwth-i6/returnn-experiments/tree/master/2...

You 100% tuned your hyper parameters and LM for librespeech, and only using librispeech data probably helped the result. When you start training systems for LVCSR it's normal for the WER to go up on some test sets. (This is why you can never expect a commercial system to get SOTA on librispeech, despite it being much better system in reality)

But these results are about in line with other CTC systems. The original Baidu paper [1] had 7.89% with original Deep Speech and 5.33% with "Deep Speech 2" where they threw more complicated networks at it and added another 11K hours of external speech traingin data.

[1] https://arxiv.org/pdf/1512.02595v1.pdf

NVIDIA has QuartzNet which contains only 19M weights and achieves 3.9% on test clean without language model and less then 3 with LM. Code (Pytorch): https://github.com/NVIDIA/NeMo Paper: https://arxiv.org/pdf/1910.10261.pdf

> Our engine is capable of streaming

A bit misleading given that decoding can't be streamed yet.

I’m not sure what you mean. The image in the blog post clearly shows small chunks of decode time happening during the speech (every second).

The API documentation says decoding starts from the beginning every time. Could just be the Python API hasn't been updated I guess?

Edit: never mind - the API documentation was just out of date.

> For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds.

You should follow Google's approach - give fast live results that don't depend on data from the future, but also go back and correct old words when you do have that data. It's kind of how humans work really.

> For real world applications, this is absolutely crucial, users want latency numbers on the order of milliseconds, not seconds.

My wav2letter research does not use streaming, despite supporting streaming, because I get lower average latency (roughly 0.02x RTF including decoding, or 80ms for a 4000ms input audio like you describe in the article) by running the entire encoder CNN at once instead of running it in chunks.

This is also nice because the CPU usage is practically nothing (0.1%) until you stop talking.

I have a web demo here [1], with a relatively terrible language model (mostly just been working on acoustic modeling so far, as it’s just me). Most of the latency is waiting for the JavaScript VAD (which is much slower than webrtcvad and I couldn’t figure out how to tune it) and waiting for the network. If you look at the network inspector, the server should report its encode and decode times.

[1] https://web2letter-west-1.talonvoice.com/

7.5% WER is much much worse than the state of the art, but it's also important to recognize that LibriSpeech is kind of a ridiculous data set. It's 1000 hours of volunteers reading public domain novels out loud. It's very unlikely to match your use case. It's widely used in academia purely because it's larger than other freely available data sets.

I just found https://voice.mozilla.org/

Besides being a great resource for speech analysis, this could be a real game changer for acquiring listening comprehension in a foreign language.

I feel that even after a few years of learning a new language I still have trouble with listening. Part of that is that it's often all or nothing, even one or two unknown words in a sentence means I can't understand the sentence. But worse is that most language teaching materials use a very small set of native speakers, which deprives the learner's brain of being able to generalize.

Some of the English samples by non-natives were completely incomprehensible. I hope the system learns to be accent free.

They have a validation process, and the data is mainly meant to train speech-to-text.

For that, it's important to include accents and even non-native speakers, especially in English.

I think that those wavenet models for TTS were able to be conditioned by gender and maybe accent, if they had the data.

given that there are more non-native than native english speakers on the planet I think it makes a fair bit of sense to include non-native data in the training set

Perhaps there's an opportunity to create a demo webpage using tensorflow.js? Especially with the new TSLite support. To raise visibility and awareness. The posenet webcam demo[1] for example, seems to have ~2k links.

[1] https://storage.googleapis.com/tfjs-models/demos/posenet/cam...

Congrats to the team on the new release! Following their progress since a while. It's an important project, and I'm very happy about the size reductions while they still delivered a WER improvement over the last release. Amazing!

Thank you for your work on the Rust bindings :)

This reminds me of something I would love to see happen but I don't have the skills to put it all together. I really think there's some potential merit to a reading coach app(lication) that listens to someone read and looks for weaknesses/disorders/etc compared to a trained model. It could provide those diagnostics to an educator, guide the content to focus on those, coach the reader directly, etc.

It all seems very doable based on what I see in the technology today, I just don't have the skills to do it.

I'll be honest... that seems like a pretty tough sell to me. You could be looking at anything from a speech disorder to a learning disability to a perfectly healthy normal child with idiosyncrasies and those things are usually left to professionals with graduate degrees. The liability seems high, you've got both the potential stigma associated with incorrectly flagging a child, or worse, missing an otherwise obvious issue that means treatment is delayed during critical years.

I totally understand where you're coming from, but the reality is that there is a lot of imbalance and delay in this type of assessment and educational support already. My wife has been an English language arts teacher and instructional coach for almost 20 years. While they definitely have standard diagnostics and regular assessments to try to keep kids on track, it's essentially best effort. And when you move out of 4th/5th grade in many districts you start getting to 90:1 teacher-student ratios with 1-2 interns available to do 1:1 reading support across maybe 500 kids. And this is in a middle class school district.

There's definitely some ways it can go bad, but even basic fluency monitoring without any active remediation attempts would be a good addition.

> even basic fluency monitoring without any active remediation attempts would be a good addition.

How do you define fluency? I guess it involves some combination of speed and accuracy. Speed shouldn't be too hard to measure, but for accuracy you'll probably need a model that's more accurate than 7.5% WER and can handle the difference in vocal range between children and adults. Otherwise the speech disfluencies you want to detect will be drowned out by the model not correctly recognizing actually pretty clear speech.

In the US, fluency metrics in ELA vary across standards/states but are generally coarse grained and somewhat subjective. Here's an example of the standard that a student is expected to meet for fluency in the fifth grade (bottom half): http://www.corestandards.org/ELA-Literacy/RF/5/

You definitely would have to use a model trained from content in the target audience (probably down to the grade level as things change so dramatically from year to year) as well as probably some labeled examples of students with various reading/speech deficiencies. This of course would lead to a lot of challenges from a privacy/regulatory/etc perspective and the entire thing would be challenging from an optics perspective (AI teachers taking over, etc etc).

Edit: One thing to keep in mind with accuracy is that there is no ambiguity what the word should be, the question is how closely the utterance matches the expected sounds of the word.

I've seen this referred to online as "Intelligibility Remediation" (a term I always forget when I'm trying to find the details months later!)

Certainly seems like an area with huge potential application.

I was very exited about this, then I tried it with the pre-trained model and I recorded "Hello this is a test message" 3 times and it was inferred as "a sassanian", "he is a paris" "he states that" ...

I can't seem to find pre-trained models for other languages (French). How long did you training take for those in English? Do you think it makes sense to start from the English one? Thank you!

I'm starting to look into the MycroftAI. It sounds like with this release I could use Mycroft+DeepSpeech on a new RaspberryPi for a completely offline smart speaker, do I understand that right?

Yes, you are correct tensorflow lite can be done on the raspberry pi. https://www.tensorflow.org/lite/guide/build_rpi

Correct. The latest release with the TensorFlow Lite model runs in real time on a Raspberry Pi 4. I'm not sure if the Mycroft integration is updated to the latest version though, as it was just released.

> DeepSpeech v0.6 with TensorFlow Lite runs faster than real time on a single core of a Raspberry Pi 4

This is great news.

I'm not very familiar with the deep learning framework ecosystem. Does anyone know what the simplest way would be to incorporate a DeepSpeech model into a WebAssembly project?

For instance, is it straightforward to compile Tensorflow Lite with emscripten? If not, can this TensorFlow model be run with tensorflow.js? Or can it be converted to some other format that is easier to use with WASM?

Tensorflow lite, interesting. Has anyone tried this with Google Coral USB accelerator? I got one laying around but very little experience with ML. I could get the USB Accelerator working with the pretrained posenet after much messing around, but my chances with DeepSpeech are small to none. This seems like the perfect fit though.

To the best of my knowledge, DeepSpeech currently isn’t compatible with any inference or training accelerators (other than NVIDIA GPUs).

But the whole point of the TFLite model is that it has low resource requirements so an accelerator for inference shouldn’t be necessary.

TFLite on Android with the NNAPI delegate is theoretically supposed to leverage hardware accelerators. We haven't tested it extensively though.

I wonder if it is any good at out-of-vocabulary words? Is it hard to teach it new things like medical terms and such?

In the post I mention a Brazilian company who's using DeepSpeech for vocabulary constrained, medical applications. They've trained their own Brazilian Portuguese models.

The English acoustic model that we released does not have a fixed vocabulary. What determines the vocabulary is the language model, which can be created from text. So if you have in-domain text, and would like to try it out, I would say the first step is to create a language model using your text and then experiment with it.

We have some documentation on how to create the LM here: https://github.com/mozilla/DeepSpeech/tree/v0.6.0/data/lm

It's not super detailed, but I'd be happy to answer questions on our Discourse: https://discourse.mozilla.org/c/deep-speech

I did write a small program to make custom vocabularies and datasets for DeepSpeech a while back when I was messing with it: https://github.com/faissaloo/deepspeech_dataset_builder It could use some TLC but it does the job.

It's using letters, not words, so in theory anything that can be spelled in English can be output:

https://2r4s9p1yi1fa2jd7j43zph8r-wpengine.netdna-ssl.com/fil...

Or, if you prefer the code: https://github.com/mozilla/DeepSpeech/blob/0427c1572ac8f7253...

(well, I suppose technically it's in https://github.com/mozilla/DeepSpeech/blob/master/data/alpha... through https://github.com/mozilla/DeepSpeech/blob/master/util/text...., but I mean come on)

In practice ... meh. It gets some things predictably wrong. They're readable, just wrong. So the good news is that WER is words it has 100% correct. The remaining errors aren't usually totally unreadable, just irritating, going from "totext" instead of "to text", but of course names go pretty wrong and don't get better. It butchers my name, of course.

If you test it yourself, please do realize it really doesn't deal very well with accents.

Looks great. Is it related to any of Mozilla's product?

We'll have more to share about that soon :)

So who will be developing an open source assistant like Goggle Home / Alexa? :)

If you do a 1 minute google search you will find that there are multple open source voice assistant offerings already, for example mycroft.

Most of which are not fully open, e.g. do not have open speech recognition.

Any recommendations on a good raspberry pi hat or microphone to test this?

Reply on news.ycombinator.com