Towards an ImageNet Moment for Speech-to-Text
thegradient.pub
thegradient.pub
Many of the companies working in this space who aren't Google/Amazon target less general domains, using domain-specific languages models to skirt around the lack of annotated acoustic data. But the lack of annotated acoustic data is a real problem, so any effort to provide data to everyone (particularly for a non-English language) is extremely admirable. I applaud the OpenSTT team for their work.
I found the "Why not share this in an academic paper" part interesting (and I expect it to be interesting to HN). As more practitioners enter the field, a lot of empirical knowledge will be gained but will be tricky to share since they will likely not know the convention of academic writing, Latex, etc. (to be clear, I think when done well papers are a good format to share information). Perhaps stuff like this and distill.pub will become more common? Seems like a good development IMO (as is researchers writing blog posts in addition to papers).
Most TV stations broadcast with closed captioning and most movies have subtitles available, which should give millions of hours of training data.
It's technically copyrighted, but as long as you don't distribute the video as well, they aren't going to care about it.
Also the complaint that the data lacks compression artifacts is completely ridiculous and absurd: if you want compression artifacts, just compress and decompress the speech yourself!
You can also pay people to transcribe audio or read text (or even do it personally), and since this is something anyone can do it can be paid at very low rates.
Closed captioning and subtitles are often used, but they are 'dirty data' - they are usually not a one-to-one match, the differences mess up training. And licensing issues are a pain; "they aren't going to care about it" is not a solution and things like background music in movies (the rightholders very much do care about it, even if just to make a point) make it pretty much impossible for e.g. some university to legally distribute a dataset that includes movie audio tracks; so they can run some experiments on it themselves but as soon as you want any collaboration, that data is taboo.
Paying people to transcribe or read works, but it's not cheap. Reading 10000 hours takes at least 10000 hours of paid work and generally a bit more than that - mistakes matter, so you need review and correction. If you try transcribing things yourself, a tiny 100 hour dataset is going to take you at least something like 400 hours, which is months of work.
So that's the point - getting a usable dataset for some language costs hundreds of thousands of dollars if you're frugal, and millions if you want good results. And there are very many languages in the world.
Let's say you decide to grab all of those TV broadcasts and use them as training data (we'll ignore the fact that very little speech is as clean as the speech in TV broadcasts).
Every mistake that occurs in the transcriptions you're using for training data represents the potential for mistraining your model when you perform a forced alignment [0]. Since most forced alignment tools perform a feature transformation step to increase how discriminative the phones are in feature space (ie how different they are from each other) and a step to reduce the number of dimensions, small errors in the transcriptions can result in huge differences in decoder behavior. Now factor in that your lexicon (the set of pronunciations you're using to map the orthographic transcriptions onto the possible phoneme strings that the speaker could have actually said) is almost certainly incomplete and very likely doesn't include nearly enough pronunciations for speakers with non-standard dialects or speakers with accents. Plus there are all sorts of typos in the original transcriptions; this often means that you need to run g2p to generate shitty pronunciations for words that actually don't exist, adding further errors into your model.
Suddenly, you start thinking that all these decoder problems you're seeing are problems with the training data as it currently exists and the only solution is to do a phonetic transcription of the training data! You can still get that done, but since you've realized some of the blame lies with your lexicon you can't just dump the videos onto Mechanical Turk and have random people do them: you now need to get people that are familiar with phonetic transcription. And you decide that since even experts mess up a fair amount when transcribing speech, you're going to get 2 people to transcribe everything and throw away everything where they disagree. You aren't Google, so your training budget isn't that big. For each hour of speech you're spending 2 expert man hours and still throwing away 6 minutes and you need 10K+ hours of data at the bare minimum! And looking at the literature, it looks to you like there is a near linear correlation between who has the most data and who has the models that perform the best!
So that's why people end up claiming that a fundamental problem with building out ASR is a lack of training data: because getting it and making it usable is very expensive, and because having lots of good training data let's you avoid addressing other difficult issues in ASR.
[0] http://www1.icsi.berkeley.edu/Speech/faq/forcedalign.html
https://github.com/gooofy/zamia-speech#download
https://github.com/mpuels/docker-py-kaldi-asr-and-model
(also supporting german language pretty good but not production ready)
It looks like their research is against an older wav2letter model (likely the conv_glu model from the original 2018 Gated ConvNet paper). Facebook has released a lot of interesting architectures since then, my favorite of which is the Streaming ConvNet arch [1]
Ideas from the post compared to my experience with wav2letter:
1. Model Stride - increased model stride to 8x
Streaming ConvNet increased the stride to 8, while the original conv_glu model had stride 2 (cite: [2]), so Facebook agrees on this one.
2. Compact Regularized Networks - use separable convolutions, add skip connections, and attention modules
Streaming ConvNets use Time-Depth Separable Convolutions, but no skip connections or attention. Facebook's "state of the art 2019" research tried a lot of architectures, the most accurate of which was a transformer model trained with attention and seq2seq criterion. However in my experience it was very large and slow to train.
3. Using Byte-Pair Encoding
Streaming ConvNets model uses 10,000 word pieces, which is a similar thought (the token set FB uses is here: [4] but I've had better luck with a sentencepiece model trained on more text data) instead of a ~26 letter alphabet from the original conv_glu model.
4. Better Encoder
The blog post conclusion here isn't very obvious, it seems to hand wave that you could maybe use a transformer architecture but doesn't go into specifics.
5. Balance Capacity
I don't completely understand what they changed here but it seems useful.
6. Stabilize the Training in Different Domains, Balance Generalization
When adding domains to wav2letter, I've basically trained entirely from scratch using all available data. I long for a catastrophic forgetting mitigation like EWC but for now I've had a lot of success by just making the data funnel bigger and more varied. Data matters quite a bit, in this post [5] I compare a model I trained on a wide variety of data, with Facebook's equivalent model as well as Facebook's State Of The Art model. The wide-variety model generalized significantly better than both of Facebook's examples that had only trained on audiobooks.
7. Make a very fast decoder.
The beam search decoder in this branch [6] of Talon's wav2letter fork was at the time significantly faster than Facebook's decoder. It was hand optimized and even includes some code for threaded decoding (which I'm not using at the moment because single threaded is fast enough for my workload). On single threaded decoding it can hit a realtime factor in the ballpark of 0.01x, or 100 seconds of audio per cpu second. With my smallest model (a significantly reduced size conv_glu wav2letter model), which is also in the 0.01x ballpark for encoding, we can consistently hit around 0.02x end to end on a consumer CPU.
To demonstrate, here is a random sample of debug timing information from my machine when recently decoding command phrases with wav2letter during real interactive use (the very long audio samples aren't the model's fault, I was testing a mic in a noisy environment):
[audio]=1260.000ms [emit]=16.164ms (0.01x) [decode]=11.326ms (0.01x) [total]=27.490ms (0.02x)
[audio]=1260.000ms [emit]=16.164ms (0.01x) [decode]=11.326ms (0.01x) [total]=27.490ms (0.02x)
[audio]=930.000ms [emit]=12.853ms (0.01x) [decode]=3.716ms (0.00x) [total]=16.569ms (0.02x)
[audio]=1080.000ms [emit]=14.936ms (0.01x) [decode]=3.296ms (0.00x) [total]=18.231ms (0.02x)
[audio]=1020.000ms [emit]=12.976ms (0.01x) [decode]=10.058ms (0.01x) [total]=23.034ms (0.02x)
[audio]=930.000ms [emit]=13.836ms (0.01x) [decode]=9.998ms (0.01x) [total]=23.834ms (0.03x)
[audio]=1140.000ms [emit]=14.535ms (0.01x) [decode]=3.589ms (0.00x) [total]=18.124ms (0.02x)
[audio]=1650.000ms [emit]=20.516ms (0.01x) [decode]=5.281ms (0.00x) [total]=25.797ms (0.02x)
[audio]=3480.000ms [emit]=31.386ms (0.01x) [decode]=32.966ms (0.01x) [total]=64.352ms (0.02x)
[audio]=29760.000ms [emit]=253.671ms (0.01x) [decode]=325.198ms (0.01x) [total]=578.869ms (0.02x)
[audio]=8850.000ms [emit]=70.844ms (0.01x) [decode]=76.267ms (0.01x) [total]=147.111ms (0.02x)
[audio]=960.000ms [emit]=13.924ms (0.01x) [decode]=8.135ms (0.01x) [total]=22.060ms (0.02x)
[audio]=1770.000ms [emit]=19.760ms (0.01x) [decode]=18.612ms (0.01x) [total]=38.372ms (0.02x)
[audio]=930.000ms [emit]=12.488ms (0.01x) [decode]=6.842ms (0.01x) [total]=19.330ms (0.02x)
[audio]=1050.000ms [emit]=15.726ms (0.01x) [decode]=7.645ms (0.01x) [total]=23.371ms (0.02x)
[audio]=930.000ms [emit]=13.899ms (0.01x) [decode]=8.714ms (0.01x) [total]=22.613ms (0.02x)
[audio]=40410.000ms [emit]=292.916ms (0.01x) [decode]=289.109ms (0.01x) [total]=582.025ms (0.01x)
[2] https://github.com/facebookresearch/wav2letter/issues/543#is...
[3] https://github.com/facebookresearch/wav2letter/tree/master/r...
[4] https://dl.fbaipublicfiles.com/wav2letter/streaming_convnets...
[5] https://github.com/facebookresearch/wav2letter/issues/577