Hacker News Reader

Comment by Nimitz14 | Hacker News Reader

It is very unlikely a E2E toolkit can outperform kaldi if they're trained on the same data.

Also I suspect these guys aren't using the latest kaldi architectures.

What's your reasoning? Google's latest published research is analogously parsimonious with this approach. They are doing decoding using a simple beam search through a single neural network. See https://ai.googleblog.com/2019/03/an-all-neural-on-device-sp...

very very interesting, didnt know this one. as soon as i am having some hours left will try to run some evaluation on this. after all, for my project only performance in german language counts

we used existing recipies for training (tuda german), slightly adapted and with additional training data. i am sure with more knowledge we would have made some extra percent ... performance is not really good, but good enough for our purposes, and with a satisfying cost vs quality trade off

You should look at the latest recipe's for tedlium, librispeech, swbd etc you'll get significantly better results with those models.

Did you miss wav2letter achieving a new 2019 SOTA on librispeech? Both projects have a _lot_ of architectures and papers in play, I know it can be hard to keep up.

Aren’t some of the peak Kaldi numbers also with _huge_ language models that aren’t so practical to deploy? The best kaldi results I’ve found here both use “fglarge” LMs, but a quick search didn’t tell me how big that actually is.

The best I can find for the test set are (from main RESULTS):

    test-clean: 4.31% WER
    test-other: 10.62% WER

From here (not sure why buried): https://github.com/kaldi-asr/kaldi/blob/master/egs/librispee...

    test-clean: 3.80% WER
    test-other: 8.76% WER

This is the 2019 wav2letter SOTA: https://github.com/facebookresearch/wav2letter/blob/master/r...

Sure, one of their techniques is to semi supervised train on 60k hours of audio, so I’ll post their best numbers both with and without that step (as you said “if kaldi is trained on the same data”)

Without 60k unsupervised audio:

    test-clean: 2.31% WER
    test-other: 5.18% WER

With 60k audio:

    test-clean: 2.03% WER
    test-other: 4.11% WER

That’s so far ahead of kaldi’s librispeech RESULTS, let’s go up the list and look at their acoustic model tests that don’t even bother with a language model:

Without 60k or language model:

    (test-clean / test-other WER)
    3.05 / 7.01

With 60k without language model:

    2.30 / 5.29

Am I missing a kaldi whitepaper here?

I found “State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention With Dilated 1D Convolutions” on wer_are_we claiming 2.2 / 5.8 using Kaldi as a base, which is impressive, but they changed the network architecture and I didn’t find this in any of the open-source kaldi librispeech subdirectories. (If arbitrary model changes count, at some point you might as well treat both wav2letter and kaldi as “tensorflow with some speech helpers” because I’m sure you can implement either framework’s best neural architecture in the other framework.)

If there’s a better paper on kaldi than 2.2 / 5.8 you should comment with it and PR it to wer_are_we, because wav2letter is currently beating every other LS result posted there.

The reason those kaldi results are noticeably worse is because they're not using RNN rescoring. The E2E models also take far more compute to train (facebook and google train their models for hundreds of epochs, kaldi for... 4!). All the papers you read game results by adding unpruned ngram LMs or massive NNLMs.

Point being if you put some more effort I believe you can get better results with kaldi. Remember we're discussing with which toolkit one can achieve better performance (given a certain amount of effort), which is not the same as just taking the results from different teams some of which have spent an order of magnitude more compute in gaming the results.

For SOTA results with a hybrid approach check out RWTH's stuff. The kaldi people have recently been busy rewriting their backend to work with pytorch.

In general using librispeech to gauge performance is a bad idea. It's read speech. I prefer to base my opinions on real world datasets.

Have to say I am impressed by how good their (FB) results are without using an LM though.

I agree that librispeech isn’t the best benchmark, but it’s the benchmark we have. If you have a significantly better benchmark that both wav2letter SOTA and Kaldi SOTA have participated in, I’d love to see it. Without some kind of numbers cited, it’s easier to assume you’re speculating. Especially because, as I think we both acknowledge, wav2letter requires a lot of training power to actually use, so I doubt you’ve done the training yourself in the same way you’ve worked with Kaldi to be able to talk about it authoritatively. (One of my goals with releasing w2l models is that people will be able to do e.g. transfer learning or distillation and not repeat the expensive up front training themselves)

RWTH has an impressive entry on the wer_are_we list for librispeech (2.3 / 5.0), but unfortunately I couldn’t find a way to actually use that research without a lot of work. Facebook has easy recipes posted that reproduce their results, which have been quite nice to work with. RWTH, Google, and Capio held an impressive top spot in the results for a while, which might be where you’re basing your judgement.

I don’t think the biggest benefit of wav2letter was the architecture (until their SOTA paper), I think it was the speed and simplicity. I managed to port the original wav2letter inference to numpy/pytorch in two small files (same model weights, same inference output) because it was such a simple convolutional architecture. (The only fiddly part was matching the output of their featurization code)

I agree the data and training overhead is a big aspect, which is why I’ve been training and releasing freely available wav2letter models on far more data than what Facebook has published using librispeech.

My upcoming goal is to reproduce the SOTA results but with stronger training data. My baseline data is around 4000h currently and is far more diverse than librispeech. Unfortunately I’m independent and can’t afford the LDC datasets, though I have some neat ideas for extending FB’s semi supervised work to even more data.

You are right I've never tried out wav2letter on a large amount of data.

Cool project! I'll try out your models.

are there any wav2letter models ready for download ?

Facebook posted some of the models listed in their SOTA paper already. I haven’t tried these yet.

https://github.com/facebookresearch/wav2letter/blob/master/r...

My models are here: https://talonvoice.com/research/ I haven’t yet posted the model I’ve been working on most recently. I’m 120 epochs into a large size model trained on all of my datasets. I also have another 1000-1500h of audio I haven’t finished prepping to train on.

Here is a web demo. It’s currently running a slightly older checkpoint of my WIP large model, and the deepspeech LM: https://web2letter-west-1.talonvoice.com/

so this is a real cool project. as soon as you finished your training I will be happy to add it as option (or maybe default setup) in the botium speech processing setup (if you want that).

Do you have any experience with online decoding in wav2letter ? Is there something like a Websocket API available somewhere ?

How does accuracy compare to what you’re using now?

the german model is from the kaldi tuda recipe with WER of 15%.

the english is from the tedlium recipe with WER of 7%.

room for improvement, but for our original purpose it was sufficient.

Reply on news.ycombinator.com