Hacker News Reader

Comment by albertzeyer | Hacker News Reader

This seems to be a CTC model. CTC is not really the best option for a good end-to-end system. Encoder-decoder-attention models or RNN-T models are both better alternatives.

There is also not really a problem about available open source code. There are countless of open source projects which already have that mostly ready to use, for all the common DL frameworks, like TF, PyTorch, Jax, MXNet, whatever. For anyone with a bit of ML experience, this should really not be too hard to setup.

But then to get good performance, on your own dataset, what you really need is experience. Probably taking some existing pipeline will get you some model, with an okish word-error-rate. But then you should tune it. In any case, even without tuning, probably encoder-decoder-attention models will perform better than CTC models.

View on HN

Author here! CTC models perform quite well and are easy to get started for beginners with an added benefit of real-time streaming capabilities. RNNT's and Encoder-Decoder like Listen-attend-spell are also very solid choices, and literature points to slightly higher accuracy with them on academic datasets. RNNT's being and extension of CTC are streamable as well.

That's right - most literature does show that encoder-decoder architectures outperform CTC. I think one of the main reasons for this is that CTC assumes the label outputs are conditionally independent of each other, which is a pretty big flaw in that loss function.

The blog does mention Listen-Attend-Spell (which is an encoder-decoder architecture) as an alternative to the CTC model.

Actually end-2-end CTC models are very good - Wav2letter, Jasper, QuartzNet - all these models are much better than DeepSpeech2.

Agreed, and that's where it seems having lots of experience working with speech data helps more than trying to brute-force it with just larger CTC models and "more" data of dubious quality.

I question that you need full attention in the acoustic model. The pronunciation of a word in the middle of phrase does not have much dependence on the beginning.

You do need attention in the language model part of the pipeline

Almost all the literature which compares CTC and encoder-decoder-attention models shows pretty well that encoder-decoder-attention performs better than CTC in the acoustic model.

See for example here as an overview (my own work, already a bit outdated, but attention has even improved much more since then): https://openreview.net/pdf?id=S1gp9v_jsm

It would seem that the best practical approach is to use RNNT as it still lets you do streaming predictions (while Attention won't really let that).

Author here! RNNT's is indeed a good approach. RNNT's with masked attention are also capable of streaming as well.

If you need streaming, then yes, RNNT is a good option. If not, encoder-decoder-attention performs a bit better than RNN-T.

Note that there are also approaches for encoder-decoder-attention to make that streaming capable, e.g. MoChA or hard attention, etc.

Google uses RNN-T on-device. But they are researching on extending it with another encoder-decoder-attention model on-top, to get better performance.

This is a quite active research area, and it has not really settled. But CTC is not really too much relevant anymore, as RNNT is just a better variant.

Recent work on transformer transducers with limited right (and left) context seem to give decent results as well: https://arxiv.org/abs/2002.02562

I opened the pdf, did ctrl+F for 'github', got zero results. Have you reproduced their "decent results"?

ESPnet apparently already has implementations of versions of transformer transducers with RNN-T loss (though, with different network architecture). At least the paper cites results on a freely available dataset, right?

Why would you not expect Google to post their code? E.g. the transformer paper had it: https://arxiv.org/abs/1706.03762

Reply on news.ycombinator.com