Why did he not used Time Convolutional Networks?
[1] https://openreview.net/pdf?id=rk8wKk-R-
[1] shows that TCN may outperform LSTM on tasks that really needs memory.
[1] https://openreview.net/pdf?id=rk8wKk-R-
[1] shows that TCN may outperform LSTM on tasks that really needs memory.
A paper with inductive results is just a first step in a really long process until ideas are commonplace accepted.
He used Transformer, why not TCN?
In this paper, Bellard makes a comparison to the Transformer, some variant of which holds most NLP records. Of note is that in this test, the Transformer did not outperform the LSTM. Is it that Transformers need to be larger to outperform or LSTMs are more suited to compression or that more epochs are required? This deserves further investigation, what are the required conditions for Transformers to underperform LSTMs?