Hacker News Reader

Comment by stochastic_monk | Hacker News Reader

I haven’t done any significant projects using neural networks for sequence modeling or analysis, but when I do, I plan to start with a temporal convolutional network, based on [0]. They argue that RNNs being standard is likely an artefact of the history of the field, while they get superior performance from TCNs.

[0] https://github.com/locuslab/TCN

View on HN

The truth is that the best architecture depends on the problem, and you get a lot out of "graduate student gradient descent", i.e., hyperparameter search and fiddling with the architecture fine grained details. You'll want to experiment with rnn, tcn, and transformer (t2t) models to find the best one for your problem.

Disclaimer, I wrote the TF RNN api.

Totally agree - it's hard to say which sequence architecture works best for your problem (are there long term dependencies, do they exist at multiple levels of scale, etc) and dataset size.

Convolutions are cheap to compute and can be more efficient on smaller data but it's also possible that a CNN outperforms when you have a small dataset and an RNN when you have a larger dataset.

For graduate students out there that would rather be doing research than "graduate student gradient descent" (or, high dimensional, non-convex optimization in your head), SigOpt (YC W15) is a SaaS Optimization platform that is completely free for academic research [0]. Hundreds of researchers around the world use it for their projects.

Disclaimer: I co-founded SigOpt and wasted way too much of my PhD on "graduate student gradient descent"

[0]: https://sigopt.com/solution/for-academia

I do a lot of sequence work (text).

While CNNs are tempting and fast to train I've never been able to get the accuracy I can from RNNs. In NLP, accuracy is important because for lots of tasks NLP is right at that inflection point of being good enough to be useful.. if it's good enough.

It's worth noting that this TCN paper gets a perplexity of 45.19 on WikiText-103. That was competitive in 2015.

The current state of the art is 29.2[1] - not their claim of 48.4 (unclear what that came from).

Still, CNNs are nice in an ensemble model if that's your thing. They do tend to pick up different things to RNNs, which can be useful.

Edit: I now understand why their reported metrics are so wrong. They use generic models to compare against. They list SOTA performance in their supplementary material (they still get them wrong though).

[1] http://nlpprogress.com/english/language_modeling.html

Thanks for digging into how their claims are misleading. As someone in an adjacent subfield, I was wondering if I was missing something. It does at least sound like an interesting potential component.

From my cursory understanding of TCN, it always outputs sequence of the same length as the input. If that is true, then its usage is severely limited compared to recurrent networks.

Please point to more substantive sources if I am wrong. I am very interested to make TCN work, as it is much faster.

Reply on news.ycombinator.com