Hacker News Reader

Ironically Schmidhuber's lab also came up with a more general version of bypassing http://arxiv.org/abs/1505.00387 to train deep networks (with similar performance), of which the popular ResNet is a special case.

View on HN

But highway network doesn't work as well as ResNet, and ResNet is simpler.

Highway connections are actually used heavily in NLP in combination with LSTMs.

And replaced by residual connections in transformers, which are absolutely dominating LSTMs now.

Transformer-XL uses recurrence, and most NLP SOTA is still with LSTMs. I’m not sure I’d expect attention mechanisms to fully replace recurrence.

Highway networks with scalar gating beats ResNet by a small margin.

Reply on news.ycombinator.com