Hacker News Reader

But highway network doesn't work as well as ResNet, and ResNet is simpler.

Highway connections are actually used heavily in NLP in combination with LSTMs.

And replaced by residual connections in transformers, which are absolutely dominating LSTMs now.

Transformer-XL uses recurrence, and most NLP SOTA is still with LSTMs. I’m not sure I’d expect attention mechanisms to fully replace recurrence.

Highway networks with scalar gating beats ResNet by a small margin.