Hacker News Reader

This deepmind paper showed big limitations on nested structures with transformers, LSTMs actually did better:

Neural Networks and the Chomsky Hierarchy

https://arxiv.org/abs/2207.02098

Interesting. Can someone quickly summarize their results? Are they saying that using languages with simpler syntax Lisp lead to LLM being able to generalize their knowledge better?

They tested them on formal languages of different power, and saw where they could generalize beyond the training data. Transformers failed to generalize pretty early on at stack machines/brace matching.

It was a good bit older of a paper though, if I remember it's somewhat expected from the pure feed forward nature of them and limited circuit depth, where LSTMs have some recurrence.

Reply on news.ycombinator.com