Hacker News Reader

I’m confused over your definition of model architecture.

Loops aren’t recursion?

Loops and recursion are fundamentally equivalent.

See e.g. https://textbooks.cs.ksu.edu/cc210/16-recursion/08-recursion...

Only if you have indexable memory that you can use as a stack, which in the context of LMs isn’t a given.

As another example, a finite-state-machine language can have loops, but it can’t recurse unless there is external memory it has access to in a way that it can serve as a stack. Regular expressions also fall into that pattern; they can loop, but they can’t recurse. For that you need a pushdown automaton: https://en.wikipedia.org/wiki/Pushdown_automaton.

Everything old is new again when you are in academia

This feels primarily like an issue with machine learning, at least among mathematical subdisciplines. As new people continue to be drawn into the field, they rarely bother to read what has come even a few years prior (nevermind a few decades prior).