Transformers Are Multi-State RNNs

arxiv.org

41 points·DreamGen··9 comments

They show that a decoder only transformer (which gpts are) are rnns with infinite hidden state size. Infinite hidden state size is a pretty strong thing! Sounds interesting to me.

not infinite, just scaling linearly with length

I've seen at least 6 such papers, all being like "<popular architecture> are actually <a bit older concept>". Neural networks are generic enough that you can make them equivalent to almost everything.

Linking the latest generation tech to the previous generations is actually really helpful from my perspective.

All of the terminology for this tech is still emerging, and it can be quite difficult to formulate a reliable mental model for any of it due to how quickly it’s changing.

If hammers were more difficult to understand, I could imagine someone writing about the fact that hammers are in fact, just a piece of steel mounted on a handle made of wood.

> Neural networks are generic enough that you can make them equivalent to almost everything.

Which to me is why papers like this are useful. They help newcomers conceptualize what the latest <popular architecture> is actually made of in terms of <a bit older concept> that the reader may already understand.

It will take some time for this information space to stabilize.

It's also helpful in that you can now take insights from <older concept> and apply it to <popular architecture>. Many things are easier to reason about when framed a little differently.

See: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr...

oblique

Has anybody proved that transformers are just kernel SVM yet?

I hope you're satisfied with Gaussian Processes: https://arxiv.org/abs/1806.07572

Reply on news.ycombinator.com