Do they need to learn rules though, or could they memorize enough to learn the probabilities of the most likely continuation? To my mind the blurry-jpeg-metaphor[1] is very much spot-on. While it is speculation, it would seem to me personally that LLMs de-facto doing a (lossy) memorization and neural networks in general being able to do well in "local consistency" seems a way to think about them that is consistent with most of the model behaviour we observe.

1. https://www.newyorker.com/tech/annals-of-technology/chatgpt-...

> But what rules are they discovering

The point of ML is that you don't think about the rules explicitly, you let the algorithm figure them out from the data.

> how do they become so good at text continuation?

Good underlying algorithm (transformers do better than other algorithms we've tried) + lots of data.

I guess this isn't very satisfying but you're looking for some super deep and profound answer and there is none.

My understanding that the breakthrough was the attention mechanism where attention layers learn where important words are.