To predict the next word, you need some model of the universe.

The intuition I’ve been building around LLMs is that they’re like search systems.

They encode their training data in their weights and do a kind of search on it given a prompt.

If leetcode problems are in their training data, then they’ll of course answer the question with ease.

Every solvable problem is solved by using known information, patterns, context, etc. We (and LLMs) are using some model of the universe and trying to coordinate it in a way that will help us solve some task. The difference between us and "old" LLMs is that we can generate new information/patterns/etc., immediately add it to our model, and use it to solve more complex problems. New LLMs such as o1-o3 are also capable of thinking over and over again and producing new information (in the current context) and trying to apply it to the current task that might not be solvable with just the information that it was trained on.

(This is my understanding, I’m not ml engineer)