We started out with very simple probabilistic models by making very strong limiting assumptions. A naive Bayes model assumes every token is independent; a Markov model assumes that a token only depends on the single token before it. But ultimately these are just models for the underlying “real” joint distribution, and in theory we should expect that the simple models are bad because they’re bad estimates of that true distribution. I think my experience with these poor approximates limits what I expected that any language model could do, and the success of LLMs is making me recalibrate my expectations.
But I think it’s important to keep in mind that the model is still just spitting out the most likely next tokens according to its own estimation of the joint distribution. It’s so good because the model of “most likely next token” is very accurate. It’s easy to fall into the trap of looking at how specific the output is and thinking “wow, out of all the possible outputs, how did it know to give this one? It could have said anything and the right thing is a very low probability event, so it must really understand the question in order to answer properly.” But another way to think about it might be “wow, it’s incredible that the model accurately estimates the probability distribution for this kind of token sequence.”
As to your specific example of things like future time stamps, I think I’ve seen folks say in the past that there is some additional input prefix or suffix information in addition to the prompt you send that is passed into the model. So the model is still a static function—it is not learning from experience within an interaction session; it doesn’t experience time or other external contextual factors as part of its execution. The operators update the contextual input to the model as time passes in the world.