Hacker News Reader

Comment by skybrian | Hacker News Reader

I think that article is misleading, because a simulator has rules. An LLM is better thought of as a storyteller, because at best it's going to follow whatever implicit rules there are very loosely and let you make rule changes of your own, more like Calvinball.

Also, whatever loose rules it has are more literary than mathematical. Plot twists often work.

View on HN

I find this to be a better explanation than "it's just regurgitating strings of text"

No, it is clearly not, and that is a very easily testable hypothesis.

Thank you for sharing.

testable how?

out of distribution tests. if the concept holds consistently over a long period, the concept is the stable thing. If not, then it's only memorizing string densities.

For a number of years we've been basically showing the first to be the case, especially as the model is scaled and the context increases, differentially against the second. String density probabilities can be surprisingly brittle, to be honest. The curse of dimensionality applies to them too, believe it or not, which I believe is why topic discussion, reasoning, and integration over longer distances of text is that differential test that shows pretty clearly that substring memorization/text density stuff is not 'just' what the model is learning. Because mathematically/statistically/from an information density perspective/etc etc otherwise it would be basically impossible, I think.

That's my best understanding, at least.

The essay is long and complicated so I'm not sure how much of it you read closely, but it specifically addresses this distinction between the simulator and the simulacrum.

In the analogy of the essay, your argument would be like saying that reality cannot be simply the application of quantum physics, because you are allowed to make new rules like Calvinball within reality which are different from the rules of quantum physics.

I do understand the difference between a simulator and what's being simulated. I still think they got it all wrong, that the simulator is better called a "writer," the simulated world is better called a "story," and the agent is a better called a "fictional character."

We know there's no deeper level to the simulation/game because we have the entire "game history" (the chat history) and we understand it in approximately same way that the LLM does. (That's what the LLM was trained to do, understand and respond to text the same way we do.) We know that the bot has no hidden state when it's not the bot's turn because of how the bot's API works.

So there's nowhere for a deeper simulation to live. It's as shallow as it looks.

https://skybrian.substack.com/p/ai-chats-are-turn-based-game...

I've always thought of them a bit like improv, since they tend to follow the "yes, and..." rule, by happily continuing whatever direction you want to go. Now that the base models have been fine-tuned to avoid some topics, that's less true than it used to be, but it still feels like the most natural mode of operation.

Reply on news.ycombinator.com