You're right, seahorse emoji is almost certainly in the training data, so we should amend my explanation to say that "seahorse emoji" is not just close to the training manifold, but almost certainly right smack on it. The rest of what I said would still apply, and my explanation would also to apply to where other commenters note that this behaviour is emitted to some degree with similar other "plausible" but non-existent emoji (but which are less likely to be in the training data, a priori). EDIT FOR THIS PARAGRAPH ONLY: Technically, on reflection, since all fitting methods employ regularization methods, it is still in fact unlikely the fitted manifold passes exactly through all / most training data points, and saying that "seahorse emoji" is "very close" to the training manifold is still actually technically probably most accurate here.

You're also right that it is a long discussion to say to what extent LLMs are statistical or probabilistic, but, I would maybe briefly say that if one looks into issues like calibration, conformal prediction, and Bayesian neural nets, it is clear most LLMs that people are talking about today are not really statistical in any serious sense (softmax values are scores, not probabilities, and nothing about pre-training or tuning typically involves calibration—or even estimation—in LLMs).

Yes, you can use statistics to (help) explain the behaviour of deep models or certain layers (usually making assumptions that are of dubious relevance to actual practice), but geometric analogies, regularization methods, and matrix conditioning intuitions are what have clearly guided almost all major deep learning advances, with statistical language and theory largely being post-hoc, hand-wavey, and (IMO) for the purpose of publication / marketing. I really think we could de-mystify a huge amount of deep learning if we were just honest it was mostly fancy curve fitting with some intuitive tricks for smoothing and regularization that clearly worked long before any rigorous statistical justification (or which still clearly work in complicated ways, despite such an absence of statistical understanding; e.g. dropout, norm layers, the attention layer itself, and etc).

Just, it gets complicated when you get into diffusion models and certain other specific models that are in fact more explicitly driven by e.g. stochastic differential equations and the like.