In terms of terminology, is it accurate to interpret the latent variables as the “world model” of the neural network?
Embeddings are a form of latent variables.
Attention query/key/value vectors are latent variables.
More generally, a latent variable is any internal, not-directly-observed representation that compresses or restructures information from inputs into a form useful for producing outputs.
They usually capture some underlying behavior in either lower dimensional or otherwise compressed space.