And they do have internal latent vectors for their own states, so there's some kind of recursion/introspectiveness dynamic.
Improvements in model performance have been made exactly by having intermediate steps stay in the form of internal representations rather than words.