Next token predictor is as useful as saying that humans are just atoms and neurones.
It is technically true but meaningless.
It is technically true but meaningless.
Anytime you hear someone saying "it can't do this, or that..." they are factually incorrect.
The denial around these issues is beyond creepy: AI has clearly hit some tender nerve in the human mind.
If we allow a file with unbounded storage (like your exception for physical computers) then an agent can easily perform the necessary state transitions and tape reads/writes.