There is no hidden state in a recurrent nets sense. Each new token just has all the previous tokens and that’s it.