> I’ll grant that I don’t think of what’s going on in my head as “predicting the next token in a sequence”

I can't speak to whether LLMs can think, but current evidence indicates humans can perform complex reasoning without the use of language:

> Brain studies show that language is not essential for the cognitive processes that underlie thought.

> For the question of how language relates to systems of thought, the most informative cases are cases of really severe impairments, so-called global aphasia, where individuals basically lose completely their ability to understand and produce language as a result of massive damage to the left hemisphere of the brain. ...

> You can ask them to solve some math problems or to perform a social reasoning test, and all of the instructions, of course, have to be nonverbal because they can’t understand linguistic information anymore. ...

> There are now dozens of studies that we’ve done looking at all sorts of nonlinguistic inputs and tasks, including many thinking tasks. We find time and again that the language regions are basically silent when people engage in these thinking activities.

https://www.scientificamerican.com/article/you-dont-need-wor...