Interestingly, modern ML does not require Turing completeness. And yet, people are considering the possibility of AGI - I would find it pretty amusing if Turing completeness isn't necessary.
Seems to me that Turing completeness is necessary, for the simple reason that I can mentally trace through Turing-complete code.
Without a piece of paper to take notes on? ;)
My own memory functions as the paper tape as long as the program is simple enough :)
Token inference by itself isn't Turing complete, but if its output can have side effects (e.g. editing the prompt for the next iteration), that's a whole different story.