It turns out that a lot of intellectual tasks are sort of weakly simulatable by just doing a great job at next-token prediction.

Yes, this is a good way of putting it. I've been saying for years, it's less that we're making big discoveries about what "AI" can do, and more that we're showing that many things humans do that appear complex actually reduce to something pretty simple. But that simple thing is still just fitting a pattern. It's the cases where it doesn't work, even if it only fails 1% of the time, that define the difference between pattern matching and actual human intelligence.

One thing that follows from that (that people don't like) is that we actually need to move goalposts about how intelligence is defined. "Pass a turing test" is not very valuable now. And as mode tasks are shown to be possible with pattern matching / next token prediction, we need to further refine tests away from these tasks to settle on a good definition of what separates human intelligence. It should be obvious that the distinction is there, but it's still tough to nail down. (I'd argue that by defining a "task" you've already done most of the work to solving it, so it's not to exciting to learn that AI can finish the job)