The time for an AI pause has passed. Asking people not to race to the finish line is reasonable when the finish line is very far away and people don't know where it is, but not when it's the obvious next step on their current path.
Zvi's an AI-doomer, so he sees the endgame as AI becoming smart enough that they don't need us anymore and then killing us to take our stuff, but your scenario is also pretty bad.
Yeah, a lot of the web is at-risk if searchers just read LLM summaries and don't click through, but we will have Skynet before that becomes a real issue and then this will all be irrelevant.
Interesting. I know the author thinks asking an LLM to make Atari games is cheating, but did he consider just randomly sampling from the assembly code of Atari games?
The basic results are interesting, but what really surprised me is that asking them to double-check didn't work. Falling for an "optical illusion" is one thing, but being unable to see the truth once you know the illusion there is much worse.