That's true for today. If what happened with chess happens with LLMs, they're going to blow past us in reasoning ability in a few short years. And it might not be LLMs, it might be a new architecture, maybe one that has an LLM as a component. What matters is that people are trying to create AGI, and they've been getting closer and closer. All our models for how things will go will break down at that point, and I think it's foolish to assume that it'll turn out well without any evidence. (you can argue that it's foolish to assume it'll turn out badly too! But it's not foolish to assume that it might and therefore we should be very careful)

Also -- life isn't fair, it's possible that there's no way to create beneficial AGI in real life, and we should not rule that out as a possibility. Maybe it'll cause problems now, and more problems (or extinction) later.