> This kind of argument is always dangerous, because it essentially resorts to moving goalposts. "Oh, AI can now do X? Sure it is amazing, but it can't do Y yet, so we're totally safe!"

It's not about moving goalposts. What you're not understanding is that, even if for you is crystal clear that AI will be 100x smarter tomorrow, those in charge cannot simply bet all on that. Right now AI cannot really replace the core sauce of mathematicians (all the "understanting" and "asking the real questions" stuff), so it'd be unwise for, say, countries to start making decisions as if AI is capable of that.