Any system is set up for and tuned around absorbing some amount of input at a time -- nutrients, say. If it gets too little, it starves and dies. If it gets too much, it breaks down and dies. A system can adapt over time, at a limited rate, and become capable of handling more or less.
Drinking a glass of water is beneficial and necessary. Drinking 100L in one go and you're dead.
The argument is that the rate of AI influx is too high for our human systems to metabolize, and so the system will break down. Not just operate in a degraded way, but break down and cause mass catastrophe. Even there, it doesn't necessarily mean the end of humanity, just 99.9% of us.
What the argument is not: AI is fundamentally poisonous. We are incapable of handling any level of AI. We will never be able to incorporate more AI. AI provides no benefit.
So quitting AI but still working on AI is not the contradiction it seems: you can quit accelerating AI and work on metabolizing it. And in fact that work is very necessary right now.