It’s just totally evidence free reasoning from axioms that are chosen by vibes alone.
Why evil AI god and not the Christian god? Why not Huitzilopochtli, who demands sacrifice?
The answer is that this is the wrong question. No argumentation can be usefully made either for or against.
With AI we’re seeing the capabilities improve rapidly and the arguments about why AGI is impossible or will be constrained for some reason are the weak ones.
Yes, but towards what? How do we know that, say, Transformer based LLMs are closer to superintelligence than earlier architectures?
To make such an assumption, there would need to be something that we could measure to track the process. To the best of my knowledge, there is no accurate definition of intelligence, nor superintelligence.
So how would we know where on the scale of [intelligent-------superintelligent] a given system is, or whether it even is on that scale?