I think the true fear is that in an AI age, humans are not "useful" and the market and economy will look very different. With AI growing our food, clothing us, building us houses, and entertaining us, humans don't really have anything to do all day.
"Hackers aren't a problem because we have cybersecurity engineers". And yet somehow entire enterprises and governments are occasionally taken down.
What prevents issues in redteam/blueteam is having teams invested in the survivability of the people their organization is working for. That breaks down a bit when all it takes is one biomedical researcher whose wife just left him to have an AI help him craft a society ending infectious agent. Force multipliers are somewhat tempered when put in the hands of governments but not so much with individuals. Which is why people in some countries are allowed to have firearms and in some countries are not, but in no countries are individuals allowed to legally possess or manufacture WMDs. Because if everyone can have equal and easy access to WMDs, advanced civilization ends.
Can you imagine how much harder it would be to protect against hackers if the only cybersecurity engineers were employed by the government.
With superintelligent AI you likely have to have every developed use and every end user get it right, air tight, every time, forever.
What on earth could this possibly mean in practice? Two elephants fighting is not at all good for the mice below.
Do we have reason to believe that giving the ingredients of AGI out to the general public accelerates safety research faster than capabilities research?