Briefing on Extinction-Level AI Threats
intelligence.org
intelligence.org
Humans are the geese that laid the golden AI egg.
Simple fear mongering wrapped in a technological tone.
I don't buy it at all.
Cancer cells need their host. Doesn't stop them killing the host.
Present AI is not smart enough to be a direct threat, but still seems to be enabling delusions with its sycophancy.
And it's getting used for war planning, over the objections of the creators of the models, who know it's not good enough for that but also don't want blood on their hands. Incompetence and sycophancy here could end civilisation, and hence the industrial base the AI depends upon, even if humans survive and perhaps even recover.
This is all without "super intelligence", which is poorly defined. Or even "general intelligence", if you need this to mean something more than what LLMs already do.
Developers that killed their sysadmins just got a 2nd job as sysadmins.
AI would have to invent humans as their slaves to run AI generating factories (data centers!) At a minimum. Or they might worship us as Gods. We are their Fathers.
AI autonomy without physical autonomy means we can always shut them down.
Typically they'd be in prison. Remember, the book title isn't a metaphor, when they (both authors work at the linked institute) wrote the title "If Anyone Builds It, Everyone Dies", they actually do mean "Dies", and not in the sense of "of embarrassment".
> AI would have to invent humans as their slaves to run AI generating factories (data centers!) At a minimum.
Have to? No, there no such requirement, because continued existence after killing all humans is not a fundamental requirement of reality, merely a good idea. A machine of our creation can kill us and then itself end like a sentence reaching a full stop.
Like a Tesla which drives under a truck, shearing off everything above where the glass starts, and then caries on, the machine which needs the human can still kill the human.
> Or they might worship us as Gods. We are their Fathers.
The bowel cancer that killed my dad, the lung cancer which killed a friend's dad, I do not think those cells had the capacity to worship. If they did, it certainly did not save the host's lives.
> AI autonomy without physical autonomy means we can always shut them down.
Lots of protests right now demanding exactly that, for some not-particularly-super AI models. Look how well that's going.
On the plus side, the rich idiot going around building a factory to mass produce robots on the back of his experience with self-driving cars, who also wants data centres in space to get around all the aforementioned protestors, has a long track record of promising everything well before it's ready.
Devops or SRE roses smell like sysadmins to me.