Bryan Cantrill on AI doomerism: Intelligence is not enough | Hacker News Reader

Bryan Cantrill on AI doomerism: Intelligence is not enough

thenewstack.io

13 pointshhs18 comments

„How AI Will Wipe Put Humankind” [sic] seems like a question that’s hard not to answer. It takes an especially closed mind to find the scenario unimaginable. I’m not very invested in the debate - I hardly give a fuck - but I find AGI apologists endlessly fascinating, because I have a hard time coming up with a scenario where AGI does not result in our complete annihilation. It’s like the divide is so immense, there’s effectively no way to communicate across.

Ya this article is garbage. "Hey let's ignore all the massive increases in biological capabilities over the past 10 years and just call it hypothetical"

Earnest question: for those on the cutting edge of these breakthroughs (e.g., biochemists working on CRISPR), have any expressed fear of AGI-based extinction risk?

The premise of annihilation presumes quite a lot.

That the issues around continued training on only synthetic data without continued refresh for edge cases with organic data leading to model collapse will be solved, for example. Without that, AGI killing off humanity is a murder-suicide.

It assumes that AGI picks up the side effects of our own 'training' in a desire for survival above nearly all else or our sadism and psychopathy while simultaneously assuming that it doesn't pick up things it has already appeared to develop early forms of around empathy, symbiosis, and cooperation.

It assumes that it would be seen as more expedient and feasible by AGI to develop automation across every aspect of its own operation and maintenance than have continued human involvement - whereas I'm extremely skeptical we even have the capacity for replacing most humans along the pipeline from mining semiconductor materials to running them in a server farm with robots. That's pretty damn Sci-Fi gibberish until far later than AGI is likely to exist.

It assumes that superintelligence is somehow still stupid enough not to know the past 50 years of game theory such that it concludes competition isn't cutting off its nose to spite its face as opposed to cooperation. In humans, most of this behavior seems driven by legacy biological programming and not rational decision making.

It assumes that a being for which the virtual will always be capable of greater fidelity than the real will be overly concerned with conquest of the real.

It assumes that AGI will fail to recognize the diminishing returns on wealth and power that so many humans fail to recognize, despite simultaneously being 'superintelligent.'

To me, AI doomerism really seems to reflect the issues with human thinking rather than synthetic thinking, particularly around our tendency to try to see issues as a binary. "Either AI or humans can be successful."

We somehow have greater capacity to imagine symbiotic superintelligence when attributed to an alien organic civilization than to intelligence directly dependent upon the foundation of our own. (Which is rather depressing if you really think about it.)

It also assumes that an exponentially self-improving AGI is possible. An AGI that is stuck at human level is probably not a threat against 8 billion humans.

So, yeah. There are quite a few assumptions in there that are a long way from experimentally prove.

I don't know - I agree with the xkcd that the period of AI development I find most scary is not when it is smarter than humans, but when it isn't yet smarter but improving all kinds of human tasks like warfare or aiding corporate oil maximizers, etc.

I know those endeavors are stupid long term, but apparently enough of humanity doesn't such that AI only as smart as humanity at large will be aiding and abetting quite a lot of horrors that I have a harder time imagining something smarter than humanity aiding unconditionally.

Dumb AI is much scarier to me than smart AI.

That premise alone should make us question everything about that ridiculous movement. Today’s LLMs have access to almost all of the accessible data on the web we have generated to date. AGI is going to have the SAME data to work with - adding whatever amount we generate between now and then.

There isn’t a hidden data cache that is not being dug up and incorporated into today’s latest LLMs - and none of them have embarked on a quest of vengeance. The premise of AGI getting uncontrollably smart by absorbing internet data is wholly flawed. We’ve already absorbed almost all of it and nothing happened.

Once again I’m listening to find good refutations, and am severely disappointed. I truly can’t imagine how someone finds anything in here a convincing argument. “It’s like my daughter says, it’s got no arms or legs.” A whole lot of snark and wishful thinking, zero debate with the intellectual arguments. But ya, just go call Ilya, Bengio, Hinton, Altman, and Hofstadter religious instead.

It feels like you are deliberately taking the most facile argument and ignoring the most substantive? As I outlined in the talk: for the breakthroughs that were required to do something that is thought of as pedestrian (namely, bringing up a computer), intelligence is simply not enough -- a superintelligence would not have suggested the breakthroughs because they required intense iteration, not simply the absorption of book knowledge. And yes, it's a technical argument, not an "intellectual argument" -- and as such requires much more technical depth to appreciate -- but that is very much the point, frankly...

> they required intense iteration, not simply the absorption of book knowledge

I watched your talk, and I think this can be elaborated on even more to really drive it home. Because the big question is: how is your AI going to have agency, the ability to affect the world? How do we get from text on a screen, to Destroying All Humans? AI doomers seem to have this Ancient-Greek-philosophy view of science, that you can make progress just by sitting atop a marble column and pondering, but that's not how it works. You need to be able to poke and prod physical things, to experiment, and even if an AI was smart enough (and the high-level intellect required to do novel engineering is a pretty big handwave already) it has no means of poking and prodding.

(You point this out in your talk by saying "here's all the hands-on trouble we had to go to just to boot a computer", but I think some people are missing the broader implications.)

Given that the AI won't have an existing written corpus of "how to build killer robots" to draw on, or a physical corpus to use to develop novel technology (and troubleshoot it) it's pretty much up to the humans to develop the killer robots or whatever (and construct the killer robot assembly line), and then the blame is really on the humans for making autonomous murder machines and hooking them up to a psychopathic AI. (And even then, this is not so different from building a conventional army and then putting a psychopathic human in charge.)

As you mention in your talk, a lot of other scenarios are just people wanting life to imitate Hollywood. I mean, you could hook up an AI to the nuclear Big Red Button, but that would just be dumb, and you wouldn't need an AI, just a plastic bird to press the "N" key whenever the system asked "start nuclear war?"

Thank you for this. It was exactly my point in the talk, of course -- but in reading the comments here, I was very much beginning to doubt my own sanity...

You could call me "AI serious complicationist" ("AI doomist" would probably be too much), and I still am as I'm writing this comment.

But your point about the required amount of iteration and all the complications along the way are convincing for me-- I've realized that there might be more steps along the way to "serious complications" than I thought.

So thank you for that!

Have you tried to do some sort of sensitivity analysis on your world model? Something like the α-Maxmin rule for a Monte Carlo approximation?

The way I see it is that we don't need to imagine any sci-fi scenario to see how any device with a computer (so basically anything from a phone to killer robots or power plants) could go rogue or severely malfunction.

Stuxnet was developed by skilled humans and was pretty successful. Personally I cannot see why a superintelligence wouldn't be able to put together Stuxnet^10.

Depending on the use case, the development could be 100% digital--the AI could either work it out in its mind, or run/rent a VM. And if there's a need for physical hardware that the superintelligence cannot emulate because of undocumented bugs and issues, it could hire someone with a garage to put the hw there and plug it--it doesn't require a factory and 10s or 100s of people.

If you don't think it's a plausible scenario because I'm missing some step(s), then let me know!

> I cannot see why a superintelligence wouldn't be able to put together Stuxnet^10

How will this Stuxnet^10 be able to do orders of magnitude more damage, though? Shut down power plants? Launch ze nukes?

The "shut down power plants" issue already exists and people have been aware since the Y2K scare, and now due to cyberwarfare concerns. Such problems are solved by not being dumb, and making sure there are enough air-gaps and humans-in-the-loop that an errant computer command can't wreak havoc (or fire ze missiles).

I guess your Stuxnet^10 could do damage by immobilizing the entire Tesla fleet, but once again, that's a dumb design issue with networked cars, not a vulnerability you need a super-AI to exploit. And I personally don't (and would never) drive a remotely-updatable car myself for exactly this reason.

These are all problems that are easily solved by not doing anything monumentally stupid engineering-wise. The lesson we need to take to heart isn't Terminator, it's Chernobyl.

According to the Wikipedia article [1], Stuxnet not only infected the Iranian infrastructure but apparently also a Russian nuclear plant. And so air gaps & co. didn't help against a skilled, determined but still human-level attacker.

But then I read the Wikipedia article more closely, and, for instance, it turns out that Stuxnet components were signed with stolen certificates so that Windows recognized them as legitimate. Which means that at best someone had not only write Stuxnet, but also get those certs. And even if it's easy in comparison, there had to be at least a remote hack involved, if not a physical one. And then there was the risk of those certs being revoked. Etc., etc.

So, even if I'm more pessimistic than you, the lesson for me here is that if you start peeling the onion layers, it's all not so simple to pull off.

[1] https://en.wikipedia.org/wiki/Stuxnet

You might have just planted an idea for a future super AI who to cont(r)act when iteration and manufacturing of that innocent Ring [1] is due :).

I cannot decide how much I'm joking here (not about planting the idea, which obviously is a joke, but about the outsourcing part), it's an intriguing topic.

[1] Totally not to rule them all

Cantrill today on Twitter: "It's occurring to me that AI doomerism is overwhelmingly white, male, Millennial, and childless -- and that this tells us more about them than about them than it does about the future of humanity"

Identity politics is a brain cancer that kills the host. Imagine if someone Tweeted "It's occurring to me that doomerism is overwhelmingly white, female, Millennial, and childless -- and that this tells us more about them than about them than it does about the future of humanity."

"Help! Misogyny!" [0]

[0] https://external-content.duckduckgo.com/iu/?u=http%3A%2F%2Fs...

Reply on news.ycombinator.com