No, this is nothing like atomic bombs. In what scenario can you kill 100,000 people with with an LLM hooked up to a Python interpreter? The AI safety grift is out of control.
Have it design a nuclear reactor, saves money, you don't notice the flaw until it's too late. Chernobyl wasn't malicious, it was in part a cost-saving exercise; how many died from the fossil fuels that were burned because of the fear of nuclear power after the incident?
How many died needlessly in Vietnam, before human journalists saw and reported on the human cost, and what happens when reporters are replaced with LLMs — to save money — that physically cannot see things for themselves?
That said, 100k is in the awkward range that political decisions regularly lead to — at most a single month's life expectancy in a nation of 100 million — so if I'd been you I would have asked about 1M+. At a mere hypothetical 100k dead, it may easily still be part of a larger scenario which saves lives overall, much like how the actual atomic bombs were an alternative to a protracted conventional war (and given the attempted coup to prevent surrender after the second, I consider the hypothetical used to justify the bomb to at least be plausible, though I am no historian and don't want to give the impression of certainty).
But even then I would say that the point is still important: disregard singularity, foom, and paperclipping, and AI is still often described as being akin to a faster rerun of the industrial revolution. Just having all the turmoil and unexpected consequences — health, social, environmental, economic, military, political — of the industrial revolution squeezed down from "economy doubling every 30 years" (which is what we saw) to "economy doubling every 18 months" (Moore's Law), is easily going to cause catastrophic consequences all by itself, even with no malice.
"Health and safety" rules had to be created because too many bosses thought it was just common sense to not put your hands into dangerous equipment while in motion, while paying people to do jobs that could only be accomplished by doing exactly that; how can we do more than guess what the equivalent to that is for AI before we see it?
What is to AI, that which global warming is to coal power?
What's the necessary consequence of everyone using AI, which is analogous to the necessity of building a sewage system in response to widespread installation of flushing toilets in cities?
We organised ourselves to build the latter, but we still haven't globally dealt with the former.
Also, you’re wrong about flushing toilets necessitating sewers. They were still manually emptying toilets in London when they built its sewage system. So sewage systems became popular before flushing toilets were widely adopted.
China building new coal plants (while also building new solar, wind, and nuclear) suggests that is not the only reason.
And it's not like any of the complaints about PV elsewhere are safety related.
> So sewage systems became popular before flushing toilets were widely adopted
What existed was not, however, sufficient. That is what necessitated building a massive new system:
"""The Great Stink was an event in Central London during July and August 1858 in which the hot weather exacerbated the smell of untreated human waste and industrial effluent that was present on the banks of the River Thames. The problem had been mounting for some years, with an ageing and inadequate sewer system that emptied directly into the Thames."""
"""During the early 19th century improvements had been undertaken in the supply of water to Londoners, and by 1858 many of the city's medieval wooden water pipes were being replaced with iron ones. This, combined with the introduction of flushing toilets and the rising of the city's population from just under one million to three million,[b] led to more water being flushed into the sewers, along with the associated effluent."""
"""The Building Act 1844 had ensured that all new buildings had to be connected to a sewer, not a cesspool, and the commission set about connecting cesspools to sewers, or removing them altogether."""
Just because a thing existed, doesn't mean they had previously done it at scale: the Ancient Greeks had electricity, railways, and steam; battery cars predate gasoline; we've today got the tech for a fully global power grid; none of this is at the scale relevant to the problems, and the same can be assumed for at least one possible thing that AI might force us to consider.
My guess would be spam and filters. Already exists, but likely to become so much more severe the old solutions will no longer hold.
Not that it really matters to my core point: what's the AI equivalent.
Spam and filters are a great example because spam filters are already AI. Literally neural networks trained to recognize spam.
LLMs are great at classification too so once they are small and cheap enough they could certainly be used as spam filters.
The project seems really quite loose and the author seems to be lacking in awareness with regards to the consequences of such a project. I do think others are right in saying, we really should understand more bout where people can and will try to take these ideas before we keep putting models on the internet for consumption by just anyone.
That's the worst part of it.
Is the current generation of LLMs dangerous? Probably not. Will the next generation of LLMs be dangerous? Maybe not, if we're lucky. The one after that? Nobody knows.
But projects like this don't so much rely on there being no danger, but rather, the question seems to not even have been considered. And that's crazy, because the assumption that things are harmless is getting more and more shaky the more capable these models become.
Good my ADHD makes me write HN comments on LLM threads instead of working on the latter, because I'm not sure I could stop myself otherwise. But this also means I can very much imagine only thinking about the cool parts, and not considering the dangerous ones - because that's how I feel about 90% of the time when thinking about current SOTA models. It's just too damn exciting.
(Hell, it's the first thing in tech in the last decade, that I find exciting.)
--
[0] - Whatever one may think of them, I find it sadly ironic how Eliezer and his disciples were considered pseudo-scientific doomsday AI cultists a decade ago, back when the whole thing was only a far-future concern, how they're still considered pseudo-scientific doomsday AI cultists today, after AI field made a series of rapid jumps, landing us exactly in the scope of things they were harping about all that time - and how I bet they'll still be considered pseudo-scientific doomsday AI cultists after someone unleashes an unaligned AGI and ends us.
I know it's technically crazy sounding, but for all we know, Earth is the only place with biological life, I think it's better to flip the coin somewhere else if that's what you want to do. Don't screw up here.
Yes, maybe the AI will come for Earth, but I feel like there's less chance of shit going really bad with some distance.
The doomsday AI cultists didn't come up with any of these ideas. The Matrix was a hugely popular movie in 1999.
chatGPT doesn't make the person who has believed since 1999 that AI is going to turn us all into Duracell batteries more correct. It is the same basic thought with with paper clips replacing batteries.
ah yes, the only people consuming such things should be The Big Corporations and Big Governments?
---
Snark aside (Sorry, I'm just a bit tired of people having a kneejerk reaction to this without even thinking),
we're talking about autonomous agents. Huginn and the like have existed for years. If they can be even somewhat better automated with NLP and the sort, it'd be excellent instead of having to actually manage and configure/code individual agents. and Guess what, these agents have ALREADY been connected to the internet. For years.
Of course, you will kill modern civilization by doing so (and many millions of people as a result), but at least the AI is back in the box, so there is nothing to worry about.
> all the computers
The average shit tier PC doesn't even have enough RAM to load an LLM capable of consistent output, much less to load it into VRAM for any kind of reasonable speed. This is like being concerned that Panda bears will become an invasive species and take over the world if we stop them from going extinct.
Besides, all you have to do to pull the plug is disable any of the billion python dependencies it needs to download every time and it's stopped dead in its tracks. Something that's basically done every other day and needs to be continuously fixed by project maintainers, so it's more like a Panda on life support that dies if somebody stops pumping air into its breathing hose. It's all fragile af.
Can you always tell if it's doing bad?
Even if you can tell when it's doing bad, if there's an app which makes you $10k/hour while running, what risk of it killing somebody if left unsupervised will be low enough for you to leave it running unsupervised? Say, while you sleep?
When there's a lot of money to be made, do you trust everyone else to (1) be as risk-adverse as you, and (2) not just convince themselves there's no risk using exactly the argument you just gave?
Let's say you personally are willing to run it unsupervised if the risk is mean 1 fatality per 8*365*80 hours when unsupervised; are you willing for every human to have such a system running unsupervised? A tenth that risk?
See https://youtu.be/ld-AKg9-xpM?t=30 for a counterpoint.
At any rate, if the AI is smart then there are alternatives to begging.
It would be poorly run because pure money is not a strong enough incentive to hire someone to commit a crime for you, because that money can easily disappear once it is seized by the local government. Usually criminal organizations operate on a degree of trust, trust that would be hard to establish from some LLM that can, at best, fake a face on a zoom call.
Satoshi's coins are easy to play around with because they are at least legal to hold and sell (for now). If they weren't, few would bother with them.
When that Python interpreter is running on the computer doing model-predictive control of your country's natural gas pipelines, or the plant that mixes your mayonnaise.
Just so you know, the Python interpreter is already there in those places, and so is Internet access, because suits like their dashboards with real-time graphs.
Remember, the S in IIoT (Industrial IoT) stands for both sanity and security :).
Yes to both, sadly.
> Because there’s no way that someone will use one to control the automated processes in a factory.
What is that saying Americans are fond of? Ah, yes - if you believe that, then I have a bridge to sell you.
> It’s not like someone showed up to the mayonnaise factory and says “I think I’ll increase the ratio of paprika to salt today, just for fun”
No, they show up at the industry trade fair in some German city, find a group of people looking like they own factories, and start waxing poetic about how their company is using the newest advances in AI to synergize value delivery and carbon supply chain footprint management, ushering in the era of ecologically responsible and exponentially profitable Industry 4.1. You know, like ChatGPT but you plug it into your plant, feed it your BOM, and it prints out money. This kind of stuff.
The article talks about LLMs being hooked up to the Internet. Big difference.
If you really can't imagine a scenario where this might lead to people dying, you're not trying hard enough.
1. The LLM needs to find an exploitable bug in a popular code base.
2. The LLM needs to write a reliable exploit for that bug.
3. The LLM needs to develop a worm that exploits that bug and spreads itself, opening access to the system.
4. The LLM needs to connect to systems and understand if they are of any significance (it found a mayonnaise plant!).
5. The LLM needs to understand the control protocols of their industrial control systems.
6. The LLM needs to understand how to make a dangerous composition from the ingredients it has on hand (let's pretend it can dump some industrial cleaning solution that is on standby for cleaning the tanks).
7. The LLM needs to assume such total control over this processing plant that it can disguise the traffic and not trigger a single alarm around malfunctions.
What you're vaguely hinting at is extremely high skilled labor. There are a few billion dollar businesses in those steps. I welcome you to go read up on the challenges in automated exploit generation. LLMs are nowhere close.
Now, you might rebut and say there are far simpler attacks, like phishing! Also an extremely hard problem. Try to send email in mass and not land in a spam filter, try to do the reconnaissance necessary to generate a believable login page. Try to leverage the sale guy's credentials to reach any system more meaningful than the company's Salesforce instance.
So once again, I ask, please walk me through a situation where an LLM gets anywhere close to killing even 1% of the number of people an atomic bomb could.
We are nowhere close to even a rudimentary understanding of how current LLMs actually work. All we have are low-level building blocks, and lots of philosophizing about high-level output.
Considering that, relying on some gut feeling about what LLMs "surely cannot possibly do" is reckless overconfidence. Not to mention that the next generation of LLMs, with potentially entirely new emergent properties, might be just around the corner, and the time to put safeguards into place is now, not when it's too late.
As for the scenario, all an LLM with Internet access would need to do is find a single remotely exploitable vulnerability in the Linux network stack. That would allow it to literally shut down the entire Internet, which would kill a lot more people than a single atomic bomb.
I wouldn't be surprised if this led to the death of 10% of the global population in about 5 years. That's 800 million people, or around 4000 times the combined death toll of the Hiroshima and Nagasaki bombings.
The Internet is the backbone of the modern world. Short of a global nuclear war, it's hard to imagine a more catastrophic event than it suddenly becoming unavailable.
The loss of many of these records in the financial and governance would certainly lead to a lot of dollar losses. But dollars aren't people.
As for economic consequences, millions of jobs would just no longer make sense, and many others would get harder and frequently much less efficient. Certain major categories of product no longer make as much sense; what's an iPhone that can't connect to the internet worth? It's just a telephone with a camera (that takes images you can't share).
It could be deadly to some degree? All it takes is for the power to go out somewhere exceedingly hot in July, as the loss of air conditioning can be deadly for the elderly. But as deadly as the nukes the US dropped on Japan? I don't see it.
Nope. It's just a camera. Because the "telephone" part relies on infrastructure that is unmaintainable without the Internet, and would probably stop functioning in a matter of days.
And boom, you're back to the 19th century, where telephones weren't a thing. Except that unlike in the 19th century, there isn't any infrastructure to make things work in that situation.
How does the hospital order medical supplies now? By paper mail? Sorry, mail can't be delivered anymore, since all postal services depend on logistic systems that in turn depend on the Internet for coordination. Also, printed catalogs haven't existed for almost 2 decades, so the hospital won't even know what supplies are available.
And either way, the supplier doesn't have anything in stock, because their entire supply chain has collapsed, because international trade isn't a thing anymore. Did I mention that GPS (and thus most of sea/air navigation) has stopped working because the ground-based infrastructure needed to keep it running depended on, you guessed it, the Internet?
Still think that won't kill more people than the atomic bombings (which killed "only" 200k)?
My point on the iPhone was about what would happen after telephony infrastructure and supply chains had recovered (however long that takes). In the immediate aftermath of the shutdown mobile phones would absolutely lose the ability to make or receive calls. But again, I expect this to recover somewhat after a few months.
Your point on hospitals not being able to order supplies is a good one, regardless. Without blood supplies or dialysis equipment (or a bunch of other things), people would absolutely die. Perhaps those first weeks would be deadlier than I anticipated.
That's trivial in the industry context. A chunk of the stack is running decade old stuff.
> 2. The LLM needs to write a reliable exploit for that bug.
That's what Metasploit is for, isn't it?
> 3. The LLM needs to develop a worm that exploits that bug and spreads itself, opening access to the system.
See 2. if that's your strategy, but there are others. Such as, plant operations staff using random LLMs-as-a-service in their work (despite corporate saying not to do that; but it's not like the bosses don't do it either).
> 4. The LLM needs to connect to systems and understand if they are of any significance (it found a mayonnaise plant!).
Not hard at GPT-4 level, will only be easier. If it starts with a goal of doing something bad at scale, it will recognize a mayonnaise plant as an eligible approach, should it "cross its mind".
> 5. The LLM needs to understand the control protocols of their industrial control systems.
All documented and already part of the training set. I know that one for a fact, because I've been "chatting" with GPT-4 about some nuances of industrial protocol, and getting it to write me example code.
Industrial stuff may be closed-source and expensive, but the documentation and specs and marketing blurbs are to be found publicly. Most underlying protocols are public and well-documented (ish). The recent push for IIoT / "Industry 4.0" is actually trying to replace most of that proprietary stuff, secure by obscurity, with web-adjacent tech - exactly the thing that LLMs know best, because out sheer openness and popularity of everything webshit.
> 6. The LLM needs to understand how to make a dangerous composition from the ingredients it has on hand (let's pretend it can dump some industrial cleaning solution that is on standby for cleaning the tanks).
I bet that, should you work around "I'm sorry, as a large language model trained by Open AI, I'm afraid I can't do that" issue, GPT-4 will happily give you 5+ different ways to make mayonnaise lethal. It's not rocket science - it's industrial food production. A chunk of the processes there exist to ensure the product won't develop chemical or bacterial contamination.
> 7. The LLM needs to assume such total control over this processing plant that it can disguise the traffic and not trigger a single alarm around malfunctions.
Nah, it just needs to spoof some PLC outputs somewhere, or a data feed that goes to the model-predictive control. There's a risk of triggering alarms somewhere, and hopefully most of the naive approaches will get caught in the late lab testing / QC stages, but still - you can get far without triggering anything but maybe a dashboard warning about some outlier values, that plant operators brush off as more bugs in the industrial software.
That's if your goal is to weaponize mayonnaise. If you want to blow up the plant, well... skip step 7.
> There are a few billion dollar businesses in those steps.
If you saw how some of those businesses work, you'd be surprised we're all still alive.
> walk me through a situation where an LLM gets anywhere close to killing even 1% of the number of people an atomic bomb could.
Nobody is saying that GPT-4 can do it on its own. But to the extent that GPT-4 or a model more advanced than it already captures some essence of generalized thinking, and given the creativity people are showing in constructing increasingly complicated chains of LLMs and classical tools to extend both the breadth and the precision of that generalized thinking ability, plus giving them every possible tool in the world, it's not hard to imagine those systems getting capable enough to screw stuff at scale.
The ultimate argument is that atomic bombs, bioweapons and even climate crisis were all done thanks to intelligent agents. Intelligence is what gives rise to those threats, so by itself, it's more dangerous than all of them.
Also:
> What you're vaguely hinting at is extremely high skilled labor. (...) I welcome you to go read up on the challenges in automated exploit generation. LLMs are nowhere close.
We've only been dealing with AI models capable of basic coding tasks for less than a year. We've barely even begun to apply optimization pressure to this capability. So even as LLMs are "nowhere close" today, I wouldn't take the bet that they will remain "nowhere close" a year from now - there's absurd amount of money and interest invested into making them capable of this, by proxy of making them capable of software dev, or high-level thinking in general.
These videos changed my mind entirely about AI risk.
I listened to about 6 hours of Marc Andreessen on this topic this weekend and he just smashes all the doomer arguments.
Even in the context of nukes he made a great point that the invention of nukes most likely caused WW3 to not happen. An all out, devastating war between the US and Soviets over Europe that was only averted and transformed to the cold war because of MAD. I have been thinking since then how nukes probably caused my father to not fight in WW3. Who knows how many of us reading this would never have existed if nukes hadn't been invented. You can't make the argument 'the Manhattan project, we blow the planet up, the end".
He also makes the point that you can't be worried about a super intelligence that is super intelligent in every way besides being so dumb that it turns everyone into paper clips. You can't have it both ways.
If that's your goal, what's dumb about achieving it?
I don’t think that argument against anti-LLM uses works as the response would be “nukes weren’t launched as we had humans in the loop. Without a stopgap like that maybe a nuke would launch if there was a 50.001% chance of our side winning”
The counter counter argument to that would be that there isn’t enough weighing to how terrible a nuclear war would be and then maybe the LLM would change its mind. But who knows.