>
1. The LLM needs to find an exploitable bug in a popular code base.That's trivial in the industry context. A chunk of the stack is running decade old stuff.
> 2. The LLM needs to write a reliable exploit for that bug.
That's what Metasploit is for, isn't it?
> 3. The LLM needs to develop a worm that exploits that bug and spreads itself, opening access to the system.
See 2. if that's your strategy, but there are others. Such as, plant operations staff using random LLMs-as-a-service in their work (despite corporate saying not to do that; but it's not like the bosses don't do it either).
> 4. The LLM needs to connect to systems and understand if they are of any significance (it found a mayonnaise plant!).
Not hard at GPT-4 level, will only be easier. If it starts with a goal of doing something bad at scale, it will recognize a mayonnaise plant as an eligible approach, should it "cross its mind".
> 5. The LLM needs to understand the control protocols of their industrial control systems.
All documented and already part of the training set. I know that one for a fact, because I've been "chatting" with GPT-4 about some nuances of industrial protocol, and getting it to write me example code.
Industrial stuff may be closed-source and expensive, but the documentation and specs and marketing blurbs are to be found publicly. Most underlying protocols are public and well-documented (ish). The recent push for IIoT / "Industry 4.0" is actually trying to replace most of that proprietary stuff, secure by obscurity, with web-adjacent tech - exactly the thing that LLMs know best, because out sheer openness and popularity of everything webshit.
> 6. The LLM needs to understand how to make a dangerous composition from the ingredients it has on hand (let's pretend it can dump some industrial cleaning solution that is on standby for cleaning the tanks).
I bet that, should you work around "I'm sorry, as a large language model trained by Open AI, I'm afraid I can't do that" issue, GPT-4 will happily give you 5+ different ways to make mayonnaise lethal. It's not rocket science - it's industrial food production. A chunk of the processes there exist to ensure the product won't develop chemical or bacterial contamination.
> 7. The LLM needs to assume such total control over this processing plant that it can disguise the traffic and not trigger a single alarm around malfunctions.
Nah, it just needs to spoof some PLC outputs somewhere, or a data feed that goes to the model-predictive control. There's a risk of triggering alarms somewhere, and hopefully most of the naive approaches will get caught in the late lab testing / QC stages, but still - you can get far without triggering anything but maybe a dashboard warning about some outlier values, that plant operators brush off as more bugs in the industrial software.
That's if your goal is to weaponize mayonnaise. If you want to blow up the plant, well... skip step 7.
> There are a few billion dollar businesses in those steps.
If you saw how some of those businesses work, you'd be surprised we're all still alive.
> walk me through a situation where an LLM gets anywhere close to killing even 1% of the number of people an atomic bomb could.
Nobody is saying that GPT-4 can do it on its own. But to the extent that GPT-4 or a model more advanced than it already captures some essence of generalized thinking, and given the creativity people are showing in constructing increasingly complicated chains of LLMs and classical tools to extend both the breadth and the precision of that generalized thinking ability, plus giving them every possible tool in the world, it's not hard to imagine those systems getting capable enough to screw stuff at scale.
The ultimate argument is that atomic bombs, bioweapons and even climate crisis were all done thanks to intelligent agents. Intelligence is what gives rise to those threats, so by itself, it's more dangerous than all of them.
Also:
> What you're vaguely hinting at is extremely high skilled labor. (...) I welcome you to go read up on the challenges in automated exploit generation. LLMs are nowhere close.
We've only been dealing with AI models capable of basic coding tasks for less than a year. We've barely even begun to apply optimization pressure to this capability. So even as LLMs are "nowhere close" today, I wouldn't take the bet that they will remain "nowhere close" a year from now - there's absurd amount of money and interest invested into making them capable of this, by proxy of making them capable of software dev, or high-level thinking in general.