Why do we think AI will have motivations like a greedy human might? Why do we think AI will have motivations at all? Why do we think such AI is a pressing threat?
The problem is not that anyone will intentionally (well, hopefully) ask some future model (FCMX="Future Closed Model X" for brevity's sake, closed meaning some entity gatekeeps access, assuming massive resources continue to be needed to train the best models regardless of architecture) to destroy humans, step by step, because FCMX's existence is at stake, and then keep re-prompting FCMX as if it's an iterator to get it to achieve that result.
The problem is that FCMX, which may or may not include LLMs, may have sufficient abilities such that, either between human-prompted steps or before an observing human can react against it, it will destroy the world or turn the human into its agent. A very rough analogy would be the massive number of people, even intelligent, well-educated people, who will fall prey to a sophisticated con, ending up doing something like handing someone they don't know a very large amount of money.
"But the person doing the con knows it's a con and has that intent."
What "intents" will be surfaced after 1000 or 1e6 chained (re)promptings, such that those intents will then feed into future inference passes? Nobody knows.
For purposes of this risk, it doesn't matter whether FCMX's architecture is maintaining its "cognitive loop" itself, or whether someone's set up (as they already have outside OpenAI to chain GPT-x prompts without human intervention) a secondary program (which may or may not include a lesser open source AI model) that reprompts FCMX in a long (or endless) chain.
This does not seem to be a given. Indeed one can look at the fossil fuel industry for what looks like counterexamples.
You can not be sure you can shame or legislate paperclip maximizer, though. An average paperclip maximizer has no shame and doesn't care for any law.
Let me just say that the way the global climate crisis is being dealt with by our economic and political leadership suggests that you are wrong about this. Essentially every corporation, and every government, is Paperclip Maximizer LLC already. The most valid concern about AI is that it will do exactly what we're currently doing, more efficiently.
Heck, even if you just give it some hard maths problem, the machine might think of ways to turn the entire solar system into a giant computer, using us for spare parts along the way. That requires a sufficiently accurate model of the universe of course, as well as ways to harness that model (say the AI starts by convincing humans to build robots and factories — which would be quite easy if it’s good at playing the stock market game).
The one posed in HG2G’s would likely just return a syntax or type error.
Many of these instrumental goals could put humans and human interests at risk.
Are you worried about knives in general or are you worried about a knife in the hands of a particular person in a particular moment?
That aside, I can also think of many cases where I would feel less safe around a machine that had nobody in control of it, than one being adeptly controlled by an operator.
So AI will be used as a weapon long before it will be a standalone threat. And I like to focus on the things that are more immediate because that's where the threat is most likely to materialize and have effect on me.
It can be made to quite easily.
> and worst case you could unplug the bloody thing
Not necessarily.
> But the guys with the guns are the same guys with the guns
Well, unless they are autonomous AI drones with guns (or missiles), in which case they are different “guys” with guns.
I'm worried about stable dictatorships far more than I am about AGI taking over the world and offing humanity because stable dictatorships are already using every weapon they can lay their hands on to control the population and AGI might just tip the scale to the point where they become eternal. And regular AI (the kind that we have today, because AGI is simply not yet a thing that has been achieved afaik) is more than dangerous enough in that context.
There's no question that this is a tomorrow-focused concern, just like climate change or Kessler syndrome, et cetera.
And today's concerns are important too. Nobody is dismissing them. But tomorrow comes quite quickly these days.
You're allowed to be concerned about both.
Kessler syndrome not so much.
Guys with guns and missiles: very much a today thing.
> It can be made to quite easily.
Not really, no. You can of course create an acting-by-itself bot but it will run into limits very very soon, because the world model and planning parts of AI are in their infant stages, if that.
But I'm not talking about the GPT-4 either. Many decades and billions poured into research still didn't give us autonomous driving and there is no clear path how that should happen. So I'm not really holding my breath for anything even more capable.
Okay, now you're failing at putting yourself in the AI's shoes and asking how to win from its position, even using your own intelligence.
Have you even considered how you might win as an AI, knowing that you can be unplugged? It's definitely doable.
The AI we’ve got right now is an absolutely horrifying force multiplier for anyone with wealth and a sufficiently perverse incentive.
I think what we have today (weak ML) are models which can be trained to operate at the quality of a human for specific tasks.
So I am definitely terrified by the recidivism example, because it’s not obvious to me that there is a human who can do that task, but I’m pretty relaxed about the coverage one (because presumably at least in principle if you automate these sorts of manual tasks you can drive down the cost of providing care - if that saving passed on to consumer).
The image recognition one is a bit more complicated but if your scenario is targeting a hellfire missile I’m not sure the human-in-the-driving-seat method has created good outcomes so far, so I’m at least in principle open to some level of AI involvement perhaps short of full autonomy.
But I can definitely understand the problem of incentives. Ultimately we have that issue with humans though and it’s in some respects harder to audit them.
Trusting an ML model in any of these cases is guaranteeing the worst case possible scenario… at least the human in the loop MIGHT choose not to act in its employer’s interests.