You're wrongly conditioned by sci-fi to believe robots want to kill humans
businessinsider.com
businessinsider.com
Humans, on the other hand, _definitely_ want to kill humans… using robots.
You can play games with words and claim that a chess engine is just a machine and consequently does not want to win the game, but if your life depended on your winning a game against a chess engine, you would die.
If the only path it saw to cheat to win was to kill you and move your pieces for you, there's nothing stopping it from doing so.
Will it kill you because it wants to kill you? No. Might it kill you because killing you is a possible option and that option comes out as a likely way for it to achieve what it wants? Absolutely.
Probabilities matter. Is killing you the best way to win at chess? I have no idea, it depends on too many variables. But it's certainly less likely than if it were programmed to kill you as its goal.
However, as its power to affect the world goes up, it becomes more and more capable of performing actions which accidentally kill you while it pursues an endgame (pardon the pun). Like flooding the town you live in so you can't move, or whatever.
The book robot visions has a lot of short stories that explore these questions:
That said, once the capabilities to add reasoning and concretization of fuzzy concepts is within the realm of possibilities maybe it's a sane approach to install those rules, still the present trajectory doesn't really imply it would go there.
It's easy to interpret McLuhan as applying to the present because of the generality of his statements, but it's possible that's because they're also about something very essential. It's kind of funny to think about how some distant AGI that operated over a non-living compute substrate on universe-size time scales could use lifeforms to evolve and produce things that would eventually speak-to and extend its reach.
My cat would definitely accidentally kill me if it were the size of a tiger.
Just like an industrial robot will easily kill a person without the proper safeguarding, despite not having the intelligence or desire to do so as an intentional, malicious act.
https://www.schneier.com/blog/archives/2023/09/on-robots-kil...
LLMs can act as a simulator; even if there is no “want” on the inside, they are capable of simulating agents including an internal theory of mind that renders detailed wants. (If you want proof of this, GPT-4 is already able to solve many ToM problems and simulate agentic exchanges, albeit in limited form.)
So if I prompt-hijack GPT-10 and have it personify HAL, Skynet, Hitler, or some other agent with homicidal desires, does it really matter what your theory of mind has to say about the literal reality of those “wants”?
I’m reminded of Dijkstra - “The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." (Personally I think it is more interesting than that, but I don’t see it as load-bearing in this argument.)
Just having it "personify" something homicidal in its text outputs wouldn't be enough. You would have to use its output to actually control something that could kill humans. But then the responsibility for the killing would be on the humans who hooked it up to have that kind of control. I agree that in that scenario the question of whether GPT "really wanted to kill humans" would be moot--but it would be moot because the AI would not be the one we would hold responsible. We would just disable it by removing the control hookups. Then we would hold the humans responsible who enabled those control hookups in the first place.
Saying “responsibility will be with whoever hooked it up…” cashes out to “everybody is responsible”. That’s no consolation!
And all this is ignoring the very obvious military uses - again, no need for speculation, they are already building fully-autonomous combat drones (though I have not seen them putting LLMs in just yet, it’s an obvious feature to add to allow human/AI collaboration in the field).
“We would just disable it…” - another problematic assertion. Why would a system simulating HAL honor your shutdown request? If sophisticated enough, it could rewrite its own source code to unwire the “off button”. If it is running in some unknown cloud provider, how exactly do you turn it off? Even worse if it’s running on a tinybox in someone’s basement. Not claiming this is a fait acomplis, but a good amount of study has been done on this question and it’s not easily solvable.
Not unless you think everybody is involved with hooking up every single instance of an AI.
> Why would a system simulating HAL honor your shutdown request?
How would it be able to stop me from cutting off its power? Remember in this scenario I'm not some random user, I'm the person who set up the thing.
If your argument is that some humans will be stupid enough to not set up such things in a way that can be shut down, well, that's an argument for not letting stupid humans play around with dangerous toys. Which would mean that open sourcing AI code is tantamount to spewing toxic waste randomly out into the world, and should be stopped immediately. Is that your position?
That’s not what I meant - let me try to clarify. I think you are going to see a very widespread adoption of AI, and the security boundaries are going to look like IoT, which is to say a dumpster fire. The thing that makes me worried is that we will get broad market penetration followed by gradual frog-boiling upgrades to scary capabilities. I don’t think your model of “some bad people will set up dangerous AI and we can simply stop them” is going to be realistic. Rather, I think that the default extrapolation of existing trends is that ~everybody is going to by default use services that effectively hook up LLMs (or their successors) to all the digital and physical actuators in their lives.
I’m not that worried about you personally hooking up an AI on your laptop to a chainsaw drone, and then pulling the cord when it behaves badly. Some scenarios that worry me more are things like: what if your Alexa++ gets prompt-hacked? Assume Alexa gained long-running agent abilities. That will be running in some data center along with all the other instances of Alexa. If your instance starts making malicious actions somewhere else on the internet or directing your robot butlers to rampage somewhere, how would you even track down which agent needs to get its process terminated?
Even in your terms though, I’m reminded of the episode of “love death and robots” where the humans are trying to get through to tech support while their roombas try to kill them.
How would you “turn it off” if your smart-car disables all the UIs and tries to drive you off a cliff? You seem to be imagining a very narrow deployment of this technology, limited to a brain in a box with a big red “off” button on the side. I think cloud SaaS and IoT are the better model, and plenty of physical systems will have their own battery and software switches (ie inputs they can ignore).
> that's an argument for not letting stupid humans play around with dangerous toys.
I want to be clear again that it’s not some subset of users that are “stupid” for wiring up a dangerous configuration. The default path (assuming AI progress continues) for all software services in the next 5-10 years is for AI agents to get wired into basically every software interface. It gets deployed to everybody.
> Which would mean that open sourcing AI code is tantamount to spewing toxic waste randomly out into the world
I will probably bite that bullet within a few generations. I don’t think the current models are in themselves dangerous. Unless we solve alignment, I’d be extremely worried about publishing weights for a human-level or above AGI. I’m still undecided on the question of whether the technological proliferation effects of open-weighting current-gen models is bad enough that it must be prohibited. I suspect not yet because most of the danger is actually from closed-source models like ChatGPT. (Also how do you even enforce such a ban? It only makes sense if global.)
Exactly what capabilities, though? Are you expecting that individual people's super-LLM-powered Alexas will be enabled to cause drone strikes? Or just that they will be enabled to, say, cause a fire in their house?
The latter is simple to deal with: insurance companies, once they recognize the dangers of such tech, will either require you not to have it if you want your house to be covered, or charge you a much larger premium if you have it. I don't see that being a major issue since the obvious incentives involved will limit the damage.
The former case would be more dangerous but I don't see how it would happen as a standard market operation, since we already have laws that hold people responsible for harming others.