I don't believe you. Maybe you could construct an arguement for AntiHuman Strong AI or point me to one?
For example, if you make the AI "learn" a utility function about making cheeseburgers by using observations and reinforcement learning, and then the AI self-improves, you are most likely dead, because the learned utility function didn't include all possible caveats about not killing or torturing people to make cheeseburgers faster. And if you think you can keep applying negative reinforcement after the AI self-improves, think again.
Yes, but you're still assuming that we're talking about an agent that can take decisions and act autonomously, rather than an inference engine that just processes data and spits out its inferences with zero autonomy whatsoever.
The former is extremely, stupidly unsafe by default, so, logically, people probably won't try to build any such thing. They'll build the second sort of thing, which will mostly just be a more advanced version of today's statistical learning.
Unless, of course, you're talking about ideological Singulatarians, who may well face legal sanction one of these days for deliberately trying to build the former agenty sort of thing as if that was a good idea.
(Protip: If you want to talk "AI safety", we can do that on the site devoted to it, but out here in broader where nobody's damn fool enough to try to build agent-y "AI", mixing up realistic ML with the kind of agent-y "AI" you'd have to be blatantly suicidal to build is an abuse of terminology.)
Besides which, "the capacity to self-improve" is actually a currently-open research problem. Currently. Once I work some things out, I've got something to report about that...
1) Your plan for AI safety is "no one will be stupid enough to build a self-improving AI".
2) You are currently working on self-improving AI.
I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-(
You mean the one I belong to? Like I said: go on LW and talk about "AI" with assumed context. It's just out here in the rest of the world where you can't assume that everyone automatically has read the literature on AIXI/Goedel Machines/etc and considers "agenty" AI to be a real thing.
>2) You are currently working on self-improving AI.
Hell no! I'm working on logic and theorem-proving in the context of algorithmic information theory -- really just dicking around as a hobby. If you want "stable" self-improvement for your "AIs", you need that. It's also not, in and of itself, AI: it's logic, programming language theory, and computability theory. And if I get a result that holds up, which is an open if, I'd be happy to keep it the hell away from "AI" people.
The main reason I don't consider alarmism warranted about "self-improving AI" (though I don't count any of FLI's letters as alarmism) is that I think of "an agenty AI" as something put together out of many distinct pieces. It's arranging the pieces into a whole and executing them that's unsafe, but also currently prohibitively unlikely to happen by accident. Naturalized induction and Vingean reflection wouldn't be open problems if self-improving "agenty AI" was so easy it could happen by accident.
I fully agree that one does not build a self-improving agenty AI under basically any circumstances, ever, even if there's quite a lot of guns to your head and various other unlikely and terrible things have happened, as the research literature stands right now.
The biggest mistake it makes is assuming the ability of goals to stay hard coded as general intelligence advances. That seems antithetical to increased intelligence. Right? How smart can you get if you're unable to change your mind?
The second mistake is the assumption of an intelligence explosion. Why? It's a lovely idea, as middle age encroaches on me I long for the rapture of the nerds, but it's just a fairy tale. Intelligence explosion is an untestable hypothesis. It's so useless it's not even wrong. Intelligence is one of a multitude of survival strategies available to agents in a world. What in the natural world or the world of human tools points to an intelligence explosion? Nothing. Intelligence is linear and hard fought, not exponential. Intelligence has to compete with all the other strategies. An no that's not a chink in the armor of my argument, because the whole thought PM thought experiment requires an explosion of intelligence.
^ this. Some of the most intelligent people live a low-key, low-consumption life, often not even reproducing. That makes me hopeful that an AI actually able to surpass humans in thinking capability (if possible) will not build an endless stream of useless paper clips.
There is a tendency to view AI as god in these circles. If it is god and be in all ways superior to us, why would it be at all blindly following the rules that we implemented in it - maximizing paperclips?
Oh and I am not saying there is no danger or weirdness ahead. There clearly is. But I don't see the paperclip maximizer emerging.
In short, the argument isn't that the AI will become more AntiHuman as it evolves. Rather, the AI's existing utility functions might not be aligned with human utility functions from the outset, which could have negative consequences. It's hard to make an AI do what we actually want it to.
Your cheeseburger AI that logically decides humanity's continued existence is detrimental to the perfect cheeseburger: Even if it manages to make the jump from iterating on beef convection models to suggesting murder, what can it actually do? If it starts adding "kill yourself" to the end of the updated recipes it spits out, how is that any more dangerous than a dumb kid on Twitter?
If you were trapped in a box with access to "just some APIs", and you could think 1000x faster than your jailers and could also self-improve, you'd find a way to get out and achieve whatever you consider to be your goals.
a) There are business incentives to give the AI more capabilities and more information about the world.
For example, if you have a cheeseburger bun factory with a complicated production line then you could use the AI to come up with more efficient layouts and machines for making cheeseburger buns. After awhile you realize that humans implementing what the AI explained is by far the slowest part of the pipeline, and ask if there's some way to have the AI build the production line by having it directly do the online ordering or 3d printing or whatever. At first you pay people to sanity-check what the AI is ordering and doing, but they never notice any problems so eventually you cut costs by laying them off.
b) Side channels and exploits are a thing.
Imagine yourself in the AI's place [1]. Is there really nothing you can do? Or is the security based around the fact that you don't try to escape?
"Hey Larry, why is the description for so many orders coming out as 'Warning: ketchup parameter set too high. See <a href="javascript:{...sudo...WebSocket...dumps.wikimedia.org.../resetpassword/?user=billgates...}">the manual</a>.'?".
"God damnit Ted, do you not see the link that says 'the manual'? Maybe you should click that before bothering me!"
c) Instrumental goals.
Any optimization process with an unqualified goal like "find a way to use this API to make lots of burgers, then do that" will favor plans where said process ends up with huge amounts of power over the world; simply because more burgers get made in that situation than in other situations. In some senses, failing to actively search for escape exploits could be seen as a design flaw because the algorithm is failing to find clever solutions that are better according to the stated goal.