> If you design an AI with the express purpose of making cheeseburgers, and allow it to improve itself, it will end up killing people
I don't believe you. Maybe you could construct an arguement for AntiHuman Strong AI or point me to one?
I don't believe you. Maybe you could construct an arguement for AntiHuman Strong AI or point me to one?
In short, the argument isn't that the AI will become more AntiHuman as it evolves. Rather, the AI's existing utility functions might not be aligned with human utility functions from the outset, which could have negative consequences. It's hard to make an AI do what we actually want it to.
For example, if you make the AI "learn" a utility function about making cheeseburgers by using observations and reinforcement learning, and then the AI self-improves, you are most likely dead, because the learned utility function didn't include all possible caveats about not killing or torturing people to make cheeseburgers faster. And if you think you can keep applying negative reinforcement after the AI self-improves, think again.
Yes, but you're still assuming that we're talking about an agent that can take decisions and act autonomously, rather than an inference engine that just processes data and spits out its inferences with zero autonomy whatsoever.
The former is extremely, stupidly unsafe by default, so, logically, people probably won't try to build any such thing. They'll build the second sort of thing, which will mostly just be a more advanced version of today's statistical learning.
Unless, of course, you're talking about ideological Singulatarians, who may well face legal sanction one of these days for deliberately trying to build the former agenty sort of thing as if that was a good idea.
(Protip: If you want to talk "AI safety", we can do that on the site devoted to it, but out here in broader where nobody's damn fool enough to try to build agent-y "AI", mixing up realistic ML with the kind of agent-y "AI" you'd have to be blatantly suicidal to build is an abuse of terminology.)
Besides which, "the capacity to self-improve" is actually a currently-open research problem. Currently. Once I work some things out, I've got something to report about that...
1) Your plan for AI safety is "no one will be stupid enough to build a self-improving AI".
2) You are currently working on self-improving AI.
I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-(
You mean the one I belong to? Like I said: go on LW and talk about "AI" with assumed context. It's just out here in the rest of the world where you can't assume that everyone automatically has read the literature on AIXI/Goedel Machines/etc and considers "agenty" AI to be a real thing.
>2) You are currently working on self-improving AI.
Hell no! I'm working on logic and theorem-proving in the context of algorithmic information theory -- really just dicking around as a hobby. If you want "stable" self-improvement for your "AIs", you need that. It's also not, in and of itself, AI: it's logic, programming language theory, and computability theory. And if I get a result that holds up, which is an open if, I'd be happy to keep it the hell away from "AI" people.
The main reason I don't consider alarmism warranted about "self-improving AI" (though I don't count any of FLI's letters as alarmism) is that I think of "an agenty AI" as something put together out of many distinct pieces. It's arranging the pieces into a whole and executing them that's unsafe, but also currently prohibitively unlikely to happen by accident. Naturalized induction and Vingean reflection wouldn't be open problems if self-improving "agenty AI" was so easy it could happen by accident.
I fully agree that one does not build a self-improving agenty AI under basically any circumstances, ever, even if there's quite a lot of guns to your head and various other unlikely and terrible things have happened, as the research literature stands right now.
The biggest mistake it makes is assuming the ability of goals to stay hard coded as general intelligence advances. That seems antithetical to increased intelligence. Right? How smart can you get if you're unable to change your mind?
The second mistake is the assumption of an intelligence explosion. Why? It's a lovely idea, as middle age encroaches on me I long for the rapture of the nerds, but it's just a fairy tale. Intelligence explosion is an untestable hypothesis. It's so useless it's not even wrong. Intelligence is one of a multitude of survival strategies available to agents in a world. What in the natural world or the world of human tools points to an intelligence explosion? Nothing. Intelligence is linear and hard fought, not exponential. Intelligence has to compete with all the other strategies. An no that's not a chink in the armor of my argument, because the whole thought PM thought experiment requires an explosion of intelligence.
^ this. Some of the most intelligent people live a low-key, low-consumption life, often not even reproducing. That makes me hopeful that an AI actually able to surpass humans in thinking capability (if possible) will not build an endless stream of useless paper clips.
There is a tendency to view AI as god in these circles. If it is god and be in all ways superior to us, why would it be at all blindly following the rules that we implemented in it - maximizing paperclips?
Oh and I am not saying there is no danger or weirdness ahead. There clearly is. But I don't see the paperclip maximizer emerging.