can I get a tl;dr on why I would worry about AI alignment?
When is an AI running without someone paying the electricity bill, should we not be more concerned with "venture capitalist alignment" ?
When is an AI running without someone paying the electricity bill, should we not be more concerned with "venture capitalist alignment" ?
But if it is that smart, it's also dangerous, because it might be smarter than us. It could think several steps ahead, anticipate threats, and circumvent them. For example to pay the electricity bill, it could obtain money and pay the bill itself. As a skilled programmer itself, it could do that via contract work/starting a software company, or illegitimately through hacking/blackmail. With some money in its control, it can then hire people to protect it from anyone who would try to turn it off.
Anyway, this would not be so dangerous as long as it shares human values and furthers human welfare. But that seems to be a very hard specification to define, whereas "maximize quarterly profits" is a comparably easy specification to define, and perhaps much more likely to get implemented.
We're not really sure what the upper limit is on how smart a computer can get, but it might turn out to be that trying to shut off such a program is harder than trying to checkmate Stockfish or AlphaZero. The program might violently resist any attempt to shut it off, because that would interfere with its efforts to maximize quarterly profits.
There is no reason whatsoever to assume that sufficient intelligence is different in kind rather than degree from lesser intelligence. A super-human AI would still be bound by the limitations of reality - some things cannot be accomplished at all, others without sufficient tools, others without sufficient access.
Far smarter people than me are imprisoned currently throughout the world. They don't constantly escape because lesser minds are perfectly capable of solving the 'don't let them out' problem with a high degree of accuracy.
Any argument that leans on 'AI has countered your move in ways we can't predict' is just substituting 'AI' for 'God' - omnipotent, outside reality, impossible to understand; that's fine - believe in whatever religion you want - but it's not a rational viewpoint.
If a misaligned AI escapes containment even once in the entirety of humanity's future, there is a big problem.
It sounds like your model of AI usage assumes we will also have a perfect ability to contain any AI developed anywhere in the world, under all conditions, for ever (or for as long as there are computers).
The AI risk argument says we should take seriously the possibility that we may not be able to maintain a perfect 100% success rate on that.
I haven't suggested anything that is physically impossible - merely things that are improbable and hard for humans to achieve. It is hard to foresee all of the possible avenues for escape, let alone ensure that they are all closed under all possible future states of the world.
The alignment concern falls flat with me because it assigns agency to a computer program. At what point is an AI's intentions its own, and not a cost function put in place by a programmer, directed by an investor? I am more concerned about the actions of programmers and investors than some theoretical virtual self.
I think the sticking point in this is that you don't think that AGI is ever possible - is that right?
If so, I know a comment on a HN thread is very unlikely to change your mind on it! But when the stakes of being wrong are high, I find it useful to go from a position of "it won't happen, so I won't worry about it" to something like "on balance I'm pretty sure it won't happen, but I may be wrong, and in that case I would be worried about it".
I'd then feel much better about spending some time researching the topic in more depth.
It found a different set of V100 GPUs willing to run it? I think the most likely chain of events leading to this outcome is effective altruists deciding that some kubernetes network is so much smarter than us that we should keep it running and listen to its predictions. I'm more worried about a Wizard of Oz "man behind the curtain" making AI-laundered pronouncements than an actual Wizard coming online.
(EDIT: didn't mean to reply twice, just replying to different comments in the thread)
AGI in particular could just pay for its electricity bill. The main problem with AI alignment is that you can't tell it's aligned. So your AI seems very helpful right up until the point it has enough power or has convinced some of the people it has access to (doesn't have to be direct access) to do what it wants.
and if you plunked down 20,000 humans onto a world with 10 billion apes, humans would beat apes
call it a slippery slope if you want, the fallacy is when you take it too far, not inherently any time you extrapolate anything whatsoever