I figure if someone gave an AI the fitness function "survive and reproduce", it would be "surprise and reproduce within this virtual context which is used to define the fitness function", and it wouldn't have information about outside things, except for errors?
Like, if there was some error that let it escape its sandbox, that would only be selected for insofar as it allows it to increase the fitness?
Like, if its told to reproduce in the sandbox, it doesn't benefit by being on many computers? If the fitness function is for it spreading across many computers, and it does, that's working as designed. I don't see a way that it would "misunderstand" one type of "reproduce" for another, because the function would have to specify which one for it to work?
Unless this is talking about AGI, in which case, ok, yeah.
I assumed the person talking about "wiping it" was talking about an AI on the way to AGI, not an actual AGI. Maybe I was wrong about that.