I'm sure I'm not the first person to point this out, but do we want a paperclip maximizer [0]? Because this is how we get a paperclip maximizer.

[0] https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...