Just a bystander but you just quoted this: > My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence.

Then did exactly what the quote is talking about, assuming we achieve better than human intelligence. > But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve.

The question is how exactly do we go from a below human intelligence to an above human intelligence. Without a clear path to that reality, making trade offs in order to protect against low probability worse case scenarios which require it doesn’t look like a good deal.

The reality that right now human level intelligences have a hard time even keeping these below human intelligence systems operating seems like a useful checkpoint. Maybe we can hold off on doing distasteful things like consolidating control until they can at least wipe their own bottoms as it were.