If it ever comes to that, it should be close to pair programming with AI and it should allow a mix of AI generated and human written code.
Like unsafe in Rust, we should be able to say "I know what I'm doing here, stay out of my way". Then let AI verify human written parts for correctness by means of tests or any other methods.
I think we sort of do this already, but it is the humans who verify the AI generated code and I believe we are a long way from swapping the roles.