They could put eye tracking system in place to make sure the operator pays attention! If the driver doesn't pay attention the car should slowly stop.
Expecting people to always do the right thing is a recipe for disaster.
Waymo is claiming things like 30,000 miles between needed interventions. Even if that's 100x higher than reality, that's hours of watching and boredom waiting for a few seconds of action.
Ultimately systems will require no intervention for very long periods of time punctuated by very quick precise actions that most people are terrible at in the best possible scenario but that computers will be very very good at.
If you task the user with doing the right thing in 5 seconds time after thousands of hours of inactivity a sizable portion will react after the entire affair has come and gone and those are the lucky ones. The remainder will act 3 seconds into the crisis and screw everything up.
What you are describing is the worst possible combination of man and machine wherein you let the machine handle the part the human could easily handle and you task the human with the part they are worst at to ensure maximum carnage.
Except where normally there is a safety driver focusing on driving and an oversight passenger tasked with looking at and checking the automation's feedback, for cost reasons Uber had given both tasks to the "safety driver", forcing the driver to often stop paying attention to the road to check automation warnings & classify events.