One of the patterns I try to follow in designing our operations is that the tasks we need to follow in an emergency should be as close to routine as we can make them. We don't have a manual override to deploy into production in case of emergency, we make sure that our normal deploy process is suitable for emergency use. Which means we won't make the emergency worse by messing up a manual deploy.
Similarly, my car has some fancy drive-by-wire features -- the steering is dynamic, and the throttle balances the electric motor and petrol engine seamlessly. But the manufacturer didn't change anything fundamental about the controls, and if the power steering fails (or the cruise control stops working, or the radar can't track the vehicle in front) I lose some affordances and some safety systems but I can still drive the car.
It is true that the non-normal modes are supposed to only very infrequently activate, but with that in mind it should surely be more important to not drop users into a totally different control regime?
Hoping someone has some more insight. I don't have anything to do with avionics in general, and the day job isn't safety-critical, but I'm always keen to learn and I've definitely learned a lot over the years that I can apply to the day job by reading about how people design systems to stay safe when a failure means loss of life.