Autopilot yells at you too if you don't interact with the steering wheel.
Although that is difficult, as in many cases the "cooperative" manufacturers have not yet implemented such features. They have only promised to implement them, sometimes in vehicles years away from production.
One very common scenario (that I have no idea how it would manage to handle) is someone driving with sunglasses on.