Think about it the other way: would you rather a below average driver behind the wheel? or a self driving car?
Everyone likes to think they're better than average (https://en.wikipedia.org/wiki/Illusory_superiority#Driving_a...).
I would rather keep the worst drivers off the road than keep the best.