If someone has a problem with a ride, the only thing they really care about is letting Uber know that the ride was terrible (I bet the rating distribution clusters heavily around 1, 4 and 5 stars).
Looking at passenger rating histories should also play a role in how these are interpreted (if someone is an asshole and rates tons of drivers poorly even if they're well liked, that should be discounted)
By doing something like this, you have a way of weeding out bad drivers, rewarding great drivers, and leaving everyone in the middle alone.