This is simply not true. I just checked: My rating is 4.73. I've probably taken a few hundred rides with Uber. I assume they adjust ratings for bad experiences, especially if you're a regular rider.
By the way: It's not that hard to get consistent 5 stars as a passenger. Be respectful of the driver and follow these steps: 1) Say something like "Hi, are you looking for Brandon? That's me. Good evening. How's it going?" 2) Suffer less than 20 seconds of smalltalk 3) Go about playing with your phone or talking with your co-passengers quietly. 4) Say "This is my stop right there by the red car on the right. Thanks a lot, you have a good night!" - If you're an extrovert, you'd do this anyway (assuming you're not an asshole) and if you're an introvert, it's a good opportunity to exercise your smalltalk with low stakes.