Ratings by drivers are very often based on irrelevant criteria like how likeable you seem, or whether they interpret your preference for silence as an insult.
This is me as a rider:
a) Always at pickup on time (or I message "be right there" if I'm a minute away)
b) I enter and say "Hi, how's it going?" and acknowledge.
c) I then put in my headphones for audible. I say nothing else the rest of the trip.
d) I get out where it's most convenient for them, and I say "Thank you", and gently shut the door.
Yep, apparently they don't care - my rating hangs out in low 4.7s.
However, Uber Black always boosts my ratings, because the drivers are more mature and rate based on what actually matters.