I have never had a problematic ride, i'm always waiting exactly where I summon the Uber to. Not once have I made an Uber wait for me. I sit in the back and don't say a word unless spoken to.. is that what i'm doing wrong?
Also, I lurk in r/uberdrivers and one thing I've found is that a) some drivers are just jerks and 1 star at the drop of a hat, and b) some of it seems regional as well in that some countries/areas are more likely to use the 2-4 star options for minor things.
IIRC Lyft only looks back over the last 100 rides, which allows old ratings to cycle out. I know I've only been dinged once on Uber in the last several years but the rating is still held down by some old dings in the early teens.
My country rates differently and I've heard stories from my countrymen of difficulty getting an Uber when visiting the USA since not everyone in Australia gives 5 stars.