What phone are you using?
In a decade of living in Manhattan, I've never had any issue like this. The location services on my phone (Pixel 2 XL) and every Android phone I've ever used have been spot on.
Location Services doesn't just use raw GPS, no, but that's what most drivers (and passengers) are using.
In any case, for several years, I only used the device GPS (instead of the full location services), and I never had any issues like what OP is describing.
I have no idea if that's true and have always been confused about whether the driver app even has maps built into it.
The Uber native maps app is almost invariably incorrect, at least here in Brooklyn. It does insane stuff like trying to put you on the BQE (interstate) to go like 800 yards or route you from Brooklyn to Brooklyn by transiting Manhattan and two bridges.
Until they figure out how to fix that they pretty much need to let drivers flip over to Waze or something.
That's unfortunate. As I said, every Android phone I had has been spot on everywhere in the city.