I think this is absolutely true; and you'd want to show drivers locations - I can't think of any way to obscure this, and it does leave a source for discrimination ("don't drive into the dangerous neighborhoods").
I'm not sure I have any good solution to that particular problem, but surely perfect is the enemy of good, and the human anonymization approach is reasonable low hanging fruit?