You're getting really close to sounding racist, and some would say you already are.
Why would there be a reason black people would give cab drivers a harder time than other skin colors? There isn't any good reason.
As for neighborhood differences, that's a problem of racism (segregation) too.
This is clearly happening, based on the anecdote of cab drivers avoiding picking up black people as well as telling the OP he is the first black person who they had a positive experience with. Some stereotypes are based in reality, and it isn't racist to acknowledge this.
What the above parent wrote is, "there's no good reason for black people to give cab drivers a hard time." This implies that, given that there's not a good reason, it is not likely true.
So, what do you think the reason would be that black people would unanimously give this cab driver a hard time, except for this one guy? Based upon your above proposition, you may also want to explain why your explanation is not racist.
I could definitely believe that black people are less likely to tip (or avoid paying altogether), as black people are poorer (on average). Black people are also much more likely to commit violent crime (on average). Note that cab drivers are likely to not be white (on average), at least in my experience. I did some research and it appears to be common for cab drivers to avoid picking up black passengers - this is pretty strong evidence that cab drivers have had negative experiences with black passengers.
It is racist for cab drivers to avoid picking up black passengers, but we shouldn't pretend that they don't have a reason for it.
When I first read this I thought of
https://en.wikipedia.org/wiki/Cow_tipping
and it took me a moment to recover!
(but read http://modernfarmer.com/2013/09/cow-tipping-myth-or-bullcrap... for a refutation)
But that isn't the cabbies problem to solve. Its perfectly reasonable for a cab driver to try to avoid high-crime areas that are much more dangerous even if he doesn't have one racist bone in his body.