A lot of people who drive cars do so because they don't like riding mass transit with the poor. Lots of people don't want to ride the train with people who are drunk and soaked in their own urine, or who are strung out on meth, or who are behaving aggressively, or are yelling at people, or... etc. Everyone who rides transit sees these things on a regular basis (Vancouver here) and I can completely understand others saying they don't want to give up their single occupancy vehicle.
I wouldn't see much of a niche for self-driving vehicles if authorities were free to enforce standards of dress and conduct on mass transit; but practically they can't (for so many political reasons in large cities) so they won't so there is such a niche available.
Sorry if this strikes you as being elitist, but I felt that it needs to be said. If we had functional mass transit that everyone felt safe using, I just wouldn't see much reason for self-driving cars.