It's nice to be able to walk places. Maybe a middle ground for having self-driving cars is to think of them as mini-trams.
The self-driving-car-mini-trams pick up and drop off at specific stops, on numbered routes, and they have a dedicated lane that can also be used by emergency vehicles (human driven).
To improve passenger safety the self-driving-car-mini-trams could be made larger, maybe the size of buses, which would have the added benefit of allowing people to stand in them, not use seat belts, and get off and on a lot more easily than crawling in and out of a car.