The positive economic effects and the transformative effect driverless cars will have on society is pretty hard to overstate. You seem to falling into the trap of thinking that infrastructure will not be updated around the assumption that cars are autonomous, and clever people will not exploit this new reality in ways that benefit the world. To attack your specific example, why do you assume the cars will drive around passenger-less, and not merely continually make productive uses of themselves, such as by servicing other passengers or acting as deliverymen for physical goods? Why do you assume people will own cars in general, when a service which provides on-demand autonomous transportation in clean, well-maintained, up-to-date vehicles will be more convenient, less work, safer, and will be cheaper? Why does the car that drops you off need to be the car that picks you up? Why do you think highways will not become wildly more efficient and safe even if the average passenger per car (one of several metrics of efficiency) goes up? Why do you think parking garages and cars themselves will not be able to be packed in more efficiently and quickly when they can autonomously coordinate their insertion and removal from a fixed area? Why do you even think that centralized parking garages will be as necessary when autonomously controlled vehicles can coordinate to efficiently fill any designated parking space that is road accessible?
When the assumptions of society change in such a dramatic way (that a human must navigate a road-based vehicle manually) the downstream effects are hard to predict but undoubtedly will follow the same path all automation has: more efficient, less polluting, and more productive use of time and energy. When you're dealing with something as ingrained and as prevalent as the automobile, you're talking about a massive boost in these things across the entire world.