An hour of driving a taxi would cost you at least $150.
There's maybe $10 of non-human cost (car usage deprecation, fuel, insurance).
ZipCar, for example, is less than $10/hr and cheap car rental is $40/day.
Self-driving taxis, due to higher utilization and less management over-head, should be cheaper than car rental and more convenient for people.
So we're talking about at least 10x rate reduction for auto-taxis than current taxis which will change behavior in a big way (lower prices will lead to higher usage which leads to higher profits).
Ultimately the economics of self-driving taxi-like service should be: it's cheaper than total cost of ownership of owning your own car, even if you use it frequently (~2hrs a day).
There are other ways of selecting for this, but a self-driving taxi doesn't need to care and can just be managed as a fleet. Yes, some miles will need to be driven by an empty car, but for the most part demand for transportation has a balance, but this balance is very hard to reach in practice with actual drivers.
I'd argue that in Manhattan, the distances are short enough and the utilization high enough that self-driving probably buys you very little over a taxi today. With the scenario you describe, the cost difference is probably greater. (On the other hand, if that's 50% of today's taxi costs, most people won't be using it to commute every day.)
Of course, there are still other cases where it's not feasible for most people to hire a driver today--say going away for the weekend because the driver costs start to dominate the vehicle rental costs. (And in that scenario, you just rent a car today if you don't own one.)
In any case, when such vehicles become available in however many years, they will probably reduce car ownership. But, as with Zipcar, I suspect the difference will be at the margins.
Also there's an issue with balancing taxi load in the outer boroughs vs midtown / downtown manhattan. There's a reason uber has been so successful even in NY at increasing the total number of passenger trips and not just stealing customers away from traditional taxis.
NYC experiences are hard to generalize though. And traffic probably means you can't inject many more vehicles around peak times. My basic point was that, given high utilizations and somewhat balanced traffic patterns, taxi/Uber drivers don't add a huge amount of cost relative to the vehicle itself.