It feels weird saying this, but there is a sort of physical threat that being behind the wheel gives you.
Same goes for when the lights at a junction are out of service. How will an AI handle that?
In general, proper autonomous cars are going to need a shit-ton of data collection at a very fine level everywhere. Together with the required sensor suite, the cost increase over a normal car in the same class is going to be huge. I'm not talking 20%, I'm talking at least twice as expensive.
This is why I think Ford are making exactly the right bet: taxi companies will adopt driverless cars many years, even decades, before private drivers. Because a taxi company saves a lot of money on not having any drivers (I'm guessing not far from 100-150k per year per car) on an autonomous car, they could easily afford to pay $150k for a Ford Fusion (normally $30k).