Once they're available (and I see no reason not to continue thinking that door-to-door self-driving is decades out), the ownership question definitely gets interesting. But I don't think it's as obvious as some think in part for the reasons you say.
Furthermore, a lot of people use their cars at the same time of day. It's also the case that a lot of auto depreciation is mileage-based rather than time-based (especially in non-snowy/salty environments).
The questions others raise about the maintenance/update requirements for these sorts of vehicles are interesting ones.
My guess is that, like Zipcar and Uber, they will make a difference at the margins but won't cause a wholesale shift. But who knows? Especially if, as I believe, this happens over a number of decades with demographic shifts happening in parallel.