By not having the human make that decision, you save lives, even if some people arrive home late.
It does raise the point of 'rescue' in certain dangerous conditions like winter storms. Extreme rain in the dark can probably normally be waited out, but snowstorms and other road-closure type conditions probably warrant a different proactive rescue type response if we'll have riders with no driving ability in self-driving cars.
Like anything it should be a graduated phase in. It will handle some of the conditions some of the time, and in time it will get better. It would be like me being frustrated I can't carry on a conversation about philosophy with my Google Home. "... but you said I could talk to it and ask it questions!!!"