Something I haven't considered much but, in a world where you hop into a self-driving taxi; how is the passenger protected from someone who wants to rob them?
I'm imaging a situation where an attacker can step in front of the vehicle, have it safely stop, before demanding you hand over your valuables, breaking a window to get in etc.
As a passenger you're trapped in an enclosed space and left without a way to escape, where as a human driver would be able to make the judgement call to do whatever is necessary to get out of danger.