The main problem with self-driving cars is that they can't "read" humans' body language. A human driver can see pedestrians and cyclists (and other cars) and have a rough idea of what they're likely to do in the next few seconds, i.e. the pedestrian leaning out on the crosswalk curb is likely to step into the road soon. Even a reaction time of milliseconds can't make up for the (currently exclusive) human ability to read other humans and prepare accordingly.