There's a big difference between seeing and interpreting. The car sees literally everything and can maybe identify most things, but a pedestrian window shopping across the street isn't really that interesting and neither is a bollard. As humans, we filter out the unimportant stuff to focus on the important stuff. We can also pickup on subtle clues in posture, through eye contact and even through the driving style of another person. The car will see all this, even in people across the street, but it won't be able to interpret it.
Humans are still way ahead of anything close to autonomy in most situations and it will take years for anyone to get close to covering 95% of situations. Removing the steering wheel needs way more than 95% covered.