The reasoning, I think, was that humans can drive using sight and a little bit of sound, so an AI should be able to do this too.
Humans can do this because we have a very rich well-developed world model that allows us to fill in the gaps. We don't do it perfectly but we can do it decently well.
Modern AIs, or at least the ones small enough to be run on smaller machines that are economical to put in cars, don't have a rich world model like that. They're doing stimulus response backed by a database. That's going to break down at all kinds of edge cases.
The way to compensate for this is to give the car superhuman senses like LIDAR. The car is much dumber than a person but it can perceive its environment orders of magnitude better than a person, which compensates well enough that it has a chance of driving at least as well as a person.