We haven't been improving on drivers, though. Twenty years ago, humans as a group were as bad as they are today and as they will be in another twenty years. Or thirty or fifty. You can add seatbelts and rear parking cameras, and you can legislate against texting while driving, but you can't legislate away tired or distracted drivers, or just plain unskilled drivers. As a group, the only way we improve is through demographic shift. But autonomous cars get better, as a whole, and consistently, and rapidly.
Maybe autonomous cars will be at the 50th percentile in five years, but they're only as good as the bottom 10% of drivers, well, we'd still be better off with those 10% of drivers switching--which would lead to infrastructure to improve it. (I think more importantly, social understanding/recognition of them. The hard problems presented in Dr. Leonard's video posted on this thread aren't peds stepping in front of them, but cops gesturing and unprotected left turns across traffic where you need to make eye contact.)
Finding the bottom 10% sounds hard, because, yes, a lot of bad drivers think they're good drivers, but I think the majority of us know we suck. For many, their insurance companies know too.