Frame and chassis have never been safer and manufacturers continue to improve. Many (most) new cars have automatic emergency braking that continue to improve. New cars seem to have an ever increasing number of air bags to protect passengers.
All these things are happening at the same time that self-driving is taking place. Tesla FWIW is pretty good at all the above despite their focus on self driving as well.
I agree it is just as ridiculous to say we need level 5 to be make something useful. Will it be decades before we have cars without steering wheels? Sure, maybe even longer. But what exists is already pretty great in most environments and only getting better. (IE, crawling along in a traffic jam at ~15mph is something I would really love to never do again and it seems self driving systems can handle this with aplomb these days)