If you have to sell self driving cars as "more people will die but some people will have more free time" it's not going to be as easy as if they are unarguably safer for everyone.
I think you are begging the question - are cars more dangerous than horses? Just from reading random history, it seems like a lot of famous people were killed by riding accidents. And even today when the vast majority of people don't ride horses, I bet you can name a famous person who had a terrible accident.