Damn, don't you think "we" as a society should eliminate the negative effects/activities? since aside of not having any utility it may bring negative outcomes..
To me, drinking(alcohol) is a pointless activity, with literally no upside. It costs money, doesn't feel good, I don't see the social aspect of it, and the danger it poses to people is well documented - it's a "negative" for society in my book. But, at the same time, I am fully happy to acknowledge that there are people who enjoy it, who like to party, and I would never say that drinking should be banned.
I know a few people who treat driving like I treat drinking - they see it as boring, without any benefit, with negative consequences well documented and they clearly think that society would be better off without it. They think that the day that manual drivers are banned can't come soon enough.
As mentioned above - we are not robots. Not everyone enjoys the same things. And that's great.
There's only a risk of killing people unless you're ISIS or a psychopath. Do we ban everything because of a few assholes? Yes, we do a lot of the time, and that's pretty sad.
Also, just to address your last point - why would manual driving cause increase in killings? In the worst possible case, it would stay at the same level as now,no? In fact, it's hard to argue that it won't get better, because autonomous cars will avoid dangerous situations caused by manual drivers better than humans would.
In other words - in increasingly autonomous environment, the risk factor of manual drivers will go down, not up.