There is the argument that they are already safer than human drivers though and it'd be morally objectionable to stop. Not saying I'm completely on that side, but it is a valid utilitarian argument.
The burden of proof is on the person making the claim of safety. For Autopilot, the argument was made comparing known AP deaths per total AP miles vs auto deaths vs. driving miles of the entire US fleet... when the former almost certainly didn't include a share of the deaths and was driven under less demanding conditions (highway miles).
Ergo there's a predictable and preventable source of accidents that must be suppressed.
a) [Broadly speaking], cars [from traditional big autos] with ADAS features are objectively safer, and,
b) Tesla Autopilot is [in traditional sense] an ADAS system, which,
c) [implies this set of facts alone make Tesla cars and implementations statistically safer] is cool, huh? Would be silly if you didn’t agree with me.
I don’t call it clever.
The utilitarian argument for driverless cars needs to be that driverless cars are better at handling the edge cases than humans, because that's when accidents happen.
The average situation is maybe 0.99 accident-free (made up number for illustration). You improve on that by being 0.995 accident-free, which is better.
I agree that “safer than some human drivers in some conditions” is insufficient.
Maybe I'm confusing the idea of average and mode, so if I am an example of what you have in mind for an average situation would help.