But it is not a lofty ideal, it has been done before. And in the field of transportation. Trains used to crash all the time. Driverless trains don't. Why? Because fail safe systems have been added on railroads that nearly totally prevent this.
Don't take my word for it, the bar for rail safety is defined in well documented standards (namely, EN 50129, EN 50126 and EN 50128). It is 10-9 hourly probability of fatal failure. You can also look up "CBTC" on the Internet. This is the reference bar for driverless.
The real reason Tesla and friends don't want to go there is because fail-safe systems like a CBTC requires some form of collaboration with devices installed on the ground infrastructure, meaning Tesla should accept to become part of a system, and not "the" system. Obviously a paradigm change for the car industry.
How long will the public authority require to understand that they must step in to force a systemic collaboration and standards between infrastructure and cars to ensure driverless safety, I don't know, but I think it's time to call bullshit on the concept that you can have an acceptably safe driverless car system without it.