AI Mistakes Are Different from Human Mistakes

Yeah, and it's not just an LLM thing either. There's a thought-experiment I like to pose for self-driving vehicles: Suppose they statistically are half as dangerous as manual driving, but this record is tainted by unpredictable and inexplicable Scary Stuff like driving up on the sidewalk to attack a pedestrian or going sideways off a bridge.

Would we be satisfied with that tradeoff? Should we be satisfied? At what point is a X% error case we can understand and characterize equivalent to a Y% error case where we can do neither? (Mitigation is another aspect.)