> Koopman, who has a long career working on AV safety, faulted the data-driven culture of machine learning that leads tech companies to contemplate hazards only after they’ve encountered them, rather than before.
I haven't yet figured out how to effectively and efficiently communicate this mindset shift to folks educated primarily in ML culture. I am not sure I ever will. The closest I've come to an elevator pitch of the mindset shift is something like: "when human lives on the line and you're taking an absolutely massive number of samples IRL, doesn't it make sense to stop thinking in terms of analysis/probability and start thinking in terms of topology/nondeterminism? Ie, when you sample A LOT, unlikely shit happens. Manageable risks if you're selling ads I suppose. But not so acceptable if you're deciding whether a giant piece of unforgiving steel saw a bollard or your daughter."
[^1]: I read the quote-block in your post and immediately thought "Koopman". Then read the article and, sure enough, they're quoting Koopman. What a tragedy. The message was not only out there, but out there so loudly that the vague shape of the warning has a particular person's name attached to it in my mind. Yet, here we are.