It’s formalized statistics translated to code by community.
Regex and grep are just as “black box magic” to those who read a blog post on how to scan some text files.
ML is statistics on arrays, whereas Unix tools are stats for a Unix system.
It’s the same abstraction at its core, new implementation.
if (personInFrontOfCar) { halt(); }
This is what people mean when they say it is a "black box." There is no source code to edit, only training data that outputs an as-is model. It's similar to getting binary blobs, for eg Linux drivers.
In contrast, I could fork regex if I wanted and add an `if` statement in the middle of it, and re-compile it. It is pure source code. Same with grep.
`if personInFrontOfCar:`
(using object detection, masking, 3D pose estimation, et/or cetera)
And actually defined policies on how to handle certain situations given that inference are possible, in the case specifically of self driving vehicles.
My assumptions come from when I'd asked someone at NeurIPS in 2019 about what reinforcement learning methods they use as they said "I don't think any self driving companies are using reinforcement learning, it is too risky". I don't mean to imply this is hard to find information either, reading a few papers would be all it takes to clear up the degree to which most self driving policies are "controllable" in the way you describe, or at least to what degree.
My main point is that I think it is the inferring of the environment (current state) rather than the chosen policy at each time step which is more of an error prone black box.