The basic problem is that machine learning systems don't really understand what they're doing. So most of the time, they do the right thing, and when they do a totally wrong thing, they don't notice. Works fine for advertising, not so much for self-driving.

Look at those systems that generate images or text from a prompt. Usually the results are good, and sometimes they are totally bogus.

As I point out occasionally, the big hole in AI is "common sense", defined as not screwing up big-time in the next 30 seconds. Until that gets solved, AI systems can't be trusted very far.

The control theory people are trying to fix this, so they can use ML to build control systems with safe behavior. The math is really tough. Way beyond me. See IEEE Transactions on Control Systems Technology to follow this. People are trying to make control theory math and ML math play together. Control theory usually has continuity - if it does the same right thing at 0.21 and 0.22, you can be confident it will do the same right thing at 0.215. ML systems do not have that property. Which is why we see those image recognition demos where some minor change in the background noise totally changes the result.