Sidestepping comedy for a bit, there are a lot of inscrutable systems that we connect to 'things that matter' all the time. The financial systems themselves are pretty damn inscrutable. Corporations are very often inscrutable.
By contrast, there is nothing remotely resembling a human mind anywhere in machine learning, and the failure modes are often, by our standards, insane (like thinking a picture of random noise is a cat). That creates a whole new level of danger when connecting to "things that matter".
And to the extent we already connect inscrutable systems to things that matter, we should be trying to make that problem better, not worse. "When you're in a hole, the first thing to do is stop digging"
Technologists tend to think that tech is value-neutral, and will therefore give good outcomes.