That's ok, but based on your example, how would a human apply an algorithm that it can't determine why it worked?

We're not talking about "Not Hotdog" here. Application of a model in a real-world at-scale scenario is a lot more than running inference and walking away.

At a bank credit decisions are evaluated by humans, frequently and often. These reviews are conducted in the forms of sampling audits, control processes, and other scenarios that involve internal bank employees and external regulators. In each case, humans will inspect the details of what occurred. This would be impossible with any type black-box model (SSL, deep NN, etc.).