I get the point that they are in different magnitudes of unknown but the analogy is still pretty good when it comes to the median programmer, who has no idea what goes on within either one. And if you argue that compilers are ultimately deterministic, that same argument technically holds for an LLM as well.

The biggest difference to me is that we have humans that claim they can explain why compilers work the way they do. But I might as well trust someone who says the same about LLMs, because honestly I have no way to verify if they speak the truth. So I am already offloading a lot of burden of proof about the systems I work on to others. And why does this ”other” need to be a human.