The way I ask it in public situations with scientists is this: If you had an AI that could output cures for diseases (IE, drugs that pass phase III clinical trial, get approved by the FDA, and are highly effective), but it couldn't explain how it worked, would you use it?

Opinions are mixed. Some folks will say that it's morally imperative to cure people even if we don't understand the specific or general principles. Other folks will insist that it's a terrible idea to hand over the comprehension of medical treatments to LLMs, because in the long term it will leave us helpless and dependent.

Ive been thinking about this a a lot, like now if LLMs can just try a bunch of things, you can just try them all now. Are there some applications where we can just brute force and test at scale and no longer need to understand? Where understanding is now in essence that it “works” and passes our acceptance tests.