> Blindly trusting these models without any verification will put innocent people in prison.
I don't think anybody is suggesting this. But if the models can gleam information/insights that humans can't, that's still valuable, even if it's wrong some percentage of the time.