Honestly that's worse than I thought. I work in the field, particularly in relation to accountable AI, and it's not OK to have models that tell you whether to check on people to make sure they're not dying unless there is also a human checking every case, which I hope is what's going on. How would you like to be different than the training data and deemed "no risk, 100% confidence" when you actually have a life threatening problem?