People take pills from known criminals, with high risk of fentanyl OD, just for fun.
Yes, I think people would indeed take pills prescribed by AI, just make it a robot wearing a lab coat.
Also pilots! I mean, pilots kill themselves and a planeload of people more than you think. Of course people would take black box ai that works.