That's actually a terrific example, because the company already had a working, field-tested manually controlled version of their product, but they wanted a computer-controlled version to relieve radiologists of all of the drudgery and opportunities for error in the manual setup. So the design spec was to "faithfully replicate the manual setup procedures in a microcontroller," which was achieved. The problem was that the UI was utter dogshit, the training materials were poor (you can't have Marketing say "hey this computer handles all the details for you!" and then expect customers to pore page-by-page through the reference manual), and most clinics weren't really enforcing or QAing the operator training anyway. And since "the computer has a handle on it," the radiologists and senior techs who were supposed to be second-checking everything basically abdicated that responsibility, driven in part I'm sure by management that expected them to now have more "free hours" for other activities that Make Line Go Up. And nobody really raised a flag that this seemed troubling until multiple children died, and at the end of the day a lot of people could say "well, it wasn't
my fault, I did what I was told to do" and sort of believe it.
If anyone doesn't think all of that is going to happen again on a massive scale as companies embrace GenAI, HAHAHAHAHAHAHA. AI will change a lot of things, but it won't fix human nature.