We have fallen prey to the idea that the magic box says you did it, therefore you did it. There is no examination of the "thought process" of the magic box. That's all protected as proprietary. In a setting where you have an arresting officer you can question the officer in court you can question his thought process his procedure you can understand why he made the decision he made and challenge all of those base assumptions. Because if there was some flaw in his initial reason for the suspicion there was no basis of charging you with a crime.
With the magic box we are allowed to do none of this. It's assumed that the software in the magic box apparently has zero bugs or those bugs would become readily apparent when you test a known set of inputs against a expected set of outputs. Anyone who's got any kind of software engineering experience knows that if you have a test of expected inputs with expected outputs you will make your software match that. without the full disclosure of the code that's running on the device for examination there is no effective cross-examination of the one witness the state has against you for violation of a crime. That's the only way we can examine the thought process of the magic box itself.
Until courts can be convinced to throw out cases when the source code is not produced this kind of thing will happen. The assumption that the magic box is right and will always be right. Because you can't even get the magic box to perform your own testing you are never allowed to examine the evidence against you.