Sincere question, as a non-expert trying to situate your comment: are you a mathematician with experience in proofs?
My point was rather more motivated by having seen so many weird ways for machines to fail/not work as expected that I wonder how to deal with that if the output were to be incomprehensible to humans.