You are correct that this is indeed a mostly unsolved problem. In chapter 15 of "The Mythical Man-Month", Fred Brooks called for all program documentation to not only for how to use a program, but also for how to modify a program [1] and, relevant to this discussion, for how to believe a program. This was before automated tests and CI/CD were a thing, so he advocated for shipping testcases with the program that the user could review and execute at any time. It's now 50 years later, and this is one of the many lessons in that book that we've collectively not picked up on enough.

[1] Side-note: This was written at a time when selling software as a standalone product was not really a thing, so everything was open-source and the "how to modify" part was more about how to read and understand the code, e.g. architecture diagrams.