> "learn programming well enough and you'll know if some piece of code works as expected without running it."

That is 100% true.

Obviously the code needs to be self-contained (not calling into other unknown code) but so do mathematical proofs.

That's what seperates them both, it's not about immediate formal checking from a compiler here, but rather if our proof is being justified adequately according with the rules of deduction and reasoning, it's a more intuitive approach in math, although you do end up knowing whether something is right or wrong, akin to programs.