In scientific computation one blends mathematics and computation. The math is highly advanced, and it will tell you important things about your resulting code such as what kind of stability properties you should expect, and how "robust" the code is (deliberately ambiguous term, but in most cases it means: does your code work equally well no matter what parameters you throw at it?).
In the realm of physics simulations, these math questions turn out to be amazingly delicate and failing to answer them before running your code on a very large problem very often leads to unexpected (and wrong) results, if the code actually ever successfully finishes executing.
There is therefore a tremendous amount of literature where these questions are answered, all containing a meticulous amount of careful detail as well as deep insights coming from all the relevant branches: linear algebra, physics, functional analysis.
The problem however is that as we gain more understanding and knowledge the math hasn't gotten any easier. Sometimes the arguments are so sophisticated that it really leads me to doubt whether the result can be trusted at all.
This leads into something which I know very little:
A better understanding of foundations really is called for here, and a foundation which lends itself better to computation than what is currently available. Computers need to start being able to verify proofs better. Proof assistants exist today, but their use is currently limited to computer scientists because they rely on deep results in that field relating type theory to mathematics - it represents a tremendous barrier to practicing mathematicians.
I hope in the future that "reproducibility" in my field means not only submitting your code so that other people can reproduce your performance claims, but submitting proof documents to be independently verified on someone else's computer. Whenever I read the small bits about univalent foundations it sounds like they are adding magic sauce to make this possible, and so I follow it closely (but don't understand it all yet).
The interesting path I spoke of earlier: I'm now learning type theory and functional programming just so that I can make sense of this branch of computer science that I originally didn't know existed.