We basically ended up creating a tool+language spec that would let us define the mapping of input to output sequences. We wrote it in such a way where we sat with scientists and pretty much mapped all the possible cases that they could think of for valid inputs and what the code should produce.
Then this tool would basically write automated tests for us, in such a way as to not only test correct behavior, but also do combinations of inputs out of order, fuzz testing, and so on. We ended up making it also check memory state, to ensure that there was no memory leaks, and analyze the memory space for required data or data that should not be there.
In the end, whenever someone was developing anything for this software, they would basically just run the tests, and it would be very good at catching possible errors, mostly on the negative side (i.e for a fuzzed input, it would result in an output that should be an error).
We could have done the same thing with a typed language, but it would have to be very strict typing to the point of something like CoQ, and it would have taken us probably the same amount of time to write that.