Only if the tests are comprehensive enough-- very few piece of cryptographic software have tests anywhere near as comprehensive as even something like GMP.

For some bugs the probability of triggering them with 'random' inputs ends up being a something like 1:2^64 or worse... so there unit tests aren't necessarily helpful.

In libsecp256k1 we include tests generated from specially constructed distributions that are more likely to generate errors (which, for example, found bugs when run against openssl that had 1:2^256 level odds of being detected by uniform random tests), as well as test generated using absurd amounts of whitebox fuzzing and symbolic execution-- but even these levels of tests aren't guarantees. We hope, however, that the set of bugs that might get past this testing is largely orthogonal to the issues that get past review or our use of formal methods.

Tests also lose their power when they don't get run on every compiler+architecture+configuration. Few pieces of software do runtime self-tests and few users (or even distributions) actually run the test suite on their particular system. When they are cross-compiling to an embedded system, it might not even be realistically possible to do so.

Sadly even formal proofs are not guarantees for security in practice, especially when you're talking about constructs more complicated than field arithmetic, because the proven properties seldom map precisely to our intuitions about what constitutes correct behavior. This is especially true for very open ended code where it gets applied to applications which were entirely unforeseen by the developer.

Right now I believe we've have a testing methodology in place for every class of serious issue we've ever had in development that would have a fighting chance of catching it even if we didn't know about the specific issue. But then again, I would have also said the same thing before the last class of errors we found that our process was unlikely to detect. :)

Then again, this doesn't really sound all that different from the kinds of security assumptions normally made in cryptography: Why is the discrete log problem (in certain groups) hard? Because no one knows of an effective attack.

The difference is that many (most?) pieces of open source software receive little to effectively no peer review.

Yes, for known attacks. Unknown attacks are harder.

https://github.com/google/wycheproof is a Google's effort to test for known attacks. If your crypto library does not mention Project Wycheproof or known attacks listed there, ignore it.

Unit-Tests are unlikely to catch cornercase-bugs. (i.e. a bug that happens in 1 out of 1^128 cases.)

Fuzzing works quite well though. (Even for the cornercase bugs that fuzzing can't find by chance because they're too rare, but modern coverage-guided fuzzing works surprisingly well for that case.)

Maybe a few but for most of them it's more about knowing about the problem. It's hard to write unit tests to find unknown behavior.

slide 2: CVE-2018-0733 was in the wild for 2 years slide 5: CVE-2017-3738 was in the wild for 4 years