Edit: if you want to see how that can happen, I like to take apart weak cryptocurrencies and show what's wrong with them. Someone paid me to do a public review of a thing called Stratis a while back, and I went to town. Here's a highlight. https://twitter.com/PLT_cheater/status/1235036182284820481
I still accept commissions doing code review. It's just too much fun.
But the main issue in provable security is that you're trying to prove real world things with math, and so far we're quite bad at it. The more mathematical the thing you want to prove is, the better.
Yes the cryptography primitives should act like black boxes, no need to peak inside but when a number of these black boxes are used together to form a high level protocol allot of subtle things can go wrong for example see the history of SSL/TLS https://www.feistyduck.com/ssl-tls-and-pki-history/