It's not about trusting the math, it's about understanding it. And not just the math, but also the protocols that use the math (the math may be secure after all, but the protocol not, c.f. the multiple vulnerabilities in SSL in recent times).
And of course, the fact that occasionally some implementations of SSL have been found insecure is in no way an argument to not use it (at least until something more secure is available).