Part of the problem here is that it is very hard to audit something OpenSSL's size. It has 450,000 lines of code. Are you going to skim, let alone audit, a mathematics-heavy codebase of that size? No.

Some of that is OpenSSL's fault, but a lot of it is inherent in implementing a 200-page (before extensions!) standard. GnuTLS is 170 KLOC, CyaSSL is 205 KLOC, and even PolarSSL, commonly thought of as the gold standard in trimming the fat, is 55 KLOC.

If you want to reduce the level of paranoia, you would have to pair down the size and scope of TLS by an order of magnitude. For the record, I support that kind of thing, but you would have to cut a lot of things people actually use, not to mention breaking backwards compatibility.