Using Heartbleed as a starting point
antirez.com
antirez.com
Yes, bug bounties on their own aren't a sufficient solution: more systematic, less ad hoc approaches are needed too, no doubt. But nonetheless I think it makes sense to double down on bug bounties, for two reasons. First, bug bounties are an approach that seems to be fairly successful, though the IBB money probably wasn't part of Neel Mehta's motivation to start inspecting OpenSSL at all. There may be better, though more ambitious, approaches, but if there's one thing that seems evident from the recent SSL/TLS bug fiasco it's that making the perfect, or even the great, the enemy of the pretty good is counterproductive here. Second, probably nothing will motivate the maintainers of OpenSSL etc. to actually implement (and keep implementing) systematic, proactive changes better than the constant, realistic threat of being pantsed again by bounty-hunting outsiders. (Remember: all technical problems are social problems in disguise, just as all social problems are technical problems in disguise. ;) ) Or failing that, fairly frequent bounty-funded bug finds would at least serve as a warning to the rest of us that the maintainers aren't doing very well.
Funding the bug bounty is one problem, but administrating and adjudicating it is quite another task.
Also at what level is the bug at the standardization level?
JT Olds asks: "Why did Go implement its own SSL library? I've scanned the list and the website and I might have missed a design doc somewhere, but as far as I can tell I can't find any motivational reason for why Go implemented its own SSL library."
agl responds: "Have you seen the OpenSSL code? I do, indeed, work with it a lot and it doesn't exactly fill one with positive feelings. NSS has a different set of issues. (I'm not familiar with alternatives like Polar or GnuTLS.)"
False. Glibc uses ptmalloc:
Which is under the MIT license.
I couldn't find any really good, recent benchmarks of different allocators, but it does seem to be the case that ptmalloc is not as fast as tcmalloc. However, i suspect that at the time at which it was adopted, ptmalloc was one of the fastest options available.
jemalloc may very well be a good thing for Linux in general but I don't think it would have changed this.
About the embedded systems, usually you don't try to cover everything, it is often a more valid a approach to have specialized libraries for very resource constrained systems. In the case of OpenSSL I guess the problem is that the implementation was hard enough to force OpenSSL to blend instead. This is where to be able to say "no" is a good idea.
To quote from your blog post:
"This is a sign that still performances, even in security critical code, is regarded with too much respect over safety. In this specific instance, it must be admitted that probably when the OpenSSL developers wrapped malloc, they never though of security implications by doing so. However the fact that they cared about a low-level detail like the allocation functions in some system is a sign of deep concerns about performances, while they should be more deeply concerned about the correctness / safety of the system."
As you stated in this bit of your post this is a sign that the OpenSSL team have their priorities backward. In other words they are NOT fit to run a security critical project. Period.
Seems easy to dismiss them and say their priorities are backwards, fundamentally at the end of the day, a huge majority of the users will simply pick the benchmark winner and use it. It a free project and their priorities are what they are, what are the priorities of the legions that used it instead of writing their own?
Of course you should try to optimize as much as is sensible, but if you don't have any process in place (unit tests, integration tests, fuzz testing, formal methods, timing attacks, whatever you can throw at it) that lend some assurance that Implementation#1 (optimized) actually does exactly the same as Implementation#0 (less or unoptimized) under all circumstances, then you should be as conservative as humanly possible... and choose to NOT optimize unless/before you have those processes in place.
(I'll happily admit that this is somewhat armchairy -- I'm glad I don't work in a particularly security sensitive area. I'm not sure I could handle all the obligations/pressures myself.)
(edit: to reduce confusion, I just want to say that Firefox uses jemalloc, otherwise my comment makes no sense. It's however disabled if you compile Firefox for use with valgrind).
[1]: https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Find-a...
In contrast, heap protection / exploit mitigation is intended to protect your software in the field from the bugs you did not know about until after the software shipped. Correct software would not need ASLR, stack canaries, NX or heap protection. But these things are widely accepted as required these days.
Desirable features are mainly heap overflow protection and double free protection (see [1] for a list of what ptmalloc does). The situation is not entirely simple because the specific protections you want (say in terms of heap metadata protection) depend on the internal design of the allocator and there are a lot of devils in the details.
There is some information around on jemalloc exploitation [2] [3], but I do not know if that information is current. I'm not aware of any jemalloc hardening which has happened since then. If you could provide any information in that area I'd be interested to know about it.
[1] https://wiki.ubuntu.com/Security/Features#Heap_Protector
Why not counterbalance the "crypto software is hard, don't touch it" idea with some simple tasks, like: check all allocs in openssl for bounds checks? "mark" missing ones? "mark" fishy ones etc.
Can a software security audit be crowd-sourced and be meaningful?
A good exemple is Daniel J. Bernstein. The man doesn't even trust the libc and wrote its own layer for all of his software (djbdns, qmail, ...). (source : http://cr.yp.to/qmail/guarantee.html #7)
I think there is a good balance between using third party library (possibly untrusted) and plain C. Kind of an analogy to centralized network vs decentralized.
I can't see how the DJB example supports the claim that "damages are everywhere". It's trivial to say that there could conceivably be a "balance" here (what human activity doesn't include the concept of balance?), but a cursory glance suggests that OpenSSL code is currently far from that point.
On the surface, a tricky problem.
Earlier, Theo de Raadt's response on a mailing list was linked. Theo being himself was to be expected, but that kind of talk doesn't help anyone.
We need to take pragmatic actions that fix problems, not exchange recriminations and continue to live with the problems.
In other words, we as a field sometimes use this algorithm:
1) Identify the problem
2) Repeat
On the other hand, if the aerospace industry was like the programming industry, we'd now be flying 10000000000 faster planes with warp drives, that take us to A-Centauri in an hour or so. And everybody would have one such plane, folded in his pocket.
If you're going to carry the analogy to that ridiculous corner then a black-hat hacker with a heartbleed equivalent blows up Earth in an antimatter explosion.
Plus, I don't recall computers having killed many people. Planes do. If planes were like computers, we'd just reboot them and carry on.
You brought up the rotten analogy. I'm talking about what humans comparatively actually do or don't do in different fields, not making an analogy. (Please look "analogy" up.) Historically, aerospace is very good about taking concrete actions that actually reduce accident rates. Historically, computer programming is pretty rotten at this. Our algorithm is literally:
1) Identify the problem
2) Repeat
It surprises me that something as important as OpenSSL doesn't have hundreds of coders studying every check in to find potential bugs, in particular obvious C errors.
(Yes, you will have to create a build with OpenSSL's custom allocator disabled to get a full run. No, it isn't hard.)
Granted, security software should be tested against invalid requests.
According to Ted Unangst, all you have to do is turn off the custom allocator in openssl to immediately hit a use-after-free bug.
He writes:
"This bug would have been utterly trivial to detect when introduced had the OpenSSL developers bothered testing with a normal malloc (not even a security focused malloc, just one that frees memory every now and again). Instead, it lay dormant for years until I went looking for a way to disable their Heartbleed accelerating custom allocator."
http://www.tedunangst.com/flak/post/analysis-of-openssl-free...
I am not claiming that it would be difficult to find invalid memory usage in OpenSSL, rather I am claiming that the particular invalid usage that led to heartbleed would be difficult to find, becuase it does not occur under normal circumstances.
OpenSSL is ubiquitous and runs everywhere including phones and low powered VPSs that everyone is using. If OpenSSL burned RAM and CPU cycles for the sake of correctness, alternatives would appear. The hard part about developing a library like OpenSSL is it has to be fast AND secure.