How Heartbleed could've been found | Hacker News Reader

How Heartbleed could've been found

blog.hboeck.de

144 pointshannob40 comments

Hey all,

I'm the author of AFL. I think this is pretty cool, but also would like to ask you all to hold on to your hats =) Here's the short response I posted to the oss-security mailing list:

http://www.openwall.com/lists/oss-security/2015/04/07/8

Hanno (the blog author) uses american fuzzer lop (AFL) and other fuzzers and has discovered (and reported with test cases) an extraordinary number of bugs: http://lcamtuf.coredump.cx/afl/#bugs

He is doing great work to make the Internet safer.

As soon as I started reading how it worked it reminded me of Microsoft SAGE [0] It also uses guided execution to find constraints and new bugs.

[0] http://research.microsoft.com/en-us/um/people/pg/public_psfi...

Google has a big fuzz-farm and Project Zero looking for this type of thing and even they did not find Heartbleed years ago. They are nabbing tons of bugs but there are many that are simply buried.

This seems to me a bit like when you do a maze starting from the finish and it is, for whatever reason, trivial to go from one end to the other.

It is neat that it is 2015 and fuzzers are cool again, though.

Not many people did serious analysis of OpenSSL just because the code was very difficult to debug.

Isn't the whole point of fuzzing that you don't really need to understand the code to find flaws in it?

No. A fuzzer abstracts away some of the need for intricate, function by function analysis, but you really want to know what the source code is doing to be successful.

Fuzzing "blind" will work...but you will miss a lot without more instrumentation than that.

Actually Google did find Heartbleed (along with Codenomicon, who discovered it independently). [1]

[1] http://en.wikipedia.org/wiki/Heartbleed#Discovery

And Codenomicon apparently found it by fuzzing [0], though I haven't seen any details.

[0] http://www.codenomicon.com/products/defensics/

I rhink there's something fishy with it: you overcome several hurdles to fuzz OpenSSL and then - miraculously - you come up with Heartbleed.

And the best of it: only Heartbleed. nothing else. Nothing more.

Looks like it really went that way, but what are the odds?

The only slightly suspicious thing that he did was that he immediately fuzzed only the first packet and no others. Although, being the first packet, it's a logical place to start. So I think the Heartbleed bug would indeed have been reasonably easy to find with this technology.

Actually, also the fact that he selected this particular packet exchange. This is a bit more suspicious, since there are thousands of other possible packet sequences. But again, the initial TLS handshake would be one of the first ones to check, I think.

He mentions in the article that he also found two other bugs but that they were previously discovered and reported by Brian Carpenter.

By fuzzing various file inputs I recently found two issues in OpenSSL, but both had been found by Brian Carpenter before, who at the same time was also fuzzing OpenSSL.

afl-fuzz is pretty powerful, no doubt.

But beyond dynamic analysis, someone wrote a static analysis feature to find heartbleed as well: https://github.com/awruef/find-heartbleed

happy to see that someone else saw that! I mostly wrote that post / code as a tutorial on how to write checkers in a symbolic infrastructure, I think it was a little successful. I've been working on making checkers like that better, but that work is depressing because people like the author of AFL spend a lot of time telling me (indirectly) that it will never work, never scale, and never matter.

How does your checker work?

the parent comment has a link that includes code and a blog post. clang-analyzer uses "symbolic execution". the work I've been doing in extension also uses symbolic execution.

I read it and thought it was super cool.

I was inspired to write my own checker to demonstrate how easy it was to my software team.

And when I get a chance I'll try to contribute some general-purpose checkers to clang.

I dunno if it will work, but IMO it matters. If for no other reason than inspiring other folks!

Yay :)

I fear these kinds of tools will be (are?) used as an argument to keep using C/C++. "Just adhere to all these standards and use all these code analysis tools" is a really bad way of handling bugs when I guarantee you it won't be used by those that need it most.

Whether they are used as a justification to use C/C++ for new projects (which isn't an argument I've seen), it's still better to have some solid tools available to find security issues in the existing C/C++ code that's already out there.

It's really dangerous to think that switching to another language will fix all your security problems. You can have safe memory, but that's about it. Serious exploits like "goto fail" can exist in any language.

No. The state of the art has advanced well past Java. And besides the kinds of bugs that a good type system can fix, there's also bugs borne from usability problems like switch fall-throughs that are trivial to fix with a better syntax.

You can have safe memory, but that's about it.

That's not nothing. How many remotely exploitable bugs of massive severity have come from unsafe memory?

Sure, there's much more work to do beyond memory safety, but there isn't much excuse for that one anymore.

I think we need to start thinking of programming as a true craft. Like watch making. Not something you learn within 2 years just to become a productive app-maker to mass produce more useless apps. But something you grow into by gaining a full understanding about how the computer works. And this requires you to go trough both assembler and C. And by raising the notion of coding as a true craft that takes time and require precision, we will gain more secure applications. Not by hiding the inherent mechanics of the machine treating programmers like kids who need training wheels.

Except we're not making the same kind of thing over and over. Programming's not a craft. It's an abnormally non-routine form of engineering. By all means, teach the college students about doping, logic circuits and binary assembly. But skip C. Skip C++. It's not cool, it's not hardcore, it's just sad.

Bug detection is a problem that is easy to verify and hard to solve.

I believe Heartbleed was found via both manual analysis and fuzzing to begin with. Good analysis here: http://www.dwheeler.com/essays/heartbleed.html

Since I had something to do with the exploit mitigation chatter, I feel I should note that there's something of difference between fuzzing and exploit mitigation. I'll also happily concede that Address Sanitizer could be a more effective exploit mitigator than omalloc.

To make random() deterministic you can also just seed it with a predetermined number. Not sure if it is any better in this case, but it helps when writing unittests and similar.

The third part of my post here [1] is that progress in security will more likely come from cheap tools than expensive humans. Pardon the self-reference of course. :)

[1] http://clipperhouse.com/2015/04/04/liquidity-open-source-and...

... and where do the tools come from?

Tools are either made by expensive humans, or forged by the Dark Lord Sauron in the fires of Mount Doom.

There's also the matter of who will run those tools. It's not like fuzzers output exploit.sh or something.

Automation and tooling are great, but they aren't replacements for human ingenuity.

Oh no doubt, but tools scale well. The tools allow us to partially but cheaply replicate the brains of the expensive smart people.

Progress in maintaining huge codebases in general. OpenSSL is ridiculously large for too few maintainers, but you should be so lucky as to have a codebase of maintainable size for the coders you have. My favourite example remains LibreOffice, where the coders are very good but there are too few of them - so they hit the automated tools, static checkers, clang plugins etc really hard.

I'm a big fan of fuzzers and believe the HeartBleed bug could have been found by one. Finding an already known bug would require a very careful researcher (to avoid providing either clues or bias). In any case, thanks for all the work you're doing Hanno!

So wasn't Heartbleed caused by a single overly large value coming directly from the attacker?

So, yeah, fuzzing could find that but it seems like wild overkill. The normal sort of manual range checking one does when implementing a protocol would of worked too...

I like to use scan-build to find, report, and fix bugs in open source C/C++ code! Another tool in the toolbox for me, nice read!

Coverity has been improved so that it now detects Heartbleed and similar defects via static analysis:

http://security.coverity.com/blog/2014/Apr/on-detecting-hear...

Reply on news.ycombinator.com