MAYHEM – automatically ﬁnding bugs and shell-spawning exploits in binaries [pdf] | Hacker News Reader

MAYHEM – automatically ﬁnding bugs and shell-spawning exploits in binaries [pdf]

users.ece.cmu.edu

66 pointswslh53 comments

Open articleView on HN

Debian mailing list: http://lists.debian.org/debian-devel/2013/06/msg00720.html

Reddit's reverse engineering discussion: http://www.reddit.com/r/ReverseEngineering/

The link to reddit mentioned in parent is to the RE subbredit. The link to the actual MAYHEM discussion on reddit is:

http://www.reddit.com/r/ReverseEngineering/comments/1h4vs4/r...

While I find the reported results impressive, I'm sad to see another publication about a closed-source application.

Nobody can reproduce the results or place their work on top of that. Research was not supposed to work that way (at least in my little universe :-), yet it gets published by the IEEE (but what does that mean nowadays?).

Sometimes research is closed source due to where the funding comes from, I'm working a project that's been funded by two different entities over the years - one of which won't even release the dataset we've been using - so when trying to open source my code I get stuck.

> This research was supported by a DARPA grant to CyLab at Carnegie Mellon University (N11AP20005/D11AP00262), a NSF Career grant (CNS0953751), and partial CyLab ARO support from grant DAAD19-02-1-0389 and W911NF-09-1-0273.

I don't know much about these grants, but DAPRA and NSF sound pretty public to me (read: the public pays for something only a handful of people get access to). Correct me if I'm wrong, though.

Yes, that was my point.

The paper documents their approach pretty thoroughly, and honestly, if it were me (and I'm a damn-the-torpedos-full-disclosure-open-access-everything kind of guy), I'd think twice about publishing a program that would give the kiddies something to do on a wet, wet, wet, day.

Well, the authors stated that they will publish the bugs.

Further, publishing a program like this has the potential of finally having software that is not easily exploitable (eg. by requiring every Debian software to pass that program).

Now that it is closed source, only a selected set of people will be able to benefit from it.

The paper documents their approach. You could reproduce their findings by building your own offline/online symbolic execution system.

Could I? How long would it take? Would I really get the same results or just similar results? If it doesn't work, how can I find out where the bug is (the paper, my interpretation, my implementation)?

Compare this to physics: if you drop this ball from that height under these circumstances, it will take that long. Everybody (with the right equipment) can reproduce this result quite easily. Not so in this case, which ironically could be even more reproducible than any physics experiment if it wasn't closed source.

Compare it to physics indeed:

https://en.wikipedia.org/wiki/Large_hadron_collider

You don't try to argue that LHC is closed source, do you?

Seems to be that rewriting the software is healthier, and a better analog to physics experiments.

When you reproduce that physics result, you're using a different ball in a different location and dropping it from a different structure.

If you simply took their program and re-ran it, it's like taking the exact same ball they originally used, climbing the exact same structure they used, and dropping it in the exact same spot. When you then get the same result, it doesn't tell you much, because you don't know if the result is coming from the ball, the structure, the location, or just a general universal phenomenon.

Rewriting from scratch doesn't help run more experiments, it slows things down.

The point is not to run more of them, but to run them better.

What good does it do to run identical software ten times? These computer-thingies are pretty deterministic. You're basically guaranteed to get the same result each time, but you have no idea if the result is because of the technique being presented, a quirk of the implementation, or just a bug.

Of course, access to source allows running better experiments! Writing the same algorithms from scratch is a clear waste of time! Please stop defending silliness.

I can start right away changing any one of these to find out, instead of spending time to try to write a program that may well not exist in the first place.

(In this particular case, I admit that posting 1.2k bugs hints to its existence.)

The important result of the paper isn't the specific list of bugs they found, which after all: (a) depends on the software they fed to the tool and (b) is verifiable without static analysis (just try to trigger the bug). The important result is the validation of the approach, which makes the paper more important than the tool.

I don't think they're doing anything unethical by not releasing their source code, but the state of the art sure could progress a hell of a lot faster if releasing code were the norm. Just look at all the papers that have built on KLEE since it was open-sourced.

Gee, they could have sold those for a lot of money. Well done researchers, you have a strong moral compass.

Yes, all of those vulnerabilities in non-privileged applications like 'ls' are worth millionssssss.

Yes, and then they combine them with a kernel exploit and, uh, suddenly your "unprivileged process" turns out to not be so unprivileged after all.

Resist the temptation to marginalize security bugs. They don't exist in isolation.

You're right, but so is the parent commenter; there is a popular meme that security bugs in general are worth large amounts of money, but in reality only a small subset of bugs command real money.

You could be a better ambassador of our industry by explaining the misconception instead of being condescending. And we wonder why developers think we are just arrogant jerks.

You're welcome to step in and explain, but I think it's pretty obvious why local vulnerabilities in non-setuid binaries aren't exactly "critical."

https://xkcd.com/1200/

That would only be accurate if you could trigger one of these bugs remotely.

...or if you can trick the user into opening a malicious file. An exploit in ls? Send them a zip, tarball, etc. with a maliciously named file.

IMHO anyone who contributes to the state of the art with regards to public security tooling is doing the world a favour. Even if the majority of these applications are not well privileged, and therefore of dubious value, applying the same technical knowledge elsewhere could earn them cash. I still applaud their efforts.

The value of a vulnerability is linked to its usefulness in accomplishing the goals held by those paying for the exploit. Based on that information, it should become obvious that the only bugs worth large sums of money are bugs that result in escalated privileges on the system or arbitrary code execution. Bugs that crash services also have value for DDoS attacks, but they're not as valuable as the two previously mentioned types.

While 1.2k bugs sounds like a lot, we can't make an inference about the severity of the bugs based solely on the number of them. Also keep in mind that the Debian repository contains thousands of packages. From the Debian 7.0 Wheezy release page:

"* more than 36,000 other ready-to-use software packages, built from nearly 17,500 source packages"

http://www.debian.org/News/2013/20130504

The best we can infer that given 1.2k bugs, there is some statistical probability that any number of them may be severe. Without studying the data, we can't say what that probability might be though.

Impressive!

Where does that 1.2K number come from, though? The abstract mentions only 29 (still a lot) and the results list 'only' 29, too, two of which where 0days. I admit I haven't read the whole article thoroughly.

From the looks of things, the 1.2K number is the number of bug reports they plan on filing after de-duping and removing ones that aren't pertinent.

The paper has been published a while ago. We have been improving Mayhem quite a bit since then, and we recently ran it on debian binaries. It found many crashes, that are not necessarily exploits. 1.2K is the number of bug reports we are going to submit to the Debian bug tracker.

As a long time Debian user I would like to thank you for choosing Debian as a target platform. The maintainers might not appreciate the bugs in the short term but in the long term this is great for everyone involved. Was there any reason why you chose Debian? I imagine bts made things easier for bug wrangling/submission.

Having received one of the bugs (that was not exploitable), I appreciated it in the short term. :P

I hope a free version of this can be built, it would be a nice additional QA step to run before release.

I see that it is for moreutils. I am having the hardest time using the interface for forallsecure.com. It seems that you can not search by package. When I browsed through the list I figured I would have come across moreutils around page 49/51 but I did not see it. I am tempted to download:

  http://forallsecure.com/programs/1  
  ...
  http://forallsecure.com/programs/22630

But that seems like a waste of everyone's resources. What was the bug?

For people unfamiliar with joeyh's contribution to Debian it is important to note that he is a legendary DD:

http://qa.debian.org/developer.php?login=joeyh%40debian.org

Our interface is severely lacking. The website is mostly for internal use at the moment. I'll try to add a program search tonight to make it more usable.

As a matter of fact, most programs are not linked to packages, and most packages are not present, including moreutils. You won't find it even by crawling.

No worries. I will choose helping Debian over interface improvements anyday:) Again, thanks for choosing Debian.

Actually, I have not received the one for moreutils yet.

However, we got 1 report for the Debian Installer, which I hope means only 1 problem was found.. which is nice, as the d-i code is mostly not written with this kind of robustness in mind.

I read the mailing list post as the authors willingly restricted the number of reported bugs to one to avoid spamming the BTS.

Since d-i is split into a ton of components, we know that all but 1 of them had 0 bugs found.

44 reports got delayed, including the one for moreutils. It should be in your mailbox now.

The response has been fairly positive in general, and it's great to see that some of the bugs have already been fixed!

All Mayhem developers are using Debian, and our cluster is running Debian as well. We've been using it happily for years. That's why we chose to analyze it first. I believe we will also have an impact on other distributions since bugs are getting fixed upstream.

Thanks for the great work!

Do you plan to release the tool one day? It will be a great asset to other operating systems developers as well.

I see. Thanks for the response. Also, thanks for choosing Debian and not some other closed source software.

The paper above and the post to the debian mailing list describe two different uses of the Mayhem tool. One with exploits, one without.

Misleading title-- abstract says 29 exploitable vulnerabilities, not 1200 .

Look at the reddit thread above.

keep in mind: bug != security hole

I have not finished reading the paper but after the second sentence in the abstract it seems that the authors clearly define what they consider to be a bug:

  Every bug reported by MAYHEM is accompanied by a working
  shell-spawning exploit. The working exploits ensure soundness
  and that each bug report is security critical and actionable.

Mayhem changed quite a bit since the publication of the paper. It now reports all crashes, and therefore bugs are not necessarily security critical.

Reply on news.ycombinator.com