Pwn2Own winner tells Apple, Microsoft to find their own bugs | Hacker News Reader

Pwn2Own winner tells Apple, Microsoft to find their own bugs

computerworld.com

30 pointsSandman20 comments

I've done some of work in the area. Most of the companies that use, or even require fuzzing in their software development procedures, have a certain problem: the same people who write the parsers write the fuzzers. When they write these fuzzers, they make some too-strong assumptions on the formats of the data involved.

Writing smarter fuzzers may be needed to fully exercise certain file formats, say where certain blocks of data need to match a CRC elsewhere. But smart fuzzers fail where the authors fail to account for edge conditions. Having both in your toolset is invaluable.

Regardless, it's sad for companies like Apple that they're not doing any of this.

Dumb fuzzers find things in Microsoft code all the time, and Microsoft spends huge on external teams to write fuzzers.

So what's your analysis of why the external teams are ineffective?

They're not.

OK, so Microsoft finds lots of bugs by this technique (according to you), but then this guy says that he finds still more bugs, with (according to him) pretty primitive methods. And then the guy says,

'"I found bugs, lots of bugs. That was both surprising and disappointing." And it also made him ask why vendors like Microsoft, Apple and Adobe, which have teams of security engineers and scores of machines running fuzzers looking for flaws, hadn't found these bugs long ago.'

So given your assertion that these teams are very effective at what they do, why do you think this guy is very easily (according to him) able to find additional bugs?

There's a mismatch between what you're saying and the gist of this article. I'm not saying you're wrong -- far from it. I'm looking to you for a resolution to the discrepancy.

This guy is in the security business. If you think your software is secure, he's out of work. If he creates the impression that he didn't even have to try that hard to find these vulnerabilities, then you get paranoid and want to hire him.

To be clear, I'm not saying the vulnerabilities weren't real, but to buy his story about them being "easy" to find, well...

I tend towards the "Charlie is just better than most of us" view of the world, too.

I tried to bring out 2 points in my comment:

1) The people who write the parsers should not write the fuzzers 2) Using fuzzers based upon more rigorous standards of the relevant file formats may not be optimal. There is an advantage to dumb fuzzers; with defense in depth one would use both.

I think that when you contract a senior consultant from iSec Partners or Leviathan to test, I don't know LiveMeeting?, you're getting a high-quality "deep" fuzzer and you're getting the dumb fuzzer too. They are getting both. They're spending many tens of thousands of dollars per release cycle to get people to write and run these tools. They evaluate projects based on whether they find things. They find things. They keep the tools and the run them on future releases.

The point I'm making is, even if you do an amazing job fuzzing your software, you will clearly still miss things.

"He went into the project figuring that he wouldn't find any vulnerabilities with the dumb fuzzer. "But I found bugs, lots of bugs. That was both surprising and disappointing." And it also made him ask why vendors like Microsoft, Apple and Adobe, which have teams of security engineers and scores of machines running fuzzers looking for flaws, hadn't found these bugs long ago."

Or, maybe the companies have already found those bugs and more, but focus their efforts on bugs that surface in the wild and don't spend resources on the others. If they fix all the bugs, then they spend money on some bugs that would never have surfaced. Just a guess.

That seems to make sense from the business point of view (well - not every business works like that, but I'm sure there are people out there who would agree). Unfortunately. Maybe internet needs more hackers these days? Leaving bugs unresolved should never equal saving money.

Waiting to fix bugs until they become an issue is smart, if a bit ruthless.

Waiting to fix security flaws until they become public is negligent.

From firsthand experience with at least one of the companies implicated in the recent stories: they aren't sitting on things. What Charlie Miller is finding is news to them.

At one company I worked for, we introduced a fuzzer into the testing of one of our products. Due to a longer than normal release cycle, we were able to collect data on bugs customers found vs bugs the fuzzer found.

A lot of bugs the fuzzer found were not found by customers, but a lot of the bugs found by the customers were found by the fuzzer. Furthermore the bugs encountered by the fuzzer identified modules that tended to be buggy and allowed even more bugs to be found by inspection in them.

I guess it comes down to how much you value you place on fixing bugs before customers run into them. I certainly feel like in the case of security issues, the value is very high, especially since this is a class of bugs that attackers will specifically search for!

This is somewhat equivalent to WikiLeaks' release strategy: tell the organization that you have evidence of XYZ problem, but don't describe the exact nature and scope of the evidence, then pressure them to come clean and fix the problem themselves; hopefully this leads to more comprehensive "cleanup" efforts that have a larger positive long-term effect.

How relevant is the WikiLeaks strategy in the field of security?

It depends on the vendor. There are vendors for which the "announce and hold back" strategy will probably work: they're the ones who market based on security. It is a Big Deal if someone has an IIS remote, or a reliable Flash, or Apache/WebSphere.

We'll see how big a deal it is for Google and Apple. Marketing based on security means more than just talking about security on your web page. It means making a business out of people who buy based on product security. I like my Mac, and I like Apple, but I have no illusions about the resilience of OS X.

Anyone able to give more detail? Maybe not on Millers specifics but how does inserting random data into a program become an exploitable vulnerability?

http://en.wikipedia.org/wiki/Fuzz_testing

He sounds naive and his logic is quite the cliche. Surprised he didn't close it with a line from the hacker's manifesto.

I'm not sure what it is with some people acting like the smartest guy in the room while refusing to share their knowledge for the betterment of society. Entrepreneurs benefit tremendously from mentors and advice from those who've made it. I'd be surprised if he didn't learn a vast portion of his trade from the openness of information. I guess he feels empowered w/ his :15 minutes and this is how he leverages it? Weak^10.

+++the pxlpshr+++

I didn't get that sense from the article - he's not refusing to release the info for his own benefit, or to be a dick; he's ostensibly doing it because he feels like this just encourages the "we'll fix stuff when other people find it in the wild" mentality, which doesn't help make software more secure in the long run.

He wants software producers to actively try to break their software so that his ostensibly simplistic tools can't find wide open holes like this.

I'm sure some egotism rolls into it somewhere, but I don't think it's altogether a bad stance... you shouldn't be waiting for your users to tell you where your security holes are.

you shouldn't be waiting for your users to tell you where your security holes are.

while I agree with that ideology, in reality you just can't assume that to be the case when you have multiple people touching the code, and features added regularly that may inadvertently introduce holes. what software has ever been 100% bullet proof?

Also, I think that statement largely contradicts the notion of 'release early, iterate often' considering it's grounded on the notion of integrating feedback and bugs reported by users. Are major corporations held at a different standard simply because they have more resources? Even with thousands of employees, it's still a glass box compared to the millions that will use it.

Reply on news.ycombinator.com