Comment by exDM69 | Hacker News Reader

Comment by exDM69 | Hacker News Reader

Parent Full thread

These books are probably great but if you look at some of the best/worst bugs found in the past few years, there's one tool that seems to come up over and over again: afl-fuzz.

I recon that the easiest way to get started with bug hunting might be to just set up afl-fuzzing on some trivial code that has lots of potential for going wrong. Stuff like JSON/HTML/HTTP parsing in C is a great candidate to find integer overflow or buffer overrun bugs in (some of the bugs in this list are exactly that). Throw some CPU time for the fuzzing, and pretty soon you should have a handful of repro cases.

The nice thing about these is that the bugs could be very trivial to fix but have enormous security consequences.

If someone wants a suggestion for a project to try some fuzzing against, the new h2o/libh2o web server and its HTTP parser component (picohttpparser) look very well written but not very well tested (only a handful of hand written, hard coded test cases). I'm pretty sure there's one or more potentially disastrous bugs in it.

edit: almost all bugs in this list seem to be integer overflow bugs. That hints that these issues were found using a static analysis tool like ClangAnalyzer or Coverity.

Please don't start this way. Michal Zalewski didn't write afl fuzzer so it would be used without deep technical knowledge. It's actually not for beginners at all despite how powerful it is.

As a general rule to start in security work, you will want to know how to program in at least one language competently, then move on to theoretical understanding of vulnerabilities, then finding them painstakingly by hand, and finally by automation.

It's perfectly fine to use afl (or other similar tools), but understand that using an automated tool without first knowing how to do things by hand and groking the theory will stunt your growth and make it hard to progress very far.

Pop open a weakened VM and exploit a few buffer overflow bugs the old fashioned way before you use a fuzzer. With code review, learn the nuances of each language before you automate your audit. You shouldn't have to rely on an automated tool to find out register globals is on in PHP (as one example).

I agree with you if we're talking about someone who is just getting started in programming and debugging. It's not a good starting point if you want to understand why these bugs exist or how to exploit them.

But if we're talking about someone who is already a competent programmer, perhaps looking to learn a new tool, there's just no point (except perhaps learning) in trying to find bugs manually in production software by reading code or by trying to write test inputs that invoke unwanted behavior. After all, most code works fine at least 99% of the time, finding the 1% manually is a lot of effort.

Tools like afl-fuzz or static analysis tools are so much faster (when you throw enough CPU time at them) in finding unwanted behavior that there aren't many reasons not to use them.

For learning and getting started with exploits, I like your idea of using some kind of "weakened" VM, but that's a whole different deal than finding and fixing bugs (not even talking about exploiting) in production code (which is where the bounties are).

I realize there's a philosophy that it's best to start from the lowest levels and work one's way up. It's one I can empathize with, in spirit, though I disagree.

The same logic means that people shouldn't use static code analysis tools, or valgrind, or even debuggers, until they acquire deep technical knowledge. While I think all of these tools help reinforce the principles.

If someone starts with the fixed and unwavering goal of security analysis in mind, then perhaps I can agree with you. If however someone is only curious about security analysis, and finds that spending a year to "grok the theory" is a high barrier, then even clumsy use of semi-automated tools may provide more concrete incentive to learn the underlying skills.

While I don't believe you are correct, another question is, how many white hats do we end up with? Even if it takes 2 years to learn security skills by using automation, and only 1 year without automation, if after 10 years there are 500 following your path, while 10,000 following my path, then that's a net gain for the good side, yes? (I assume that there is such a thing as "good enough", and that it's relatively stable. Obviously if there is only a market for 500, and your 500 are always better than my 10,000 then that changes the dynamics.)

Finally, it's also good to have even the script kiddies on the side of good than the side of lolz.

>> The same logic means that people shouldn't use static code analysis tools, or valgrind, or even debuggers, until they acquire deep technical knowledge. While I think all of these tools help reinforce the principles.

This probably sounds controversial, but I agree. I don't think you should use Valgrind until you understand how Valgrind works. This doesn't mean early C programmers shouldn't use Valgrind - you can read the documentation and theory behind Valgrind in a day. But definitely do that. I think maybe "deep technical knowledge" wasn't the right term for me to use. A better term would be "technical understanding" - know how it works, and know how to find the different classes of bugs it can find, but you do not need to be capable of writing the tool yourself.

Now let me clarify this, and my earlier point about afl - I think you should use them, and generously, and pretty much always once you know what you're doing. But if you use them without understanding the fundamentals, you will get caught up in false positives/negatives. Always use it as a supplement, not a crutch.

However, I agree with what you're saying about someone's level of dedication to security analysis. Using afl is better than not using afl, so if you're not a dedicated security guy, then you're definitely right that someone should use it.

I would still caution anyone that using an automated tool without fully understanding how it works will lead to an incomplete picture of the application's security posture.

Well in that case, I have only a surface level understanding of Valgrind, but I use it for those rare times I code in C. I also don't know how afl works at the code modification level, other than that it asks gcc to add some instrumentation, but I've used it to find bugs. :)

I completely agree with your last statement, with one proviso. You will always have an incomplete picture of the application's security posture. The fundamentals include hardware and OS behaviors, power consumption, RF emissions, and other fields which are very far from purity of the programming language model. Acquisition of all of this knowledge is very difficult, but is also rarely needed.

I think for most cases it's okay to have an incomplete understanding of the tools. It depends on the threat model.

This idea of people not using tools until they understand how they work (or in the Matasano shorthand, until they can implement them, in "build your own light saber" fashion) is I think very true of professional testers, but not true for developers.

If your full-time job is going to be about developing a radar for assumptions that developers make and a deviousness about breaking those assumptions, tools --- and most especially opaque tools --- totally hamstring you. We both know plenty of testers that develop a kind of blindness from relying on tools too much.

But I'm guessing most of HN doesn't really understand the mindset that's involved in pentesting full-time. I certainly wouldn't tell a developer that they should avoid valgrind.

Reading some of the actual reports, these look like straight code findings (see for instance the numbered step-by-step explanations). They also aren't particularly complicated bugs.

afl-fuzz gets a lot of press because it's neat, because like Peach it's a well-packaged version of a tool every tester has their own half-assed version of, and because lcamtuf wrote it and everyone likes lcamtuf.

It is not, however, responsible for most security bugs. Most testers don't use afl-fuzz.

Python flavored afl-fuzz for reference: https://bitbucket.org/jwilk/python-afl

Reply on news.ycombinator.com