Hacker News Reader

Comment by billyhoffman | Hacker News Reader

I wouldn't call this fuzz test because it focuses on simulating legitimate actions. Do random clicking. Do random scrolling. Do random mouseovers. Do random [any DOM event]. Yes, they are doing this at a massive scale, but the key here is that germlin.js can only do things a user can do. In short, Gremlin is just automating actions; basic legal, legitimate actions. Still cool, and great for stress testing or looking for race conditions in your UI code, but not fuzz testing.

Fuzz testing, on the other hand, is about modifying the data, to interacting with the UI. To use an analogy, fuzz testing Microsoft word might involve corrupting various structures in the DOC or DOCX file or in an OLE embed with malformed data and seeing how the parser/program reacts. The Gremlin.js equivlent would be just clicking on a bunch of buttons in the Word UI really really fast.

Both are helpful, but they are testing different things

View on HN

Not entirely true.

Fuzzing is not about modifying data. I don't think that's the right way to describe fuzzing.

It's about testing the durability of a program by trying all kinds of data, as greedy as possible. This includes known problematic inputs and random inputs to match some expectations. By random it can either be totally random (any length, any pattern) or protocol-aware.

Random clicking is a form of random data testing because you are trying random input stream to a program. Your argument is not entirely wrong either. By fuzzing his UI he may trigger the browser to crash. He may trigger his monkey to crash. Fuzzing is a very general technique. I can write a fuzzer that fuzz Firefox's UI Australis. What will I look for? Maybe after opening 100 tabs and closing the 45th tab the titlebar disappeared. Or resize the browser from range s to range w I will find some range will cause the UI to look ugly (style overflow, etc). Or after clicking on the scrollbar several times consecutively the browser crashed.

Barton Miller, the "Father of Fuzzing" did UI fuzzing by simulating actual keystrokes and mouse clicks. See ftp://ftp.cs.wisc.edu/paradyn/technical_papers/fuzz-nt.pdf

In his Forwards page, he even mentioned "Monkey" http://pages.cs.wisc.edu/~bart/fuzz/Foreword1.html

One nice way of describing fuzzing is that it is running a search problem (searching for bugs) using more or less guided Monte Carlo method for the program's input space. Generally the testing should also be automated (test case injection and testing oracle), so that one can beat the program with enough test cases for the fuzzing to be viable.

Relating to the practical example: if one is trying to find bugs from the UI code (or it is the only way to feed inputs to the program), monkey method of fuzzing is the way to go. But if one tries to test the deeper layers of the program, it is beneficial to try to find the lowest layer of inputs we can access, since it enables faster input of test cases and thus makes the fuzzing more effective.

One way of the other, my opinion is that both are fuzzing, by the definition I gave for it :-)

I like the way you treat fuzzing as search program.

It's a nice way of summing up the general principle. Unfortunately I can't take the credit for coming up with that one :-)

Reply on news.ycombinator.com