Hacker News Reader

Comment by bajsejohannes | Hacker News Reader

> thanks to Zhendong Su and his team whose fuzz testing prevented many bugs going into the release.

http://web.cs.ucdavis.edu/~su/ claims 1228 bugs found (counting both LLVM and GCC). Impressive!

With fuzzing it's possible to find distinct bugs (or at least bugs that trigger in distinct code locations) without ever further investigating the bug in person.

Your bug report can simply consist of "this input file causes a compiler crash".

Crashes are the easier bugs to report, but often the most important compiler bugs are silent miscompilations that go unnoticed. With C compilers in particular, it's not easy to non-conclusively demonstrate a bug/convince the developers that the compiler is actually buggy (a lot of C standard lawyering is at play when it comes to subtle undefined behaviors).

[disclosure: I have been involved with the initial phase of "EMI" compiler validation work that's linked] One of the great strengths of EMI has been its ability to identify a lot of miscompliations in particular. In fact, while we did not fix the bugs ourselves, I would say the majority of the time spent was probably in reducing the bugs in a short program that manifested the issues and engaging with the developers to prove the bug is not an undefined behavior. I remember from my days that the majority of bugs we reported were miscompilations, not crashes, which took a lot of time to report. Even with simple crashes, you probably need to reduce the bug as the developers don't usually appreciate attaching a 7MB source file to your bug report!

Indeed! On the Mill project we leave boxes crunching hours and hours of csmith/creduce, and we don't watch them do it :)

SPE looks to be very nice too.

I haven't seen a Mill press release hit the front page for awhile.

I'm not good at subtly hinting.

What's the Mill project? Is that the "new computer" thing or am I missing something.

Yeap that's the project :)

We have our own llvm backend which is a front end to our own "specialiser".

We use csmith to fuzz for compiler bugs and creduce to reduce them. This starts with C and we even validate the output of the sim against clang x86 output.

How should someone go about contributing to your LLVM backend?

http://millcomputing.com/#JoinUs :)

http://millcomputing.com/topic/mill-computing-in-2017/ is our plan this year

So it is meant to be a different architecture? What market is it targeting?

Is mill low power, high performance, embeded, radiation hardened, etc? Will it come in small packages with RAM and ROM on the die? Will it support a different architectural view (maybe all the memory will be non-volatile)?

A lot of questions and very little answered in the page.

Give a man a bug report, he will fix his program one day. Give a man a fuzzer, he will be fixing bugs for the rest of his life.

Alternatively, you could do what I do and just not write bugs.

This works if you don't have coworkers.

I'm kidding! It doesn't work even then.

In those kinds of environments, typically I just will the program into existence, bug-free.

I agree with you cynicism when dealing with a software vendor like Oracle. Where bug reports are fixed for only the reported instance alone (if at all).

For a FOSS project like the LLVM I think getting >100 parser bugs might cause a slight re-evaluation of the current code structure. Maybe I'm a bit too optimistic, but the LLVM team has done a fine job of fixing outstanding issues.

Lastly Fuzzers like the AFL (American Fuzzing Loop) attempt to model input, sort of doing a lazy search for bugs (taking parser run-time into account). So when it finds a cluster of bugs they're normally clustered in the source code as well.

i don't think the comment was meant to be cynical, i think it was meant to show how much time and effort fuzzing can save when trying to generate reproducible test cases.

I wonder how many of the bugs were squashed by Dr. Su himself and how many were DISs or other forms of work supervised by Su. Maybe he did kill that many bugs on his own but that seems outrageous (without taking time to actually see what kinds of bugs they were).

If you look at the homepage, he has 4 separate links for LLVM bugs which differ only in the reporter's email address. The first, and largest, is his which suggests that he substantially did the actual work since the other members of his team appear to be reporting problems using their own contact info

The professor himself very likely did none of it. My guess is that someone like Dr. Su probably wouldn't even have enough time to work on fuzzing LLVM, and would likely be more concerned with writing up the next proposal to fund his lab :P

From what I've seen in academia, the supervisor typically generates ideas, and the grad students and postdocs test them out and see if they have potential. The discussion goes back and forth until a decision is made on whether or not to pursue an idea.

This differs from group to group of course, and is not the case in smaller groups, but it's generally how things work.

I think this can vary across people/projects and perhaps true in a large number of cases (this could also be said for the management in business, and it is debatable whether spending time on the ground doing the actual thing would always be the most leveraged thing one can do). Irrespective of that debate, at least for this particular instance, I can confirm that the professor in question himself [who happens to have been my doctoral advisor :)] went through a whole lot of trouble personally, down to reducing many many individual bugs and you can just look at the bug tracker activity under his name to independently verify. (I have long graduated so this is not a "paid Glassdoor review" either and ran into this as a frequent HN lurker!)

I happened to come across here... For this specific case mentioned in your first paragraph, it is not true.

Thanks for letting us know! That's quite impressive honestly :)

I can anecdotally confirm this to be true in my experience as well.

Reply on news.ycombinator.com