Hacker News Reader

Comment by sanxiyn | Hacker News Reader

Most advances are about filtering false positives. If you do "simple" interprocedural analysis and dump the result you will find lots of bugs together with lots of false positives and you can't sell that.

In other words, all the advances are in warnings you don't see. Yes, all Coverity warnings you see, are simple. I agree.

Quoting from https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-li...

> Since the analysis that suppresses false positives is invisible (it removes error messages rather than generates them) its sophistication has scaled far beyond what our research system did. On the other hand, the commercial Coverity product, despite its improvements, lags behind the research system in some ways because it had to drop checkers or techniques that demand too much sophistication on the part of the user.

View on HN

:) Cool that you recognize that.

This is correct. Coverity does a bunch of specific analyses designed to eliminate False positives. This includes analysis to determine which data states are not possible for a given code path, and doesn't report those specific issues. This also works using data across function calls.

For C/C++/C#, Coverity has by far the lowest false positive rates, which make it probably the best in class for those languages.

Disclosure: I used to work at Coverity.

Fascinating! The specific example you mentioned sounds publishable, any idea? I mean, I am just curious, I no longer work on this.

I now have quite different perspective wrt false positives; that false positive rate is not important. It's all about perspectives. Name the tool bug search engine instead of bug finding tool. You rarely look beyond the first page of search engine result. Develop a ranking algorithm such that all alarms in the first page is relevant. You can use any probabilistic voodoo to rank. Etc. I don't know whether this can work. But I think it's worth a try.

I think the takeaway here is that Coverity have just made a (clever) business decision to eliminate false positives, so that (nearly) 100% of their defects are "correct". It's quite possible that there are real defects that they don't show because their confidence is less than 100%.

No, not really. "false positives are the enemy" (or at least, in my alternative model, relevancy) becomes glaringly obvious for anyone who tried this at scale. The question is how to reduce false positives (or improve relevancy). The part that "real defects not shown" exist is called unsoundness and pretty much everybody now agrees you should be unsound. (Or being sound is a different market.)

Of course, this is not obvious to people who haven't tried, so it is reported over and over and over. The most recently, from Google. Lessons from Building Static Analysis Tools at Google (2018). https://cacm.acm.org/magazines/2018/4/226371-lessons-from-bu... I mean, I could have told them.

Select quotes:

> Unlike compile-time checks, analysis results shown during code review are allowed to include up to 10% effective false positives.

IMPORTANT NOTE: 10% effective false positives means much less than 10% false positives! In above quote, Google defines true bugs marked as not-a-bug by a developer as effectively false positives. If what you say is true, but users misunderstand, it doesn't count.

Reply on news.ycombinator.com