Coverity Scan Update | Hacker News Reader

Coverity Scan Update

community.synopsys.com

49 pointsfcambus33 comments

Coverity Scan regularly goes down for hours or days.

In February of 2018 it was down for over a month with no word or ETA on when it would be fixed. I hadn't thought about it since then (we discontinued use), but researching it now they released a statement saying that it was hacked. There was not a single status update during the outage. https://www.theregister.co.uk/2018/03/19/coverity_scan_crypt...

I wonder who the hosting provider was. I'm not seeing much in the news about one that "unexpectedly ceased operations", just the expected background news of scattered outages.

Based on DNS history for the domain it looks like it was https://twitter.com/nephoscale | http://nephoscale.com/

Wild! I wonder why they picked a host that I'd literally never heard of before this moment? Not that I claim encyclopedic knowledge of virtual hosting providers, but still.

Coverity was an acquisition. The hosting company was probably ran by a friend of the founders (both seem to be based in the bay area).

Weird... seems they just disappeared with no announcement at all?

Coverity is really good. It is a pity some of its advances, effective in practice but not really "publishable", will forever remain as proprietary secret.

Source: I worked on static code analysis product and we extensively black-box tested Coverity.

What kind of advances are you thinking of?

As best I can tell, most of the warnings are either things that can be figured out for a single translation unit and (some) compilers will eventually incorporate as warnings, or things that can only be figured out by analyzing / linking across many translation units - which a compiler can't do, but the actual "advance" is simple enough if you have all the function definitions to hand.

Most advances are about filtering false positives. If you do "simple" interprocedural analysis and dump the result you will find lots of bugs together with lots of false positives and you can't sell that.

In other words, all the advances are in warnings you don't see. Yes, all Coverity warnings you see, are simple. I agree.

Quoting from https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-li...

> Since the analysis that suppresses false positives is invisible (it removes error messages rather than generates them) its sophistication has scaled far beyond what our research system did. On the other hand, the commercial Coverity product, despite its improvements, lags behind the research system in some ways because it had to drop checkers or techniques that demand too much sophistication on the part of the user.

:) Cool that you recognize that.

This is correct. Coverity does a bunch of specific analyses designed to eliminate False positives. This includes analysis to determine which data states are not possible for a given code path, and doesn't report those specific issues. This also works using data across function calls.

For C/C++/C#, Coverity has by far the lowest false positive rates, which make it probably the best in class for those languages.

Disclosure: I used to work at Coverity.

Fascinating! The specific example you mentioned sounds publishable, any idea? I mean, I am just curious, I no longer work on this.

I now have quite different perspective wrt false positives; that false positive rate is not important. It's all about perspectives. Name the tool bug search engine instead of bug finding tool. You rarely look beyond the first page of search engine result. Develop a ranking algorithm such that all alarms in the first page is relevant. You can use any probabilistic voodoo to rank. Etc. I don't know whether this can work. But I think it's worth a try.

I think the takeaway here is that Coverity have just made a (clever) business decision to eliminate false positives, so that (nearly) 100% of their defects are "correct". It's quite possible that there are real defects that they don't show because their confidence is less than 100%.

No, not really. "false positives are the enemy" (or at least, in my alternative model, relevancy) becomes glaringly obvious for anyone who tried this at scale. The question is how to reduce false positives (or improve relevancy). The part that "real defects not shown" exist is called unsoundness and pretty much everybody now agrees you should be unsound. (Or being sound is a different market.)

Of course, this is not obvious to people who haven't tried, so it is reported over and over and over. The most recently, from Google. Lessons from Building Static Analysis Tools at Google (2018). https://cacm.acm.org/magazines/2018/4/226371-lessons-from-bu... I mean, I could have told them.

Select quotes:

> Unlike compile-time checks, analysis results shown during code review are allowed to include up to 10% effective false positives.

IMPORTANT NOTE: 10% effective false positives means much less than 10% false positives! In above quote, Google defines true bugs marked as not-a-bug by a developer as effectively false positives. If what you say is true, but users misunderstand, it doesn't count.

Has anyone tried LGTM / Semmle QL for automated code review? They claim 100K OSS projects are using the service. It's a bit hard to find technical information on the product, but they have found CVEs in mainstream products, including iOS.

https://lgtm.com & https://semmle.com/ql

I work on C/C++ analysis at Semmle, and am happy to answer any questions you might have. (We also support C#, Java, JavaScript, Python... and Cobol!)

We have a few high-profile projects using our automated code review (marketing people tell me I'm not allowed to call it 'PR integration' any more). One example is on the AMP Project, where we caught a regex injection vulnerability in a PR before a human looked at it: https://github.com/ampproject/amphtml/pull/13060

Our default analysis has found a few other vulnerabilities (remote buffer overflows due to misuse of snprintf in rsyslog and Icecast spring to mind) but, honestly, I think our strength lies in the fact that you can write custom queries that find bugs specific to a single codebase's foibles.

For example, my first ever CVE was for a vulnerability in ChakraCore. Google Project Zero found the original bug - type confusion caused by failure to check a flag indicating that the last element of a list should be cast to a different type - but we wrote a query to verify that code accessing that particular list always checked the flag. So when some new code got introduced with the same bug, we noticed as soon as we re-ran the query on the new commit.

Is there a public language reference on the query language? The Help section of the website only has intro examples. How long does it take for a new developer to learn to write non-trivial queries?

Any technical comparisons with other static analysis tools?

It would be useful to have a public database of QL sample queries alongside matching OSS code snippets. In theory, this could be constructed from Github, but presumably LGTM already has that historical viewpoint.

I think the QL handbook is what you're looking for: https://help.semmle.com/QL/ql-handbook/

Other QL references (such as the reference manuals for the language-specific libraries) are linked from here: https://help.semmle.com/QL/ql-home.html

Personally (having joined the company reasonably recently), I haven't found it particularly hard to figure out how to express my intentions in QL. Most complex queries start their lives as simple ones that return too many results; the complexity comes from adding extra conditions to refine the results, and is generally manageable. However, I do occasionally have difficulties where a query I'm developing returns no results, and my years of experience debugging imperative programs don't really help as much as I would like in figuring out what's wrong. It can also be quite easy for beginners to write queries that are interminably slow for non-obvious reasons; I believe we're working on documentation to help with that.

We do have a database of all the default queries here: https://help.semmle.com/wiki/display/QL/Built-in+queries

But I notice that not all of them have examples, and many of the examples we do have are fairly contrived. I like your idea of taking snippets from matching OSS projects.

Edit: our CEO gently reminded me that, if you look at the help page on lgtm.com for a particular query/rule/alert, you can get a list of all results across every OSS project we analyse, e.g. https://lgtm.com/rules/1506728586782/alerts/

Thanks for the references, that last link is very cool!

> It can also be quite easy for beginners to write queries that are interminably slow for non-obvious reasons; I believe we're working on documentation to help with that

RDBMS have the same challenge. Data visibility for query parser internals can help, https://www.embarcadero.com/images/dm/technical-papers/sybas...

> We also support C#, Java, JavaScript, Python... and Cobol!)

Where do these rank in chances of future support?

  - Go
  - Rust
  - Lua
  - Swift
  - Haskell

apparently golang is, but d is not: https://discuss.lgtm.com/t/which-languages-are-in-the-pipeli...

and no swift for now: https://discuss.lgtm.com/t/does-lgtm-support-swift/1575

Go support is currently in development, so very likely :)

As for the other languages there, there are no plans currently on the roadmap for 2019 to add any of them. However if this is something that particularly interests you, we'd encourage you to apply for a job and note that you'd like to add support for a particular language :)

Systemd and tesseract-ocr both use it for example:

https://github.com/tesseract-ocr/tesseract https://github.com/systemd/systemd

systemd have also written their own QL query: https://github.com/systemd/systemd/blob/master/.lgtm/cpp-que... https://lgtm.com/projects/g/systemd/systemd/alerts/?mode=tre...

(full disclosure, I also work at Semmle)

We at rsyslog use it as part of our CI pipeline. It's a pretty cool tool, done by awesome folks. A real big plus is the responsiveness to questions and when something needs attention. When we came to LGTM (at that time in beta for C), we already had a pretty clean code base. The reason is that we use other static analysis tools. LGTM did nevertheless find some new and interesting things, include a real vulnerability[1]. We don't use custom QL right now (time is soo short), but opt in to some extra canned queries. For example, we prevent commented-out code via them. LGTM is a check REQUIRED to pass CI [2, sample PR].

We are really happy with both the product and the team and I can recommend diving into it. Nevertheless, don't rely on a single static analysis tool, no matter how good it is. If you use C/C++, you should at least also have clang's static analyzer in use.

[1] https://rainer.gerhards.net/2018/06/how-we-found-and-fixed-c... [2] https://github.com/rsyslog/rsyslog/pull/3403

We use it in https://shields.io/ for JavaScript. It finds semantic bugs that are really hard for humans to find. I think it especially shines at finding problems in old code. Yes, old code is good code, though it may not be secure. Also unused code, like React state that is set but never read.

I used it for Nock, too, to make some quick fixes: https://github.com/nock/nock/pull/1301/files

In Shields we have the PR app turned on and it keeps us alert to problems during the review process. It's slower than CI, though usually problems manifest early enough in the review process that they can be fixed before merge.

I honestly thought it is gone!

All links are dead, and synopsis.com’s big Corp style website isn’t helping one bit.

> Coverity Scan is a free static code analysis tool for Java, C, C++, C# and JavaScript. It analyzes every line of code and potential execution path and produces a list of potential code defects.

There we go, I had no clue what this even was. Do a lot of people here use it?

Coverity have one of the best static analyzers for C++ and CSharp. Not a surprise considering they have had* ex-Microsoft compiler engineers such as Eric Lippert working for them.

I understand from speaking to C++ engineers who have extensive experience in embedded / industrial applications that Coverity is used extensively there.

Personally I've never seen the need to apply it to CSharp projects because the language is naturally safer than C++ and you get a lot of "bang for the buck" by using the tools built into Visual Studio and from JetBrains.

* He works at Facebook now :facepalm:

"A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World" (2010) is a classic popular level account of Coverity. It is very funny, in an unintentional way. Highly recommended reading.

https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-li...

We pay for Coverity for our closed source products and are quite happy with it. But you have to have a project committed to staying on top of static analysis to really get the value out of it. Our product domain means we do, and for all our static analysis tools we have a fix-or-document rule where we have to address every warning before a release. Coverity is great for us.

I have access for two opensource programs using it (bitchx and znc). The webui is a bit like a full blown IDE. It has found a few bugs (some are CVE's) in various projects.

https://github.com/znc/znc/search?p=1&q=Coverity&type=Commit...

https://github.com/bitchx/bitchx/search?p=1&q=Coverity&type=...

I've used it for the OSS project I'm working on (https://ossia.io) but at some point the coverity scans just started failing, and they never replied to my support requests.

Coverity was the first really good static analysis tool -- a classic example of serious academic research (the "Stanford Checker") being spun off as a commercial product.

Unfortunately they lost their focus and got acquired; if they had taken a different course they could be an essential security tool.

Wouldn't it be great if professional websites will someday get to the level of non-professional websites? E.g. by giving this announcement page a proper title: "Coverity Scan Outage".

Update is a change, this is an outage.

Reply on news.ycombinator.com