In-Depth: Static Code Analysis (2011) | Hacker News Reader

In-Depth: Static Code Analysis (2011)

gamasutra.com

122 pointstzhenghao13 comments

It's important to keep in mind that there are vast differences in static analysis tools. Some of the commercial ones that John mentions in the post are quite powerful but also very expensive and out of reach for most startups / smaller companies. Most of these started off as spin-offs from research groups, and there's hundreds of man-years of research behind some of them, which (IMHO) makes the technology hard to replicate in the open-source world.

Then there's open-source static analysis tools, which unfortunately are often much less powerful. Especially for dynamically typed scripting languages like Python it's very difficult writing good static analyzers because there's no type information available (projects like mypy should improve that though). In my opinion, when working with such languages gradual typing like done by MyPy / type annotations in Python or transpilation like Typescript does is a very good first investment when developing in these languages, and will also make static analysis easier later.

There are some studies (I can't find the link now) that compared various measures for improving code quality in terms of cost and effectiveness, and the takeaway was that if you're starting out with QA manual code reviews are usually the most cost-effective way to improve software quality. Of course you will need to invest some time in getting a high-quality review process set up, and ensure that there are developers in your team that have the experience to do meaningful reviews.

For the examples mentioned in the post, it seems like the Clang static analyzer might be enough? I’ve never seen anyone using a commercial analyzer; are these significantly better?

I have limited experience with them but I think tools like Coverity or Veracode are significantly better, but maybe they are just better at marketing.

Coverity actually offers free scans for some open-source projects, here's the report for the Linux kernel:

https://scan.coverity.com/projects/linux

They have a large list of other projects as well (it seems you can't drill down into individual errors and only see summary statistics though, which sucks):

https://scan.coverity.com/projects

I didn't find anything comparable for Veracode, but comparing the Coverity results with a static analyzer like Clang should give you an idea of the effectiveness. It's not a magic bullet (IMHO) but it definitely can find bugs that other tools don't see.

Coverity is pretty good, but shows quite a lot of false positives in my experience. This may obviously depend on the project, of course.

What I really wish for is an easily understandable way to teach these advanced static analysis tools about why something is a false positive, so that the tool can avoid showing the same false positive again in the future when a minor detail of the surrounding code changes.

> What I really wish for is an easily understandable way to teach these advanced static analysis tools about why something is a false positive, so that the tool can avoid showing the same false positive again in the future when a minor detail of the surrounding code changes.

People have been doing research on this for decades. The problem is that the way that humans understand code and the way that static analysis understands code are wildly different and the root cause of a false positive is either fundamentally not possible to avoid or so far removed from the final analysis that you can't really address it.

Static analysis results are often brittle in fundamental but non-obvious ways. For example, a human might think that moving code between two functions shouldn't change much. But when you are using K Context Sensitivity, suddenly just wrapping some code in a helper function will completely fuck your points-to results.

My experience in most projects I have been involved is that code reviews usually come after unit tests, which come after documentation.

While back, i did a tool that uses clang sa, tidy and cppcheck to provide more information to our existing Coverity scans. For coverity scans, mostly the meaningful stuff was enabled and results compared to clang & cppcheck where not that different. Next time id just to with those and not bother with coverity unless licensing is already there and some aspects require coverity..

Anyway, more is better. Coverity is nice once you get it setup and you have recentlish version but combine it with open source offerings too!

Interesting that back when McPeak was working on Elsa/elkhound clang wasn’t available. clang is definitely a game changer for c/c++ static analysis.

Plenty of idiomatic Python still contains enough magic to stymie mypy et al.

Here is a good serie of articles on static analysis tools : http://btorpey.github.io/blog/categories/static-analysis/

Two others worth checking out that focus on soundness with low to no false positives are RV-Match and TrustinSoft Analyzer:

https://runtimeverification.com/match/

https://trust-in-soft.com/products/

The RTV people are the one's behind K framework and an executable semantics for C with a GCC-like interface:

http://www.kframework.org/index.php/Main_Page

https://github.com/kframework/c-semantics

That RV-Match builds on an open framework, does well on benchmarks (yeah skepticism here), and has little to no noise (most important) is what I like most about it. I say most important since digging through false positives makes developers not want to use a tool. I haven't tried either product myself, though.

Here's a list of open tools that did well in small-scale experiments with and without false positives:

https://klee.github.io

http://www.cprover.org/cbmc/

https://cseweb.ucsd.edu/~rjhala/blast.html

http://www.splint.org

http://spinroot.com/uno/

http://cppcheck.sourceforge.net/#features

https://en.wikipedia.org/wiki/Sparse

https://cpachecker.sosy-lab.org

Paraphrasing Dijkstra: “Using static code analysis shows the presence, not the absence of bugs.”

See: (1) “Finding Heartbleed with CodeSonar”, (2) “Why Do Software Assurance Tools Have Problems Finding Bugs Like Heartbleed?”

[1] http://blogs.grammatech.com/finding-heartbleed-with-codesona...

[2] https://www.swampinabox.org/doc/SWAMP-WP003-Heartbleed.pdf

I think that's a misapplication of the quote. Testing just checks an algorithm on specific inputs. The proofs check it for all inputs. Some static analyzers are designed to show absence of bugs of specific type for all inputs. They're more like proof.

Astree Analyzer and TrustinSoft Analyzer are examples that claim to prove absence of errors with sound analysis.

Reply on news.ycombinator.com