Agreed. Their definition of static analysis is laughably bizarre as well; there are many security and safety properties that can be proved by static analysis but not by dynamic analysis. Additionally, many of the problems they attribute to static analysis (non-specific versioning and recursive dependencies) are caused by the package managers themselves, and can be solved using static analysis if the build tool is well defined and well behaved.

Their tool has a few major weaknesses:

  1. Builds are not reproducible - Reproducible builds require pinned package versions, which they specifically avoid. This could result in security holes if a dependent package version was bumped after the parent project was tested.

  2. Subject of analysis - Their main target of analysis is the build file. This severely limits the extent of their analysis and requires them to build new tools for each build tool used.

  3. Underestimation - Since a code path must be exercised in order to analyze it, you are guaranteed that the set of vulnerabilities detected is a subset of (or equal to) the true set of vulnerabilities. This is the opposite of what I believe should be the default: Always prefer false positives over false negatives. Both static analysis and binary analysis allow the programmer to over-estimate their analysis, guaranteeing that all vulnerabilities will be found.

None of the terms they define share the definitions that are understood by everyone in the field.

Static analysis refers to analysis techniques that understand code without recourse to execution, usually during compilation. (Static binary analysis does exist, but the difficulties of disentangling code from data in native code makes it usually impracticable). The Java FindBugs tools or Clang's static analyzer are good examples of static analysis techniques.

Dynamic analysis refers to analysis techniques that rely on gathering information using actual executions of a program. Even pedestrian techniques like profiling and code coverage are considered dynamic analysis techniques.

Binary analysis is a completely orthogonal concept. It refers to the analysis of programs without recourse to the source code, and is usually understood to refer almost exclusively to native machine code as opposed to bytecode languages like JVM or .NET IL. Most binary analysis tools are dynamic (such as Pin or Valgrind) but there do exist static binary analysis tools such as GrammaTech's CodeSurfer.

This doesn't really capture the whole picture of techniques, though. Symbolic execution and its step-child concolic execution aren't best thought of as static or dynamic analysis.

If you consider the build system to be a language in itself, I think this is appropriately called dynamic analysis.

For example, a profiler for a Makefile should record the time taken for each build step as the Makefile is running. It's not about profiling the program being built.

Good point. The wikipedia definition is my understanding as well. Analyzing during build time is what scan-build does. They call it the Clang Static Analyzer.

Does this look at code at all? Or is it just some build tool to make sure you're only using Approved Versions of Approved Components?