Agreed. Their definition of static analysis is laughably bizarre as well; there are many security and safety properties that can be proved by static analysis but not by dynamic analysis. Additionally, many of the problems they attribute to static analysis (non-specific versioning and recursive dependencies) are caused by the package managers themselves, and can be solved using static analysis if the build tool is well defined and well behaved.

Their tool has a few major weaknesses:

  1. Builds are not reproducible - Reproducible builds require pinned package versions, which they specifically avoid. This could result in security holes if a dependent package version was bumped after the parent project was tested.

  2. Subject of analysis - Their main target of analysis is the build file. This severely limits the extent of their analysis and requires them to build new tools for each build tool used.

  3. Underestimation - Since a code path must be exercised in order to analyze it, you are guaranteed that the set of vulnerabilities detected is a subset of (or equal to) the true set of vulnerabilities. This is the opposite of what I believe should be the default: Always prefer false positives over false negatives. Both static analysis and binary analysis allow the programmer to over-estimate their analysis, guaranteeing that all vulnerabilities will be found.