Hacker News Reader

Pray that you never have to enter domains where the security audits and assessments must be always done by an "approved supplier".

We have had to cancel engagements with suppliers who I have thoroughly enjoyed working with, and whose results were nothing short of spectacular.[ß] And all of this because certain regulators had their rules written by an established auditing firm who managed to make sure that only a select few (less than 5 globally) companies can ever perform their required assessments.

Regulatory capture through security auditing companies should deserve its own circle of Hell.

ß: Have you ever had a pentest report you could forward directly to your engineering teams and be certain that they understood the contents accurately? I have. But I can't use the supplier anymore, because they do not enjoy a royal charter.

Not just languages - Generally if the scanner isn't aware of the specific framework you're using it's unable to find entrypoints or reliably figure out sources / sinks and filters.

Many modern frameworks have one or more of: dynamic configuration, compile time annotations, reflection, IoC etc which make it very hard for a "first principles" scanner to make sense of what can actually happen at runtime.

What exactly is a "'first principles' scanner?"

Sorry, was perhaps poorly worded. By "first principles" I meant a static analyser working using only an understanding of the language (say, Java) and the application source code, looking over the ast/cfg etc using only built-in language rules.

You get much better (read: maybe useful) results if you happen to have a "rule pack" for the specific framework you're using which provides specific hints on sources + sinks, "gotchas" and how things are wired together. As a somewhat obsolete example, I would not expect a sast working only from "first principles" to be able to do anything useful with a Spring XML configuration file.

On the whole my experience is that these things work very well on certain types of codebases - e.g. naive PHP they can "go to town" because of the huge footgun surface and fairly direct control and data transfer.

Stuff with lots of "magical" framework features and indirection (where even a human reviewer can often have trouble finding the implementation from the interface being invoked) they often silently fail to do anything useful.

> static analyser working using only an understanding of the language [...] looking over the ast

The part I quoted is called "tainting". Or if you are more academically inclined, "data flow analysis".

When it works right, it is an incredible force multiplier in security. You get detailed, actionable and above all helpful error messages directly from CI, because as a static analysis tool it's pretty fast and can be made part of the common linting pass. As you hinted, it does require a suitable config setup and/or code annotations to mark sources and sinks. And when it does work, it can eliminate an entire class of vulnerabilities - good taint analysis will prevent you from even accidentally using user-supplied data in anything that involves relaying, storing or displaying information.

The downside is, when it doesn't work, it's a source of unhappiness. Debugging a taint failure because the AST analysis gives a false positive can be infuriating.

I suspect this means a scanner that can derive all necessary information without any configuration. For example, consider a scanner looking for API endpoint authorization inconsistencies. Does the scanner need you to describe your authorization scheme, or can you simply run the thing and it figures it out?

This can be easy or hard depending on how bespoke your application is. If you're using something like Ruby on Rails, then there's a paved road that a scanner can preconfigure to understand your application. If you're using a homegrown authorization framework, then a scanner will likely have a hard time understanding your application and will need to be configured.

Even Fortify and other tools which have to compile beforehand don't do so well IMO. Someone is going to come up with an actually good SAST and print money...

I’ve had really good experiences with Fortify so far. It took me a lot of hours to figure out how to write custom rules but once I was able to do so it became super valuable.

I didn't write about this because it wasn't toward my exact point about FP, but you're totally right. This will be equally as chilling for new web frameworks.

This cannot be stated enough. The company I worked for purchased "the leading gartner SAST testing solution" before they even started a project and had a framework chosen. It turned out the SAST solution didn't support the latest major version of the framework we were using at the time, and didn't support the latest version of the language we were using. Even worse, it never seemed to ever report an issue, and took hours to run at times, eating up build minutes.

I ran bandit on the code base just for fun one day, and we had four hits and it took 5 minutes to run. It took a while, but I finally convinced the powers that be we were better using the tool we could verify works, rather than trusting the SAST vendor that it worked.

Fortify took almost a decade to support Python 3 and there were some hefty lags on Java updates, where multiple times it was like “do you want to use the supported Java or the one which Fortify can scan?”

waves, original Bandit author here. It could've done more, but pretty cool to see how useful it's been and where it's got to.. OSS is fun.

This. I work on an Android app written in Java. We were thinking about migrating to Kotlin years ago, but our blessed tool - Fortify - only just released a technical preview for Kotlin 1.3.50 in Fortify 20.1.0, and already Kotlin 1.4.10 is out.

My impression is that the current owners of Fortify lack the resources that HPE had and the product is suffering as a result. It's best suited for maintaining old code bases.

Wow. Great. Would you be able to publish your analysis?

Unfortunately I cannot as it is not my property. Plus one vendor made my client and myself sign an NDA even for the most basic information. What I can say is to not put too much faith in the Gartner Magic Quadrant for this space. The best thing to do is select a couple of representative projects for your company and get each vendor to let you do a POC and then compare the scan results from each. The process of making this happen will be long and painful as it is managed mostly by sales teams rather than the tech folks. In the end though, it's the best way to know what will work for your particular set of development technologies. The good news is pretty much every vendor was receptive to the idea of doing a POC even if it took a while to get each one set up.