Hacker News Reader

Microsoft has used products from Semmle (the now-GitHub and thus now-Microsoft division whose tech is in GitHub code scanning) for a few years, and I've personally used it on occasion.

From that limited experience, I'd say that false positives are less of a problem with Semmle's checkers than with other security-focused static analysis tools. This is partly due to Semmle checkers being much more customizable; Semmle has developed a declarative query language called CodeQL* which its checkers' built-in and user-provided rules are written in. Microsoft's security development lifecycle has a lot of mandates which are captured by custom CodeQL rules precisely enough to match their intent.

You can see some examples of how Microsoft uses Semmle here: https://msrc-blog.microsoft.com/2018/08/16/vulnerability-hun...

* https://github.com/github/codeql

There's a genuine security fatigue issue (much like event fatigue) that comes from false positives. Unfortunately that doesn't reduce the value of the scanning - the onus is on the false positives.

At the very least, running and pruning scans should happen on projects so that at least we can have the conversation. It's like PCI (as an example, not an ideal); PCI isn't perfect, but at least it encourages a conversation about security. Today we're at the point where almost every single organization at least discusses security; but I remember when PCI first came out. I can't tell you have many times people used to ask why it was problematic to store passwords in plain text.

It this the best step forward, probably not. Is it a step forward, absolutely.

Automated proof of vulnerability is valuable. Kafkaesque pattern matching, is not.

These warnings are disorientingly and illogically complex in a surreal or nightmarish way?

You did find a mild way of putting it.

That's only true based on a certain level of care, which sadly is not always the case. Say you have a codebase that wasn't built well. Let's say your code scanner finds you at the very least the text "-----BEGIN R/DSA PRIVATE KEY-----*-----END R/DSA PRIVATE KEY-----". In my codebases that'll find nothing, but for many people, they just found SSH keys committed in plaintext. The same is true for any 1k/2k (++etc) string. It highlights what could be a private key and worth at least looking at.

Neither is proof of a vulnerability. Both I will gladly, always 100% validate.

* will match {cipherkey} so I do not want it.

Which is of no issue of they're just there for easy mocking etc.

I agree with the parents point, if there is value in such tests, it's so miniscule that it dwarfs the added effort to filter out all the false positives if the developers aren't total newbies. And if they are, code reviews by more experienced developers should be a given, making it redundant as well.

Just because a dependency isn’t deployed to production doesn’t mean it’s safe. Remember when eslint tried to steal people’s npm credentials?

https://news.ycombinator.com/item?id=17513709

Sure, but the reported issues are usually not nearly that severe.

Most of the time what I see is

"A dependency of a dependency of a dependency of Webpack is vulnerable to a Regular expression Denial of Service attack" or prototype pollution or something like that.

Webpack generates code which might be loaded by users and additionally it's very common to run it on a CI server (which can sometimes have network access out to other machines at a firm).

In general, it's quite strange to me that vulnerabilities in `devDependencies` are considered less important than those in `dependencies`. These dependencies are generally for tools that are run within your company network, and contrary to what people insist, that seems quite risky to me.

What would be the attack vector in this case? Assuming that it's a vulnerability rather than an actually malicious package, how would an attacker exploit something running on your CI server? The only way I see is if they already have the ability to modify your source code, at which point it's of course already game over.

"It's already game over", a favorite phrase of many, but a sign you are giving up before the race has even started. Security and insecurity theater at once.

While it's not always considered a risk, there are many cases where being able to run code on a CI server is a vulnerability.

But ReDoS does not give them that ability

You don't install your dependencies' devDependencies, so by definition 100% of the devDependencies on your the libraries you use are not a concern! I'm talking as a library author here (thought that was clear?), when a random vulnerability scanner marks one of my devDependencies as having an issue, that in principle won't affect the people using my library (except in some very very rare extreme cases).

Also, many times those "vulnerabilites" are not really applicable because the application code can not trigger it if it never ends up in the vulnerable code path (for example when a vulnerable regex is never passed untrusted input).

I imagine those cases could be detected using static analysis but the current vulnerability reporting tools that do not even look at actual code are completely useless in that regard.

How many of your customers, who can detect these same vulnerabilities, care about whether or not it's triggered? What about the fact that these are all latent bugs waiting to be hit with an approach like this?

It needlessly accumulates risk when you prioritize explaining why things don't need to be fixed rather than just lowering the bar to fixing them

Github's notification system is incredibly spammy if you have a lot of repos, and there's no obvious way to manage it. There's also a huge need for an "unsubscribe all" in the notification inbox.

There's also not a severity indicator - some minor issue not encountered in normal use is just as noisy as an extremely important issue that affects every user.

Other tools like Jira and Gerrit are far better at this.

The GitHub notifications story is really quite poor for developers, it's extremely difficult to get an alert routed only to the person who triggers it.

That's actually a very interesting point in regards to switching from StackOverflow to Github - I have noticed the trend that what I could normally find on Stackoverflow, I now often find on Github issues

Yeah, I wouldn't mind if they added an extra tab for Q&A, with additional features to make it work like a community wiki (like what SO tries to do).

Already in beta: http://github.com/vercel/next.js/discussions

Seeing many people surprised or like "finally someone said it", I thought this was very common knowledge? I might have been in some specific communities at the critical point 2-3 years ago, where bootcamps and other educators would advice new devs not to go to StackOverflow but instead go to Github. It didn't occur to me back then that the consequences would be so bad.

At some point many beginner devs migrated from StackOverflow to Github because their really bad question were being closed there, and now they just overwhelm open source authors

I'm glad it's not just me seeing this - My repos aren't even that popular and some of the issues just seem to be "help me build my project..."

I started just quoting them back prices when they send me random emails like that. What'd you think, they disappear quickly!

Now that's an idea I can get on board with.

I don't mind help in finding these problems, what I LOATHE though is when people trust the tool more than me and (for example) prevent me from pushing something that disagrees with the tool. So somewhere I imagine some manager will force their devs to make this tool happy and that's wrong.

It is tricky. I would say the perfect tool still needs to be developed. And I agree with you. Tools need to get better in detecting dependencies. However, it can be helpful to know at least that you rely on vulnerable code even if it does not get deployed to prod. To be automated, security scans are like unit tests. Just because a unit test is green does not mean your app is working and vice versa. So security scans are more like test levels: unit tests, integration tests, and end-to-end tests. Different scans, different results, and it takes us, humans, still to put it in perspective.

For every automated tool we need another automated tool that cancels first one.

> Open Source authors (including myself) have complained of automatic security scans. They yield way too many false positives, increasing the burden of maintaining repositories.

Unfortunately all tools have either false positives or false negatives, and in practice often both. Tools can (and should) take steps to minimize them or their impact. Nothing makes you use any specific tools; if you don't like what a tool does, don't use it.

> Specially troublesome are when e.g. the "vulnerability" (if it's even one) is in a devDependency that is not deployed to production.

This particular GitHub tool is for analyzing source code, and would not not normally analyze your dependencies. So that doesn't seem relevant in this case.

Of course, someone could mindlessly use this tool (or look at its results) and complain. That's easy, just ask for funding to fix the problem, or at least a pull request that fixes it.

We have this problem with our non- OS project but found that after the initial review (of which most were false-positives) we could permanently suppress them with a comment and fail the build for net-new work without a huge impact on developers.