Welcoming Semmle to GitHub
github.blog
github.blog
Basically, Semmle offers a static analysis tool that operates on your source code as a graph (from what I understand) and points out bugs and security holes in your code. Github is now offering that for free on repos at all tiers.
[0] https://github.blog/2019-09-18-securing-software-together/
For what it works for, it works nice. But it is not a pancaea.
Security vulnerability finding is almost certainly the wrong target for Semmle - I am unsure why they are trying to push that angle. There are much better stories in things like refactoring and understanding. (I say this having overseen a number of deployments for various reasons, some successful, some not)
(I mean this in terms of capability, not sudden popularity)
It would move the needle on a bunch else. It is a good tool for sure (and im very happy for them), i just think they will disappoint people by pressing this particular narrative, and wouldn't do so with a different narrative
With certain languages and a strong and diverse ruleset Semmle has it's strengths. In particular with native code (C, C++) and decent rules I have seen Semmle be very successful at finding certain classes of bugs.
https://help.semmle.com/lgtm-enterprise/user/help/generate-d... says "LGTM generates a database for each commit stored in a repository. Each database is a relational database that represents the structure of the codebase for a specific revision, or snapshot, of the code.", though a triple store could qualify as relational here. I couldn't find much more than that about the implementation details though.
I have been using Semmle daily to automate much of the vulnerability discovery process and I am extremely satisfied.
We run it over millions of lines of Java code and have not yet run into scale or perf problems.
Developing custom queries and defining security invariants in a logic language is, quite honestly, a joy.
Mountains were moved to make it scale, but that has been achieved. Semmle can scale with work - it just takes a lot of effort and code.
To simplify a bit you can think of most static analysis algorithms in terms of graph problems where nodes are statements and functions and edges are flow of control and calls. On large codebases the amount of edges, nodes, and calculated data is just too big to keep in memory. The trick is to break the graph intelligently into parts, calculate some sort of summary information for each of them, distributing between cpus or computers, move up to the supegraph of graphs and perform higher level calculations on it.
You're right though, it's not a panacea, and it could probably be great for other uses too.
Secondly, I know Microsoft loves it, which is presumably where your telemetry comes from, and I know a lot of security people on Twitter are fans of the technology, but I've been asking around and "love it" is not the signal I'm getting from software security blue team people. "I installed it, I guess it does some stuff, we never think about it" is the modal feedback I've seen.
I'm very interested in hearing success stories about this; the problem Semmle addresses is a huge part of the cost basis for my practice, and I'd love to hear that someone has gotten it working well.
I mean think about it, if you wanted to write a query against the AST of a target, would you find that useful? Or in a given codebase, if you find one bug, would you like the ability to capture that in a query that can tell you if a similar mistake was made elsewhere?
Out of the box, it isn’t going to give you much value. It’s the power of the query language, if it’s your job to do that, where you’ll see the benefits.
But don’t take my word for it, just try it out.
Their licensing model may be problematic for your use case though. I only vaguely understand what you do, but last I asked them about it, it’s not possible to get a personal license that a security person can use for multiple projects, and my read was that they had no interest in selling to individuals anytime soon.
Think of a query like: Find all calls to function A that have an output pointer-pointer of type B in the last argument position and also have a boolean return type, verify the input dereferenced B is NULL, then verify that iff A assigns output to B that the return type is false and that the calling frame also includes a later call to function C to clean up B. This type of thing run on millions of lines of code.
This can get as crazy as you want it to be and did work, but there is a “but.” The query language is so verbose and powerful, there were many, many ways to represent the same query that all had drastically different performance profiles. The docs were woefully underwhelming in that regard — they simply stated what a member did, but not anything at all related to memory or performance implications. That, coupled with the fact that nearly every complex query during dev hit the wall of the JVM killing it for taking too much time/mem, it became apparent that they must have tooling for the tooling to analyze performance and memory profiles of queries (not unlike all the effort put into SQL over the years). Also, no debugger; resorted to “datalog printf debugging” (ie, including internal clauses as outputs and chopping off later parts of the query...)
I spent a few weeks only writing queries and in every non-trivial one I needed a senior person there to review/rearrange a few clauses to go from e.g. 30 minute runtime to 15 seconds. That left me with a feeling that I was constantly fighting the docs and lack of tooling and would constantly need their help to tune things. Nothing was fundamentally wrong with the queries — it was just not documented anywhere that some filtering should be done caller->down vs. other kinds of checks in the same query should be callee->up.
I had questions regarding scaling up to 1B+ LoC like others in the thread but didn’t really get that far.
It did successfully find the C/C++ bugs I was looking for once the queries had hours of investment from both sides, and we were also able to find bugs in a large custom JS codebase by mocking all the things it would need to understand to eval the code. Whether or not that investment makes sense is an individual/team/org question, but if they ever seriously invest in a query debugger and profiling, they’ll be pretty hard to beat.
I've also met repeatedly about it with all Google customers over the past few months, Both those currently using it, and those that stopped.
Prior to that, I had met with most that used it but stopped, at the point they started using it. There were several large scale attempts/efforts to use semmle in various ways, by various teams.
I'm trying to be as nice as possible here, since, as I said, I think it is a great tool for a lot of things, and I've been a strong supporter for this kind of technology for those cases (for years, in fact, as i'm sure some folks at Semmle can tell you) so I'd rather not burn it all down, which I expect is what would happen if i did a point by point explanation of everything bad about it.
So I will instead reiterate my claim, and take my downvotes ;)
If you are comfortable sharing more, I’d be curious what you found that it struggles with, without burning it to the ground. It’s possible that different security teams use it in different ways, and it might be more suitable depending on your expectations. It’s also possible that the feedback I heard was during the honeymoon period, and practical issues outweigh the utility once you use it more.
But I’ve seen real 0-days found with it, first hand and second hand, so I’m having trouble reconciling that with your account that it’s not useful for security.
Sure, but Fermin was also offered a fairly ridiculous amount of money and a serious promotion :)
I mean, he doesn't not believe in it, of course, but i also think most folks would have taken the role in his situation.
IE it's not the kind of offer that really required a lot of faith
I'll try to write a bit more later after i think about how to frame it :)
If your data is closer to a model where security bug hunters whose sole job is to find vulnerabilities and audit code, and it was deemed not useful in that scenario, then yes, I am at odds with your claim. Admittedly, that’s a pretty niche set of customers. If you don’t learn Semmle QL, and you aren’t writing queries, it’s probably not for you.
Its run in both self-service (output to developers), guided (output to product security oncall of security engineers) and used ad-hoc to power up manual security reviews. Depending on the accuracy of each rule and the impact of the pattern of security flaw the rule finds it is promoted to ultimately output to developers directly.
It finds about a third of the security vulns we unearth each year.
I respect your point of view around our technology. You may like it or not (some folks love it), but please do not make statements about me you do not really know :)
And to be clear, I believe this technology makes security researchers scale on different aspects. At least I had first hand experience with this and our goal is to make security easy for non security folks... this technology enables us to do this.
Happy to sync in private over a coffee!
Static code analysis tooling is never the end all be all for vulnerability research, but it does let you express vulnerability patterns for implementation type vulnerabilities and find them at a mass scale (that is if your rules/queries are legit).
CVE-2019-5876
CVE-2019-16230
CVE-2019-16231
CVE-2019-16232
CVE-2019-16233
CVE-2019-16234
CVE-2019-15026
CVE-2019-14192
CVE-2019-14193
CVE-2019-14194
CVE-2019-14195
CVE-2019-14196
CVE-2019-14197
CVE-2019-14198
CVE-2019-14199
CVE-2019-14200
CVE-2019-14201
CVE-2019-14202
CVE-2019-14203
CVE-2019-14204
CVE-2019-14437
CVE-2019-14438
CVE-2019-14438
CVE-2019-14498
CVE-2019-14535
CVE-2019-14534
CVE-2019-14533
CVE-2019-14776
CVE-2019-14778
CVE-2019-14779
CVE-2019-14777
CVE-2019-14970
CVE-2019-15119
CVE-2019-14524
CVE-2019-14523
CVE-2019-7307
CVE-2019-11476
CVE-2019-13115
CVE-2019-3570
CVE-2019-13110
CVE-2019-13112
CVE-2019-13113
CVE-2019-13108
CVE-2019-13109
CVE-2019-13111
CVE-2019-13114
CVE-2019-3560
CVE-2019-9721
CVE-2019-9718
CVE-2019-9717
CVE-2019-9720
CVE-2019-9719
CVE-2018-20222
CVE-2019-3828
CVE-2019-6986
CVE-2019-5414
CVE-2018-4460
CVE-2018-16491
CVE-2018-16489
CVE-2018-16490
CVE-2018-19476
CVE-2018-19477
CVE-2018-19475
CVE-2018-19134
CVE-2018-16472
CVE-2018-18820
CVE-2018-4407
CVE-2018-16487
CVE-2018-4259
CVE-2018-4286
CVE-2018-4287
CVE-2018-4288
CVE-2018-4291
CVE-2019-5413
CVE-2018-16461
CVE-2018-16469
CVE-2018-16486
CVE-2018-16460
CVE-2018-16492
CVE-2018-11776
CVE-2018-8018
CVE-2018-8294
CVE-2018-4249
CVE-2018-8013
CVE-2018-5388
CVE-2018-1295
CVE-2018-4136
CVE-2018-4160
CVE-2018-1000140
CVE-2017-15692
CVE-2017-15693
CVE-2017-13904
CVE-2017-15089
CVE-2018-6834
CVE-2018-6835
CVE-2017-15713
CVE-2017-12634
CVE-2017-13782
CVE-2017-7545
CVE-2017-14949
CVE-2017-14868
CVE-2017-8046
CVE-2017-8045
CVE-2017-9805
CVE-2017-1000207
CVE-2017-1000208
CVE-2017-12612
CVE-2017-0141
Clearly, it works.
So it's more advanced than simple "know bad dependencies", but it's also not quite "new vulnerabilities".
I'd push something else for security, though, to complement it. RV-Match is my favorite commercial one because they built on a formal semantics for C, it's set for low false positives, and they open source a lot of stuff. They have something for Java and smart contracts, too. Past that, what's good depends on what language you use.
I'd rather a company like them not acquire them in favor of one that is constantly developing new services with no attempts to financially drain 3rd parties. Not to say good won't come out of Github integration given huge number of projects in it.
That's also not mutually exclusive of the emotive intent you are describing. What makes that Orwellian though?
Github appears to be going the aqui-hire route with Semmle, dependabot, pullpanda etc, where as I don't think Gitlab's made an acquisition for a year or two.
https://about.gitlab.com/devops-tools/ https://about.gitlab.com/stages-devops-lifecycle/secure/
It just uses existing open-source analysis tools, but orchestrates them all into a single tool by coordinating a bunch of docker images.
> The total purchase price of the deal, paid in cash, will not exceed $1M and will be the total and only compensation for the entire deal.
This implies that in addition to self-funded ventures, they are looking for fire sales from failed start-ups.
It would allow a small team of hackers to have a decent exit without having to go through the whole startup road.
The surprise is really that they're not being more aggressive in their acquisitions.
I've not heard from Yahoo in a year at least, do they still exist ...
Maybe that's not exactly what you were looking to "hear from" Yahoo about, though...
https://lgtm.com/projects/g/rust-lang/rust/snapshot/f5aa590b...
exist_ok is available from python 3.2, so this isn't a good impression.
Am sure this will bring some amazing advances to Github and thus a huge % of the developer community.
What a way to begin an article.
1) identified str.replace('[ABC]+', '') correctly as a bug (looks like a regex but is string literal)
2) identified various unnecessary code that TypeScript overlooked
3) identified double-unescaping of html (this one would have probably gone unnoticed for years)
And a bunch of other stuff. No actual vulnerability in our case, but still very useful. I'm enabling their checks on every future PR.
This was TypeScript but they support the rest of our stack too (Python, Java). I wonder if this includes Kotlin - will try.
(Non native speaker here). Am I misunderstanding something, or is the author explaining that humanity can not progress without the open source community?
Aspirational, as in, they wish that github would be the key factor in human progress. Maybe I can say that even plainer: to the leadership at github, the progress of the human race depends on github.
The open source community depends on github.
That's what it's implying.
(The reader is left to identify that as wishful thinking.)