Security scorecards for open source projects | Hacker News Reader

Security scorecards for open source projects

opensource.googleblog.com

37 pointsaberoham13 comments

Received these results for my own open source project:

    ./scorecard --repo=github.com/bookstackapp/bookstack
    <removed status text>
    RESULTS
    -------
    Active: Pass 10
    CI-Tests: Pass 8
    CII-Best-Practices: Fail 10
    Code-Review: Fail 10
    Contributors: Pass 10
    Frozen-Deps: Pass 10
    Fuzzing: Fail 3
    Pull-Requests: Pass 7
    SAST: Fail 0
    Security-Policy: Fail 10
    Signed-Releases: Fail 0
    Signed-Tags: Pass 10

Results appear fair and accurate. I am confused though in how this project is intended to work at a higher level. The blogpost states:

> The goal of the Scorecards project is to auto-generate a “security score” for open source projects to help users as they decide the trust, risk, and security posture for their use case.

Will there be a centralised site to gather and display scores for open source projects? Or will it be up to the open source projects themselves to integrate this into their pages and, if so, what does that look like? Some kind of badge or a listing of the results as above? Just trying to understand how end-users will be interpreting the results in a consistent and trusted manner.

We are still thinking of ways to capture scores for various open source projects in an automated fashion and/or work with GitHub to display this kind of information as part of their user interface. Right now, project maintainers need to run it themselves.

The usage example in the README gives Kubernetes a 10/10 on security. Would it score that highly if the tool took into consideration all of its dependencies? https://github.com/kubernetes/kubernetes/blob/master/vendor/...

Probably not. I think the intent is for big shops to quantify the risk of adopting an open source project. This is just a way to codify how teams justify whether a project is "mature enough" for adoption.

Appears to be an open source alternative to Black Duck Software.

https://www.blackducksoftware.com/

The goals are different. Our goal is to help both open source developer improve their project so that it is more secure for the community in the long term. It also gives the consumer this information for free, rather than paying / subscribing to a service.

there's an open issue related to this: https://github.com/ossf/scorecard/issues/27

It seems like an awful waste for me to run these on each of my dependencies in real time. I expected some list that is just maintained and pinged for values.

Also, this scorecard doesn’t look for CVEs or problems in particular versions. It seems like it’s much more important that there’s a valid vulnerability in version 1.04 that I’m using than the current version has code reviews for everything.

The reason I care is that I wish there was some stamp of approval on pypi packages that would make it easier for me to trust particular packages and releases.

I think you miss the point then. The things it's looking for are indicators security practices are even in place. If you wanted vuln scanning, many solutions for that today exist. This is almost like a reputation score. The higher it is, the more likely vulnerabilities will be addressed in the future (or outright prevented).

I like the idea, I think it will just be tough to work out the right heuristics.

Having a quantitative, ordinal scale gives the wrong picture I think. Since 10/10 sounds good, but doesn’t actually mean something is safe.

Also the name is “scorecard” that again connotes that a high score is good. I don’t get a 10/10 on my assignment because I did my homework, said hello to the teacher each morning, and was polite to my classmates.

Those are all positive things and if I wanted to measure students they may be a part of the grade. But if I only graded on these factors that’s not terribly useful.

Process measures are good when it’s not possible to measure outcome, but I think in this situation more outcome measures could be factored in.

These are just some initial metrics we picked for v1. Please suggest more by filing by a github issue here - https://github.com/ossf/scorecard/issues Over time, we will add more real time metrics such as open vulnerabilities, etc

Yes, we will automate score capturing over all projects in real-time in the future. https://github.com/ossf/scorecard/issues/12

The work to capture current vuln state is tracked in https://github.com/ossf/scorecard/issues/52

Cool, thanks for working on this.

Reply on news.ycombinator.com