Hacker News Reader

I want to first say: I agree with your comment.

I think the problem is that companies would say "okay, let's take the leaner path"; they then proceed to implement their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will.

(And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…)

But you're right with your hospital analogy: it's that the company does not want to put forth the resources to do the job right, either by doing it right themselves or by doing the verification work & upstreaming fixes; they'd rather put forward a bare minimum to do shoddy work.

And in all my years of experience, I still am no closer to understanding how to fix that.

> We need to start calling this out as the negligence that it is.

People absolutely hate this, IME.

> their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will.

The interesting trade-off here is that they probably aren't going to make the exact same mistakes as everyone else.

Which means more vulnerabilities, but fewer industry wide vulnerabilities where everyone gets pwned.

I'm not sure that's worth it, but it is something that should be mentioned when discussing the situation.

edit: me -> be

The third option is to pay someone else to certify things (maybe the developers of the thing, or a third party entirely).

The problem gets out if people find out “Caddy is certified by Google” and then don’t bother doing their own.

> (And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…)

This is a large part of why I expect the binary protocols HTTP/2 and HTTP/3 to be more reliable and less buggy than the textual HTTP/1 once all the dust settles: you won’t get far without reading at least parts of the specs, and they’re harder to implement, so there will be fewer implementations, with a higher average quality.

If you implement HTTP, surely your service is behind nginx. So it does not matter much, nginx is not going to send evil data to you.

> And in all my years of experience, I still am no closer to understanding how to fix that.

Does it need fixing? This may be a case of the cure being worse than the disease (the velocity loss outweighing the security risk).

> Companies must stop expecting others to do their job for them. OSS devs already wrote the code for free because, ostensibly, it was fun

This is not really true. Open source is much more than hobbyists doing things in their spare time for fun. A huge amount of open source is developed by and released by paid software professionals as part of their jobs. Some of those companies are directly doing it as part of their actual business offering. Others are doing it because they value owning the mind share in a space. Then, a huge amount of other OSS is developed by people who want to enhance their professional reputations.

EDIT: I see the article actually confines itself to OSS software that was written as a hobby. In that case I think my words above are probably out of context. But by the same token, the whole article scope is diluted a lot. Yes, if you have pinned your project on something someone wrote on the weekend for fun with no intention to maintain it, you've got problems. But let's not brand all of open source with that problem.

I generally advise my clients to mostly trust open source with lots of well known and documented professional eyes on it like reproducible builds of programming language compilers, standard libraries, and well maintained OS kernels.

Where I normally have them focus their resources is on the often thousands of dependencies that are, mostly, written has hobby projects by randos.

It doesn't matter if it's written as a hobby project or not. People need to stop making assumptions and just expecting work to be done for free for them.

No wonder why maintainers burn out and companies are hesitant to open-source projects... random people come by and make demands with zero plans to pay for it.

> random people come by and make demands with zero plans to pay for it.

Perhaps we should then oppose the whole "hustle culture" which such attempts are part of (i.e. make demands without willing to pay, see whether they stick).

I think the reality is, if its a hobby project supported by one person then money doesn't really solve it. No company should put itself in the position of being reliant on some random individual on the internet whether they are willing to pay them or not.

Of course, adopting this stance takes away opportunity for many small time open source projects to become popular in the first place, and the general incentive and opportunity for open source to succeed.

Money solves this by paying the original person or someone else to do what they need and provide the level of service and guarantees that they need. The whole point of open source is that it doesn't have to be the original developer.

I'm glad you brought this up. It's really core to the value of open-source as an activity. It creates the possibility for a project to become much greater than what it would have been if it was limited to the person or organization that originated it.

If one good thing can be said about the crypto sphere, it is that it is the only corner of the internet where nobody expects to get anything for free. Access is already gated by a wallet containing real money; you're going to use it. This leads to some surprisingly healthy dynamics around services and software that cost money to run but would be hard to charge for in web2.

This is a thought-provoking take (as someone who really thinks open source devs have no obligation to their users). It is clearly possible to abuse that -- just dump the software necessary to use your product into open source ecosystem, abdicating responsibility while still technically supporting your product.

I dunno. I'd say that this still doesn't establish a customer/provider type obligation -- the open source code is still offered without any quality guarantees -- but it is entirely fair (and reasonable) to be extremely skeptical of a company that doesn't offer a contract-defined paid relationship for support.

And part of ostensibly "corporate" OSS is the stuff that devs convinced management to put out there because they didn't want someone else to reinvent that particular wheel. It might be supported for as long as it's used internally (and if you're lucky, some devs will continue unofficially supporting it for a while after that - but at that point we're already into de facto hobby territory).

These are all good reasons to contribute to free software but I think you've missed the biggest one of all, the fundamental engine that makes the entire concept work: making the software you already use, more useful. If you need a feature or a bugfix, you can write it yourself, upstream it, and never worry about it again. That's a big part of why companies employ people to work on free software, and it applies just as much to individuals.

I frequently find myself advocating for better supply chain security. But if I asked developers (or their managers) to review/audit 2,000 NPM dependencies, I would almost certainly fail. No other company they know of is doing that, and asking them to start is predicated on the entire industry being wrong. That tends to be a tough sell - even though I agree with you, convincing others is a whole 'nother ball game!

What kind of arguments do the organizations you consult for find compelling? I find it extraordinarily difficult to convince others. There is a strong bias towards the status quo.

I often suggest that they either review it all, pay a company like mine to review it all, or do tree shaking to remove most of the deps and make it lean enough that they can afford to maintain reviews.

Failing those when a legacy codebase is just too big to simplify or review, I will generally pivot to retrofitting in accountability to something that can be trusted. This often means working with the team to design small, auditable, reproducibly buildable standalone services for security critical functions that can run in a TEE or HSM. These often take on critical signing, encryption, and policy enforcement responsibilities such that a compromise of any single employee or the larger legacy application can be tolerated.

Gotta have multiple knobs you can turn so the org can choose the cheapest path that honestly meets the required threat model.

It's a "thing" in the Medical Device industry. I don't recall if it's now mandated or merely suggested in the recent FDA Cybersecurity Guidelines (still a Draft, not yet a Regulation), but creating a Software Bill of Materials for OTS/SOUP that's in your project is now recommended. Doing this allows you to track and analyze the impact of discovered vulnerabilities that may be in your shipped product.

I'm far from a security expert, but the concept of creating and maintaining an SBOM seems to be gaining traction across a number of industries.

Why do you think this is controversial? Whether a company works with another software company that is bonded and/or a person uses OSS if something bad happens to the customer it still is reflective on the company using the software in a negative way. No company would refute that.

I rarely have seen court cases in regard to customer damage try to quantify negligence, because the court system is missing a lot of nuance in our industry. Pragmatically speaking the courts are ruling on the severity of the customer impact. There can and will always be an argument that is subjective about negligence in regard to how much you protect yourself from a malicious event vs the severity of said event. This isn't specific to software engineering either like concert venues that are mishandled and result in accidental death.

Your comments around npm dependencies not being reviewed and shows an engineering team is negligent seem contextually correct depending on the damage of said system the engineers are managing. If it's a bank system that leads to fraud then I agree. If it's a start up that runs a website; I hardly categorize this as negligent. Every company I have worked for has understood this trade off. If you are trying to be over zealous about the definition of negligence then I could understand how that would be controversial.

Do you think CTOs should be held criminally negligent if they lose data that is of national security concern?

Because that’s the only way I see us getting audited dependencies as a commonality.

I would agree fintech companies, or any company managing a lot of PII, must take supply chain security much more seriously than say a fashion blog or a video game. The level of negligence is anchored to the potential for harm, but it is IMO a rare case where a successful company manages without a lot of PII or payment details. Even Deezer, who /only/ sells booze, was sued for a data breach and had to give out thousands of $10 checks in a class action. Their negligence hurt them and their users.

For context, most of my clients are high risk with large PII footprints or various forms of fintech. Even in fintech and banking, dependency code review is unheard of, and supply chain attacks are happening in the wild targeting those orgs.

I kind of do intend to come off alarmist about this, because it is very alarming and is likely going to get a lot more people harmed than it already has.

I think you meant to refer to drizly. I would say this is an example of a company that put their priorities on growth rather than security and it worked out for them. It sounds like Drizly didn't think about security at all, and in the end it cost them 0.6% (worst case they settled for $7M and were aquired for over $1.1B) of their value. Looks like their executive team prioritized the right things to me.

Making a decision for or against more security is more about risk mitigation. If the courts are just going to slap companies on the wrists for data breaches I don't see a strong argument for intense security protocols for your run of the mill e-commerce business.

Ha ha. Yes. Drizly is what I meant to say. Too late to edit now.

I would say it caused them reputational harm as well. It would have likely been a lot less trouble to just hire a capable security engineer or two and do some basics.

To your point, we need these things to hurt a lot more, but it is a start.

People have been conditioned to ignore security. Too many big public incidents, too many emails telling them their data was exposed. They don't care anymore.

I think you're fighting an uphill battle here.

It took something like 100 years to normalize washing hands in the medical industry. I am in this for the long game.

Well of course security consultants believe every company should hire security consultants to review every dependency in the stack all the way down.

But for the people who sign the checks, do you have ROI numbers to support this approach?

How many exploitable vulnerabilities in open source dependencies that could have led to financial loss have your reviews found that an off-the-shelf SCA tool didn't find?

FWIIW I advocated for external review of security critical dependencies and hired external firms for this long before I started my own firm.

The ROI is not having your company ended by someone malicious that reviewed that code before someone acting in your interest does. That said, I normally work with companies that -really- need users to trust their security practices so this can become a sales and marketing win. When competitors are forced to admit they do not review the code that handles customer data or money, and you do because it is obviously the responsible thing to do... it helps customers choose you.

As to your practical example question, many companies depend on Terraform, for example. Until I reviewed the AWS password generation function it was producing passwords seeded by the current time. An attacker that knew what time an account was created had a path to guessing the password.

https://nvd.nist.gov/vuln/detail/CVE-2018-9057

Do we find things this impactful super often? No. But when we do find them all of our clients can get early warning and be recommended general practices that avoid this type of issue entirely.

We have also privately exploited supply chain attacks when legally possible more than once as demonstrations for our clients.

While the best examples are public, you do not have to look hard to realize supply chain attacks are all too often -easy-.

Researchers did a scan of all npm packages who have maintainer email addresses with expired domains. AKA packages anyone can buy their way into control of.

https://arxiv.org/abs/2112.10165

Thousands of them. I bought the maintainer email domain of the NPM foreach package just to prove a point.

Generally though we spend most of our time doing blue team work. If you architect your application well with well placed secure enclaves you can get away with any single system or human being comprised without a loss of user data or funds. Then it becomes possible to more freely use libraries and move fast in other parts of your stack with lower risk.

Reviewing 100% of all deps company-wide is not the only knob one can turn.

I have two thoughts to share about this from what you've said:

> we must stop expecting [Open Source] devs to have any responsibility for code they produce.

Yes, I agree with this. And something that's been in my head for a while, which often discussed, is how people often "magically think" something along the lines of:

"If we just paid these devs for support, then we'd solve the support problem!"

Unfortunately, I don't think is reality. If you're a dev working at Google and you toss a library on NPM, you probably can't be paid to support it because your employer forbids it.

The type of person writing libraries Open Source is not always going to be motivated by money!

> Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path.

There was a post a few months ago[0] that has been talking about a way to "democratize" this review process. It's been sitting in my head and it's actually a part of a project that I'm working on now[1]. (My apologies for that not being documented anywhere yet though!)

Essentially, if every company on this planet has to review every Open Source dependency, that's a massive waste of human effort because there is a ton of overlap between that work.

Instead, what if you could have one person/entity do that work to review the dependencies and then you "buy" that dataset for a fraction of the price? If it takes 10 humans a year to review every dependency you have, then it's cheaper to pay 1 human salary to buy that data. (And we do the work once but then sell it 100+ times.)

That's the core principle that's been in my head for a while and that's how we're trying to build a company right now.

Unsexy problems aren't very "shiny" (like Stable Diffusion or NFTs) but they are important!

0: https://news.ycombinator.com/item?id=32037562

1: https://github.com/lunasec-io/lunasec

I also am spinning up some projects and tooling to have standardized, cryptographically signed, language agnostic, vcs agnostic, and publicly publishable/searchable code reviews for any open source project.

I have been thinking about this deeply for years, built internal tools like this for companies, and been pleading with package manager maintainers for collaboration.

Some of my clients have made said they would strongly consider publishing internal dependency reviews to such a platform if it existed.

Sounds like we have some common goals. Would love to keep in touch on new developments. Contact info on https://lance.dev

Have you looked at crev (see my other comment)?

I have. I actually worked a few desks down from dpc when he was creating it and we talked about it at length. I felt then and now that it has good goals but a very limiting implementation in that it does not pursue a portable spec and instead anchors a very opinionated format to git, and github, instead of cryptographic keys held in hardware controlled by reviewers. I want to see the same keys that sign git commits also sign reviews, for instance.

I think for broad adoption a review system should ask essentially the same questions as crev, but store them in a format like in-toto including signatures by the reviewers created with a user choice of pgp smartcards, ssh keys, or webauthn devices. These reviews would be anchored to hashes of a particular state of a particular tree of code and not to any type of VCS or distribution system. Important code is distributed via perforce, mercurial, cvs, subversion, and tar files depending if we are talking about big corps, old codebases, or linux distro building blocks. A good OSS review system should be also be usable by teams in their internal proprietary codebases too if we wish to see wide adoption. Even for OSS we may wish to share some reviews as standalone objects privately while security embargos are in place, etc. Proofs should also be verified standalone easily from local cache, when github is down, when original repos vanish, etc.

Something that meets these broader needs will make it easy for large orgs with very different internal setups to participate and play nice with other supply chain security tooling efforts by the OpenSSF using in-toto for Sigstore, Witness, etc.

My experience and visibility into internal review tooling of multiple companies tells me many need something very different than crev, but crev proved to me many people have real interest in this problem which I really thank dpc for.

The biggest blocker for starting this project is the human review spec settling in in-toto https://github.com/in-toto/attestation/issues/77

Some relevant discussion here too: https://gist.github.com/lrvick/d4b87c600cc074dfcd00a01ee6275...

Thats a great criticism of crev that definitely makes sense.

I guess you may have also read about `cargo vet`? I feel like that is a worse model than crev, but is likely to win over it unfortunately.

https://lwn.net/Articles/897435/

Like crev I feel cargo-vet is solving the right problem, and I am glad it exists as a stopgap today to encourage more review, but both designs lack portability and cryptographic signing.

They fail to address a major problem in supply chain security which is that malicious actors often get write access to repos, or their mirrors, allowing for fake commits or reviews.

Cargo itself, like most language package managers, shows complete disregard for supply chain security from the unsigned curl/bash recommended installation process. When I have needed trusted reproducible builds of cargo in my supply chains, I have had no choice but to build it myself, only to find you need a recent version of cargo to build cargo. It is a circular nightmare.

Debian at least does mostly best effort reproducible builds and signs them, so it seems like the least bad option today for security critical rust projects to build with. Linux/BSD package managers that rely on web of trust signing and reproducible builds often feel like the only responsible adults in the whole industry wide software supply chain story.

I have talked to multiple package manager teams and there is almost universal hate of the very concept of author/distributor/reviewer cryptographic signing citing that it is so hard for participants to learn that it even being offered as optional would put people off.

IMO if someone lacks will to tap a blinking Yubikey, TouchID, or similar to sign their security reviews and commits, they likely should not be doing security reviews or distributing security critical tooling.

My hope is to make signed review tooling easy enough that people will run out of excuses not to use or emulate it.

Thats incredible what you say about packaging people hating signing, I come from the Linux distro space (Debian) where support for it is almost universal. I OpenPGP sign all my git pushes, commits, tags, tarballs and uploads to Debian.

BTW, you might want to look into the Bootstrappable Builds project. They are working on starting from ~512 bytes of machine code plus a ton of source all the way up to a full distro. Started by Guix folks, hopefully it will eventually trickle down to more mainstream distros. Apparently rustc is bootstrappable via mrustc, which only requires a C/C++ toolchain. There is some discussion of that in the comments on an LWN article about the Rust/GCC projects.

https://bootstrappable.org/ https://lwn.net/Articles/841924/ https://lwn.net/Articles/907405/

The comment on review overlap reminds me of crev, a socially scalable review system:

https://github.com/crev-dev/

That's what TideLift's goals are too. https://tidelift.com/

Hey, it sounds like you'd like my essay on this: The Asymmetry of Open Source: https://matt.life/writing/the-asymmetry-of-open-source

I agree that the responsibility for quality and security of open-source lies with the consumers of that code. But legal liability for negligence is another thing altogether. If the ultimate customer or user wants their vendor to accept liability for failures or defects then they need to negotiate that up front in a binding contract. If customers don't want to pay for that then that's their problem, they can take whatever garbage their vendors shovel out and then deal with the consequences.

Even shorter:

An entity gets to demand exactly as much from me as we agreed it could demand.

My giving it more does not mean it can demand more.

Therefore, without an agreement, nobody is allowed to demand anything of me.

(Laws are another matter, of course, unless you bring the social contract into it.)

> Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something focused on your needs on stop of the standard library yourself.

Why stop at the standard library? What about the rest of the language toolchain? Compilers, interpreters. What about the OS?

Those much more often have full time paid people whose actual job is to review them. E.g the standard library of Go has security and quality review by Google. It is reasonable to assume they are investing more in supply chain security than you are in those portions.

The random dep you grabbed that some hobbyist hammered out in a weekend... may not have 2FA on github, or maybe their email domain is about to expire, and it is very likely no one reviewed it.

If you choose to use the latter because it does what you need, then you are now the first party accepting the additional responsibilities of reviewing and maintaining that code to the level of safety appropriate for your use case.

If you are making a video game for personal use, then maybe the worst case scenario is tolerable for you to not care. If you are handling PII or money of other people... then suddenly you /must/ care.

He who cashes the check shall be the defendant in the lawsuit. If things went well the people behind the compiler, interpreter and OS wouldn't have received a dime from your successful enterprise, now that it's gone sideways you want to share?

The idea is risk reduction, not removal.

An OS is backed by thousands of developers, and usually billions of dollars. Theres negligible risk that the momentum stops.

If that OSS library, that is fundamental to your business, has three developers, then maybe you do have a problem.

I think that a lot of people should be asking themselves about the state of vulns and other bugs in linux. In fact, I'd call the state of correctness of the linux kernel and surrounding systems to be something resembling an industry emergency. It shouldn't be possible for bugs to regress in the kernel, but they do! A lot of very ordinary best practices aren't actually followed in the kernel world.

The linux kernel is a security shit show, but it is the only compatible option in most cases. You can at least strip it to the bone, apply Kernel Self Protection Project guidelines, and run it offline in TEEs or HSMs for your most sensitive operations. This is approaching as good as we will ever get.

Linux is a massive C codebase one can never /fully/ trust or harden and high security applications with sufficient resources should favor formally verified microkernels like SeL4 whenever possible.

It is true that Linux rules the world and is absolutely the right platform for a huge number of projects. I'm not saying "Linux is bad, reach for something else." I'm saying "Linux is the foundation for basically the entire computing landscape so the industry needs to take a real hard look at finding a way to make it safe."

I think it can be done. But I think it needs industry-wide effort in a huge way.

I think it is going to be much easier to produce a formally verified Linux ABI compatible microkernel than to ever fix Linux itself at this point.

> on top of the standard library

The standard library should also not be trusted by default, but it doesn't change as frequently which I imagine was what you were thinking.

>Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something in-house focused on your needs on top of the standard library.

That's not the comparison, the comparison is review an open source program vs. implement + review an in-house program.

When I read these kinds of sentiments about OSS I always think of Windows desktops and Internet Explorer in places where they absolutely don't belong. Yes, it is bad to have an enormously deep and complex stack with a big fat attack surface, almost none of which is relevant to the application. That badness isn't mitigated by the stack being commercial and coming from a single vendor.

> we must stop expecting devs to have any responsibility for code they produce. The responsibility is on those that consume it.

Why talk about the future ?

Microsoft, Google and other SW makers do not take responsibility for the code they produce. "Oops, we have an RCE. We're sorry. Please update "

> If you make no reasonable attempt at vetting your supply chain and harm comes to users as a result, then IMO you should be liable for negligence just like a restaurant serving food with poisonous ingredients.

You trying to kill FOSS for end users here?

> If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies.

Occasional security issues can just be the cost of doing business. And that's OK as long as your company isn't doing something critical like running the banking system. Needing to restore your shopping site from backup isn't the end of the world.

There's also the separate and even rarer problem of possible malicious non-internet dependant code in dependencies which would be relevant if running an xray machine or a nuclear power plant (e.g. things disconnected from the internet) but Ive never heard of a malicious non-internet dependant (so not something that tries to connect to a remote server) code in a dependency.

FOSS for end users is free and their own personal responsibility so I never see an issue here. When a business runs the code for others, then responsibility and liability kicks in.

Per my other comments, fintech we can all agree on, but if your organization manages PII then it has an obligation to make a best effort to ensure randos from the internet cannot execute any code they want against the databases.

If your organization has no PII and no financial exchange then I would agree the risk and need for review is much lower, but if those things are true then it is probably a nonprofit or a hobby.

> on top of the standard library

What makes the standard library so special? Or do you advocate fully developing that in house as well?

See my other comments on this. I tend to have people focus their cycles on easy targets for supply chain attacks such as libraries where there is no evidence -anyone- is doing review.

Most programming language standard libraries generally have at least some first and third party reviews from large organizations like Google, etc. That may not be perfect, but it makes those a much more difficult target than phishing some student programmers github account.

How did you get into this type of consulting?

My software engineering and sysadmin background overlapped a lot with security, and I found flaws and recommended security solutions at every company... so I pivoted my career to full time security engineering for multiple companies. I eventually saw enough massive oversights and targeted attacks to operate as though any person or system that lacks strong accountability for their every action is compromised.

Most companies can not afford to meet this kind of zeroish-trust threat model, so I moved to roles in fintech companies where they -must- think this way as they are highly targeted.

I realized though the highest value I provided to employers in addressing security problems happens in the first few weeks, then spot check pentests and advice a few hours a month after that while I build tooling. I also got tired of writing internal-only security tooling and practices I knew so many companies really need. I concluded the only assured way for me to be able to open source my tools and practices and get them refined by collaboration and exposure with a lot more organizations was to start a company with that mission.

I founded Distrust and my full time employer at the time graciously agreed to be my first part time retainer client. Turns out, many companies want pentesting and full stack security consulting retainers where someone can integrate into their team and help architect practical solutions unique to each company. Even companies with their own security teams benefit from a part timer with a perspective on security problems and solutions at many other companies with similar threat models.

Word of mouth has kept my schedule full enough to raise rates and justify building a small team last year. Best career choice I ever made.