There is no “software supply chain”
iliana.fyi
iliana.fyi
This will sound pretty harsh, but if your company chooses to use open source code that does not have capable, paid, full time professionals reviewing it for security and quality, then your company is signing up for that responsibility. If you make no reasonable attempt at vetting your supply chain and harm comes to users as a result, then IMO you should be liable for negligence just like a restaurant serving food with poisonous ingredients. Broadly normalized negligence is still negligence.
This should not be controversial, but it is. Washing hands in hospitals was once controversial too but those advocating for it had irrefutable evidence on their side. The medical industry did not want to pay the labor cost of hygiene, and we are seeing the same in the software industry.
https://www.nationalgeographic.com/history/article/handwashi...
Ask yourself if it cheaper to fully review, sign, compile, and maintain third party OSS code or to write something in-house focused on your needs on top of the standard library. Pick one. Both are real options. Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path.
Companies must stop expecting others to do their software review job for them. OSS devs already wrote the code for free because, ostensibly, it was fun. You are an ass if you ask them to do anything that is not fun, for free, to make your company safer or more money. Such actions make it not fun anymore, and make them stop entirely.
I do not know why companies have code review policies for code written by peers, but if the code is 2 million lines of NPM dependencies essentially copy/pasted from randos on the internet it is suddenly okay to ship straight to prod and give said randos full control of the data or property of millions of people.
We need to start calling this out as the negligence that it is.
Why stop at the standard library? What about the rest of the language toolchain? Compilers, interpreters. What about the OS?
The random dep you grabbed that some hobbyist hammered out in a weekend... may not have 2FA on github, or maybe their email domain is about to expire, and it is very likely no one reviewed it.
If you choose to use the latter because it does what you need, then you are now the first party accepting the additional responsibilities of reviewing and maintaining that code to the level of safety appropriate for your use case.
If you are making a video game for personal use, then maybe the worst case scenario is tolerable for you to not care. If you are handling PII or money of other people... then suddenly you /must/ care.
An OS is backed by thousands of developers, and usually billions of dollars. Theres negligible risk that the momentum stops.
If that OSS library, that is fundamental to your business, has three developers, then maybe you do have a problem.
Linux is a massive C codebase one can never /fully/ trust or harden and high security applications with sufficient resources should favor formally verified microkernels like SeL4 whenever possible.
I think it can be done. But I think it needs industry-wide effort in a huge way.
Most companies can not afford to meet this kind of zeroish-trust threat model, so I moved to roles in fintech companies where they -must- think this way as they are highly targeted.
I realized though the highest value I provided to employers in addressing security problems happens in the first few weeks, then spot check pentests and advice a few hours a month after that while I build tooling. I also got tired of writing internal-only security tooling and practices I knew so many companies really need. I concluded the only assured way for me to be able to open source my tools and practices and get them refined by collaboration and exposure with a lot more organizations was to start a company with that mission.
I founded Distrust and my full time employer at the time graciously agreed to be my first part time retainer client. Turns out, many companies want pentesting and full stack security consulting retainers where someone can integrate into their team and help architect practical solutions unique to each company. Even companies with their own security teams benefit from a part timer with a perspective on security problems and solutions at many other companies with similar threat models.
Word of mouth has kept my schedule full enough to raise rates and justify building a small team last year. Best career choice I ever made.
This is not really true. Open source is much more than hobbyists doing things in their spare time for fun. A huge amount of open source is developed by and released by paid software professionals as part of their jobs. Some of those companies are directly doing it as part of their actual business offering. Others are doing it because they value owning the mind share in a space. Then, a huge amount of other OSS is developed by people who want to enhance their professional reputations.
EDIT: I see the article actually confines itself to OSS software that was written as a hobby. In that case I think my words above are probably out of context. But by the same token, the whole article scope is diluted a lot. Yes, if you have pinned your project on something someone wrote on the weekend for fun with no intention to maintain it, you've got problems. But let's not brand all of open source with that problem.
Where I normally have them focus their resources is on the often thousands of dependencies that are, mostly, written has hobby projects by randos.
No wonder why maintainers burn out and companies are hesitant to open-source projects... random people come by and make demands with zero plans to pay for it.
Perhaps we should then oppose the whole "hustle culture" which such attempts are part of (i.e. make demands without willing to pay, see whether they stick).
Of course, adopting this stance takes away opportunity for many small time open source projects to become popular in the first place, and the general incentive and opportunity for open source to succeed.
I dunno. I'd say that this still doesn't establish a customer/provider type obligation -- the open source code is still offered without any quality guarantees -- but it is entirely fair (and reasonable) to be extremely skeptical of a company that doesn't offer a contract-defined paid relationship for support.
What kind of arguments do the organizations you consult for find compelling? I find it extraordinarily difficult to convince others. There is a strong bias towards the status quo.
Failing those when a legacy codebase is just too big to simplify or review, I will generally pivot to retrofitting in accountability to something that can be trusted. This often means working with the team to design small, auditable, reproducibly buildable standalone services for security critical functions that can run in a TEE or HSM. These often take on critical signing, encryption, and policy enforcement responsibilities such that a compromise of any single employee or the larger legacy application can be tolerated.
Gotta have multiple knobs you can turn so the org can choose the cheapest path that honestly meets the required threat model.
I'm far from a security expert, but the concept of creating and maintaining an SBOM seems to be gaining traction across a number of industries.
But for the people who sign the checks, do you have ROI numbers to support this approach?
How many exploitable vulnerabilities in open source dependencies that could have led to financial loss have your reviews found that an off-the-shelf SCA tool didn't find?
The ROI is not having your company ended by someone malicious that reviewed that code before someone acting in your interest does. That said, I normally work with companies that -really- need users to trust their security practices so this can become a sales and marketing win. When competitors are forced to admit they do not review the code that handles customer data or money, and you do because it is obviously the responsible thing to do... it helps customers choose you.
As to your practical example question, many companies depend on Terraform, for example. Until I reviewed the AWS password generation function it was producing passwords seeded by the current time. An attacker that knew what time an account was created had a path to guessing the password.
https://nvd.nist.gov/vuln/detail/CVE-2018-9057
Do we find things this impactful super often? No. But when we do find them all of our clients can get early warning and be recommended general practices that avoid this type of issue entirely.
We have also privately exploited supply chain attacks when legally possible more than once as demonstrations for our clients.
While the best examples are public, you do not have to look hard to realize supply chain attacks are all too often -easy-.
Researchers did a scan of all npm packages who have maintainer email addresses with expired domains. AKA packages anyone can buy their way into control of.
https://arxiv.org/abs/2112.10165
Thousands of them. I bought the maintainer email domain of the NPM foreach package just to prove a point.
Generally though we spend most of our time doing blue team work. If you architect your application well with well placed secure enclaves you can get away with any single system or human being comprised without a loss of user data or funds. Then it becomes possible to more freely use libraries and move fast in other parts of your stack with lower risk.
Reviewing 100% of all deps company-wide is not the only knob one can turn.
I rarely have seen court cases in regard to customer damage try to quantify negligence, because the court system is missing a lot of nuance in our industry. Pragmatically speaking the courts are ruling on the severity of the customer impact. There can and will always be an argument that is subjective about negligence in regard to how much you protect yourself from a malicious event vs the severity of said event. This isn't specific to software engineering either like concert venues that are mishandled and result in accidental death.
Your comments around npm dependencies not being reviewed and shows an engineering team is negligent seem contextually correct depending on the damage of said system the engineers are managing. If it's a bank system that leads to fraud then I agree. If it's a start up that runs a website; I hardly categorize this as negligent. Every company I have worked for has understood this trade off. If you are trying to be over zealous about the definition of negligence then I could understand how that would be controversial.
Because that’s the only way I see us getting audited dependencies as a commonality.
For context, most of my clients are high risk with large PII footprints or various forms of fintech. Even in fintech and banking, dependency code review is unheard of, and supply chain attacks are happening in the wild targeting those orgs.
I kind of do intend to come off alarmist about this, because it is very alarming and is likely going to get a lot more people harmed than it already has.
Making a decision for or against more security is more about risk mitigation. If the courts are just going to slap companies on the wrists for data breaches I don't see a strong argument for intense security protocols for your run of the mill e-commerce business.
I would say it caused them reputational harm as well. It would have likely been a lot less trouble to just hire a capable security engineer or two and do some basics.
To your point, we need these things to hurt a lot more, but it is a start.
I think you're fighting an uphill battle here.
I think the problem is that companies would say "okay, let's take the leaner path"; they then proceed to implement their own (say) HTTP library, riddled with bugs which will never get fixed, because it will never have the same person-hours sunk into it that a (major/respectable) FOSS HTTP library will.
(And the number of times I've watched a dev attempt to implement some standard while steadfastly refusing to read the standard. Then I proceed to find bug after bug, trivially … because I am reading the standard…)
But you're right with your hospital analogy: it's that the company does not want to put forth the resources to do the job right, either by doing it right themselves or by doing the verification work & upstreaming fixes; they'd rather put forward a bare minimum to do shoddy work.
And in all my years of experience, I still am no closer to understanding how to fix that.
> We need to start calling this out as the negligence that it is.
People absolutely hate this, IME.
The interesting trade-off here is that they probably aren't going to make the exact same mistakes as everyone else.
Which means more vulnerabilities, but fewer industry wide vulnerabilities where everyone gets pwned.
I'm not sure that's worth it, but it is something that should be mentioned when discussing the situation.
edit: me -> be
The problem gets out if people find out “Caddy is certified by Google” and then don’t bother doing their own.
This is a large part of why I expect the binary protocols HTTP/2 and HTTP/3 to be more reliable and less buggy than the textual HTTP/1 once all the dust settles: you won’t get far without reading at least parts of the specs, and they’re harder to implement, so there will be fewer implementations, with a higher average quality.
Does it need fixing? This may be a case of the cure being worse than the disease (the velocity loss outweighing the security risk).
The standard library should also not be trusted by default, but it doesn't change as frequently which I imagine was what you were thinking.
> we must stop expecting [Open Source] devs to have any responsibility for code they produce.
Yes, I agree with this. And something that's been in my head for a while, which often discussed, is how people often "magically think" something along the lines of:
"If we just paid these devs for support, then we'd solve the support problem!"
Unfortunately, I don't think is reality. If you're a dev working at Google and you toss a library on NPM, you probably can't be paid to support it because your employer forbids it.
The type of person writing libraries Open Source is not always going to be motivated by money!
> Some of my clients actually do (or pay others for) security review of every single NPM dependency they use in prod. If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies. Find a leaner path.
There was a post a few months ago[0] that has been talking about a way to "democratize" this review process. It's been sitting in my head and it's actually a part of a project that I'm working on now[1]. (My apologies for that not being documented anywhere yet though!)
Essentially, if every company on this planet has to review every Open Source dependency, that's a massive waste of human effort because there is a ton of overlap between that work.
Instead, what if you could have one person/entity do that work to review the dependencies and then you "buy" that dataset for a fraction of the price? If it takes 10 humans a year to review every dependency you have, then it's cheaper to pay 1 human salary to buy that data. (And we do the work once but then sell it 100+ times.)
That's the core principle that's been in my head for a while and that's how we're trying to build a company right now.
Unsexy problems aren't very "shiny" (like Stable Diffusion or NFTs) but they are important!
I have been thinking about this deeply for years, built internal tools like this for companies, and been pleading with package manager maintainers for collaboration.
Some of my clients have made said they would strongly consider publishing internal dependency reviews to such a platform if it existed.
Sounds like we have some common goals. Would love to keep in touch on new developments. Contact info on https://lance.dev
I think for broad adoption a review system should ask essentially the same questions as crev, but store them in a format like in-toto including signatures by the reviewers created with a user choice of pgp smartcards, ssh keys, or webauthn devices. These reviews would be anchored to hashes of a particular state of a particular tree of code and not to any type of VCS or distribution system. Important code is distributed via perforce, mercurial, cvs, subversion, and tar files depending if we are talking about big corps, old codebases, or linux distro building blocks. A good OSS review system should be also be usable by teams in their internal proprietary codebases too if we wish to see wide adoption. Even for OSS we may wish to share some reviews as standalone objects privately while security embargos are in place, etc. Proofs should also be verified standalone easily from local cache, when github is down, when original repos vanish, etc.
Something that meets these broader needs will make it easy for large orgs with very different internal setups to participate and play nice with other supply chain security tooling efforts by the OpenSSF using in-toto for Sigstore, Witness, etc.
My experience and visibility into internal review tooling of multiple companies tells me many need something very different than crev, but crev proved to me many people have real interest in this problem which I really thank dpc for.
The biggest blocker for starting this project is the human review spec settling in in-toto https://github.com/in-toto/attestation/issues/77
Some relevant discussion here too: https://gist.github.com/lrvick/d4b87c600cc074dfcd00a01ee6275...
I guess you may have also read about `cargo vet`? I feel like that is a worse model than crev, but is likely to win over it unfortunately.
They fail to address a major problem in supply chain security which is that malicious actors often get write access to repos, or their mirrors, allowing for fake commits or reviews.
Cargo itself, like most language package managers, shows complete disregard for supply chain security from the unsigned curl/bash recommended installation process. When I have needed trusted reproducible builds of cargo in my supply chains, I have had no choice but to build it myself, only to find you need a recent version of cargo to build cargo. It is a circular nightmare.
Debian at least does mostly best effort reproducible builds and signs them, so it seems like the least bad option today for security critical rust projects to build with. Linux/BSD package managers that rely on web of trust signing and reproducible builds often feel like the only responsible adults in the whole industry wide software supply chain story.
I have talked to multiple package manager teams and there is almost universal hate of the very concept of author/distributor/reviewer cryptographic signing citing that it is so hard for participants to learn that it even being offered as optional would put people off.
IMO if someone lacks will to tap a blinking Yubikey, TouchID, or similar to sign their security reviews and commits, they likely should not be doing security reviews or distributing security critical tooling.
My hope is to make signed review tooling easy enough that people will run out of excuses not to use or emulate it.
BTW, you might want to look into the Bootstrappable Builds project. They are working on starting from ~512 bytes of machine code plus a ton of source all the way up to a full distro. Started by Guix folks, hopefully it will eventually trickle down to more mainstream distros. Apparently rustc is bootstrappable via mrustc, which only requires a C/C++ toolchain. There is some discussion of that in the comments on an LWN article about the Rust/GCC projects.
https://bootstrappable.org/ https://lwn.net/Articles/841924/ https://lwn.net/Articles/907405/
What makes the standard library so special? Or do you advocate fully developing that in house as well?
Most programming language standard libraries generally have at least some first and third party reviews from large organizations like Google, etc. That may not be perfect, but it makes those a much more difficult target than phishing some student programmers github account.
That's not the comparison, the comparison is review an open source program vs. implement + review an in-house program.
An entity gets to demand exactly as much from me as we agreed it could demand.
My giving it more does not mean it can demand more.
Therefore, without an agreement, nobody is allowed to demand anything of me.
(Laws are another matter, of course, unless you bring the social contract into it.)
You trying to kill FOSS for end users here?
> If you can not afford to review 2000 dependencies then you can not afford 2000 dependencies.
Occasional security issues can just be the cost of doing business. And that's OK as long as your company isn't doing something critical like running the banking system. Needing to restore your shopping site from backup isn't the end of the world.
There's also the separate and even rarer problem of possible malicious non-internet dependant code in dependencies which would be relevant if running an xray machine or a nuclear power plant (e.g. things disconnected from the internet) but Ive never heard of a malicious non-internet dependant (so not something that tries to connect to a remote server) code in a dependency.
Per my other comments, fintech we can all agree on, but if your organization manages PII then it has an obligation to make a best effort to ensure randos from the internet cannot execute any code they want against the databases.
If your organization has no PII and no financial exchange then I would agree the risk and need for review is much lower, but if those things are true then it is probably a nonprofit or a hobby.
Why talk about the future ?
Microsoft, Google and other SW makers do not take responsibility for the code they produce. "Oops, we have an RCE. We're sorry. Please update "
When you decided to use it, you took on all of its technical debt and design decisions as your own. You can't fault the author if you use it in an environment they didn't have, or didn't intend it to be used in, or in a manner that wasn't intended. They left an artifact, they did NOT make an agreement to support it.
If someone makes a mistake in a book, there's no obligation for the author to correct it. If someone misunderstands the ideas and goes on to do some horrible deeds using it as justification, it's not the author's fault.
Open source software is, for the most part, an unsupported gift economy. It isn't commercial, and it certainly isn't a chain... there's no delivery to be had... something was tossed over the wall, and it may never happen again.
We Tims have been waiting for the next episode of Hello Internet for a while now... it may never come. That is also the nature of open source software.
We are all living in a post-war era. That war was over whether and how much of the software in the world should be locked up behind proprietary licenses and paywalls. To win the war, Open Source had to produce software that was less awful than the proprietary software it was trying to unseat. It did. Companies went out of business or adapted.
Now everyone is trying to pretend that the choice all along was between someone else's software and making your own. That wasn't the choice, and it's revisionist history to claim otherwise. A war was one, territory was occupied, and the 'treaty' at the end of that war was "make stuff that isn't awful", which is being peeled away layer by layer.
The above is the opening to my thesis as to why SaaS is now trying to eat Open Source: FOSS is abdicating responsibility for quality, and so the only way to get quality is to harass FOSS authors who think we are ungrateful whiners, or pay for improvements on the backs of paying customers who just want software that works, not to be told to put up or shut up.
The only sane way to defend against the re-privatization of FOSS software is to stop treating your users like they need to grow up and realize that you're deflecting. Unless authors accept that you have an ethical responsibility that comes with occupying a corner of the collective consciousness, it's going to be walled gardens everywhere you look.
We aren't paying people to 'steal' your code. We're paying people to act like professionals.
It's supposed to be a gift economy, not a plunder economy. You're given something for free, with little to no strings attached. All that is politely expected of you is to A) not be a dick, and B) reciprocate in kind, if the situation arises.
There never was a treaty (not even an informal one!).
In businesses, the reason why open source software was adopted was typically because it enabled cost savings (e.g. saving/reducing license costs).
Ecological systems or economic competition are far better metaphors, because they are emergent and decentralised, rather than patriotic and authoritarian. “Nature red in tooth and claw”, or monopoly rent and marginal price → 0.
Revisionist? The 1T$ Microsoft has not “lost the war”. The biggest FOSS wins have serious corporate sponsorship (OSes, databases, browsers, dev tools, languages, etcetera).
Linus is the most practical of the three, which is probably why we still talk about him more than the other two. But we wouldn’t talk about any of them if “practical” was the only adjective we had for them.
If you want to celebrate Microsoft becoming open source as an ally, knock yourself out. But the older people still don’t trust Microsoft (see also the Scorpion and the Frog), so it’s more of a war trophy. This is probably as “beaten” as MS gets, but they can always wake up.
I think you are assigning things broadly to "FOSS authors" that were explicitly disclaimed by most such authors from before companies started using FOSS.
FOSS won people over by saying we are entitled to useful software without having to mortgage ourselves to corporations. Well, it got what it wanted and now it calls users entitled. You’re damn right they are.
Can I download the sources and easily patch the gmail client on my phone as easily as I do that on debian? No.
q.e.d.
This isn’t “users” and corporations making a mess of the software world together. It’s all of us. As you guys are so fond of telling everyone else, if you don’t like it, change.
It's not the users that are acting too entitled. It's software producers that are doing it, by failing to perform their essential role of making sure the software they use is adequate for their purposes, and demanding that the FOSS components they're using do their their work for them.
> We aren't paying people to 'steal' your code. We're paying people to act like professionals.
I think you have a narrow view of the situation, or you are trying to misrepresent the reality we are living in.
We are signing up for OSS-derived SaaS mostly because of the existence of cloud-computing network effects. There have been many fights fought by companies 'acting like professionals' that could accept payment. They are re-licensing their software because developers and companies are utilizing a forked cloud-vendor-specific clones of their work. Forking software to provide it to a captive audience doesn't seem professional, it seems anti-competitive.
That said, most popular licenses allow it, so it is legal, but I don't think it is ethical.
There was no war. There was only "I'm not using your code unless I can see what it does and do whatever I want with it." It wasn't a statement of conflict, it was one of choosing ones own way without compromise. Then some people said "I like that idea, here, I made this, do what you want with it, but it's not mine, it's yours, problems and all."
That's all there is to any of this. There is no ethical responsibility. People doing something for fun for free dont owe anyone any degree of professionalism. If you want to burden yourself with the expectations of others that is your right, but just because you do it doesn't make it some standard everyone has to live up to.
Then stop calling yourself a software engineer. Engineering has an ethical responsibility associated with all of their work.
Go be a software developer with your no responsibility. Work on Dark Patterns. Work on spying on your customers. Work on making “Developer” an epithet so the rest of us can use you as an example to push for actual Engineering status and the protections that go with it.
Why do we expect more grace and generosity from the hosts of social gatherings than we do with software that will touch the lives of millions of users? That’s madness.
“Oh you don’t like that there are no snacks? Well then bring your own.” would get you blackballed in any other setting. It’s antisocial, which kind of goes with our stereotype. Most of us don’t like it, some of us don’t see how we feed into it.
Nothing about tossing a blueprint into the world entitles you to have blueprint hosting for free and without requirements.
Folks who don't like that there are terms and conditions on hosting software in repositories are free to host it themselves.
The advantage of github is that people have accounts and it's easy for them to send patches or discover projects.
But I'm fully capable of using git via email.
And what's the commercial alternative? I'm personally not aware of one.
Also, the liability thing is a grey area - it works for the US (mostly, Tornado cash is currently testing these waters) - since there are open issues with developers of "questionable" software traveling to some countries. Heck, even the US has pulled some OSS developers aside at customs because of their work on OSS projects.
[1] https://en.wikipedia.org/wiki/Row_hammer
[2] https://en.wikipedia.org/wiki/Meltdown_(security_vulnerabili...
> And what's the commercial alternative? I'm personally not aware of one.
There's commercial support for Spring:
https://spring.io/support https://tanzu.vmware.com/support/oss
It goes even farther than that: you can’t fault the author even if you did use it in the intended manner.
> Neither Microsoft, nor the device manufacturer or installer, gives any other express warranties, guarantees, or conditions.
And their liability is limited to...
> at its election, either: (i) repair or replace the software at no charge, or (ii) accept return of the software (or at its election the device on which the software was preinstalled) for a refund of the amount paid, if any.
So, maybe they'll fix, or maybe they'll refund you your money and you discontinue your use of the software. Maybe they'll issue a fix that you incur the cost of installing and testing.
[1] https://www.microsoft.com/en-us/Useterms/Retail/Windows/10/U...
But it gets worse. Often the people promoting the software aren't the people who wrote it, and they may make claims that the authors never intended.
None of these things mean that the authors accept formal liability, but morally speaking, if you're encouraging people to use your software and you're participating in an ecosystem promoting its usage, and it turns out to have bad security flaws, then maybe you shouldn't be so encouraging? Be upfront that it's just something you threw together. Add a few speedbumps. Maybe don't publish it on npm? Disrupt the hype train.
Disclaiming all responsibility should imply that you also don't hype it up and set people up for failure.
If you're releasing something and have no intention of supporting it, simply put "NOT INTENTED FOR USE IN PRODUCTION" at the very top of the readme file. It's that easy, imho.
So if you really want to discourage usage, you should say something in the README.
So waste everyone's time instead of just telling them?
> If you have a hard time discerning, abide the license, don't use code in production that disclaims liability, ever.
That's basically all code so that's not useful advice.
> not useful advice.
It's very useful advice. If you can't discern whether something FOSS will be useful to your business or not then don't base your business on FOSS code. Simple.
Uh, the whole point of the comment was that license doesn't say how serious the project is.
It's not about boilerplate disclaimers...
> It's very useful advice. If you can't discern whether something FOSS will be useful to your business or not then don't base your business on FOSS code. Simple.
"don't use code in production that disclaims liability" is useless. All code disclaims liability. The "if you can't discern if it's useful" doesn't convert such a pointless phrase into good advice.
It's like saying "if you can't determine which foods are healthy, don't eat food that comes in amounts". If someone can determine, the advice doesn't apply, and if they can't determine... you just told them to starve? That advice helps nobody.
And it is about boilerplate disclaimers. That's my entire point. "Add a boilerplate disclaimer reiterating what your license says if you really really mean it" is what your argument boils down to. How do you make it not boilerplate? Require then to write a 200 page report in their own words? If the license disclaimer means nothing, no number of disclaimers will mean anything more. If the advice helps nobody so will an additional disclaimer.
A typical business is going to die if it can't get software.
> And it is about boilerplate disclaimers. That's my entire point. "Add a boilerplate disclaimer reiterating what your license says if you really really mean it" is what your argument boils down to.
No, please read what people are saying.
The request is for a basic description of how serious the project is. Whether it's a super slapdash proof of concept, or a basic implementation with half the features stubbed out, or a mostly complete but unstable project that, or something battle-tested and heavily documented and maintained full-time, or other places in between or outside those lines. Whether they even want bug reports; whether they even want patch submissions.
All of those very different projects could have the exact same license. The exact same disclaimers.
The request is for information that does not belong in a license, and is not put into a license. Nobody is asking for a restatement or rewrite of the license.
I can see putting some message of intent as a developer, but people change their minds. Lots of the problems we see today that we are discussing involve people who started out with an intent to maintain full time but then for burnt out or otherwise lost interest. A message of intent like you say does nothing for a business already dependent on their work. This scenario is the more common problem with these dependency chains and what I thought we were discussing, most slapdash proof of concept projects say so in a README. If you can't discern whether you can depend on something, then you need to be prepared to take over the project or live without it.
A business will not write its own OS, and commercial OSes have the same disclaimers. You didn't say "if you have a hard time discerning, don't use FOSS code that disclaims liability". You said "if you have a hard time discerning, don't use code that disclaims liability". (slightly paraphrased)
> I can see putting some message of intent as a developer, but people change their minds.
And then you can update the message. All the more reason not to tie it to license files.
> most slapdash proof of concept projects say so in a README
Sometimes. Less often can you tell the difference between a mildly serious project and a very serious project at a glance.
> A message of intent like you say does nothing for a business already dependent on their work.
> If you can't discern whether you can depend on something, then you need to be prepared to take over the project or live without it.
Anything could change, but knowing the intent gives you a much better starting position. And it's better to put it somewhere it can be seen in under a minute rather than after a chunk of time in investigation, especially because there might be a whole pile of dependencies to examine.
But there usually is, that is what software licenses are for, with typical statements like
> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
I discovered last week that Github no longer allows you to push changes via https, and thus I have not yet published a minor fix for an insignificant piece of open source software I have been lackadaisically maintaining for the last eight years. Perhaps some day I will get around to jumping through their new security hoops... or perhaps I won't. In this case nobody is likely to care, but it makes me think these kinds of organizations ought to be extremely cautious about introducing extra friction to the workflows of people who are giving away their time for free.
Or, heck, if no build is required: why use git at all, just use github itself. You can edit, create PRs, on new branches, all without ever needing your own desktop. Perfect for small code changes (especially typo fixes).
It's working fine for me.
remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.You can't push via HTTPS Authentication user/password anymore.
The point here is not that my personal experience matters, but to illustrate an idea from the linked article: every time you introduce some new point of friction to an open-source distribution process, someone is going to wonder "do I care enough to deal with this"?, and sometimes the answer will be "no".
The entire software industry has occasional change, and I have a hard time believing that 5 minutes to update your account to use a more secure system is that much of a burden (they even provide you a link to the docs!). You seem to have some other reason for really taking a stand about this.
"But what about a keylogger / what if your password manager is compromised / etc." if someone has root access to my machine and can read my encrypted documents or log my keystrokes, the game is up for an SSH private key as well. There's no reason to say my SSH private key sitting on my hard drive is more secure than my GitHub-specific password sitting (encrypted) on my hard drive.
SSH key pairs are more secure on average because passwords are used by everybody and SSH key pairs are used by security nerds. That's it. That's all there is.
I'm not actually taking a stand about anything; github can do whatever they please. I'm just sharing a funny story from my recent life which illustrates a point from this article: there is no software supply chain, there is only an army of volunteers bearing gifts. If you try to demand that these volunteers raise their standard of gift-wrapping, no matter how small and reasonable a request that may seem to you, some of 'em just won't show up anymore.
Of course I have spent more time here talking about the problem than it would have taken to fix. So what? This conversation is fun! I am happy to spend my free time this way. When I choose to spend some of my precious free time on grungy, irritating chores, however, there's a long list I would rather deal with before this nuisance from github would come up: washing the dishes, taking out the trash, running the laundry, sweeping the floor, scrubbing out the toilets... those all being activities which improve my environment in ways I actually care about.
That said, to opt out of companies' supply chains, stop publishing software with permissive licenses.
He includes complaints about Github there, but those are easily resolved by not using Github.
I believe that the author and I are on the same page here, generally. Have I misunderstood his essay?
Absolutely not. There's plenty of software under GPL used everywhere, including the Linux kernel, GCC and tons more.
Unless one thinks there are no supply chains in Gene Roddenberry's Star Trek, which is obviously wrong.
Money isn't what makes it a supply chain. Pull dependencies on provenance (including origination, transformations, even aggregation) make it a supply chain.
I think both are perfectly valid options. Just make informed decisions and in doing so, demonstrate business competence.
Try explaining software packaging and distribution to a non-technical person.
Using analogies like "software supply chain" helps people better frame a problem/concept in their heads.
If the author scraped the title and just focused on how they are large companies attempting to force their will on open source maintainers to protect their supply chain, they'd have a stronger piece.
"A supply chain is a network of individuals and companies who are involved in creating a product and delivering it to the consumer. Links on the chain begin with the producers of the raw materials and end when the van delivers the finished product to the end user. "
https://www.investopedia.com/terms/s/supplychain.asp
The exchange of money, guarantees, quality assurance... None of it is required.
The last node in the supply chain for virtually any real-world product is "human beings extract the material from the earth in some way/shape/form". No human being paid "the earth" money for this, yet it is a fundamental component of supply chains (and, recently, a fundamental component of many supply chain issues). "The earth" also doesn't offer any guarantees or quality assurance.
Attempting to interpret these processes as requirements to be part of a "supply chain" excludes the core foundation of traditional supply chains.
Even a relatively literal read of the words "supply chain" is reasonably appropriate, even for FOSS. Wikipedia's current definition for supply chain is:
> In commerce, a supply chain refers to the network of organizations, people, activities, information, and resources involved in delivering a product or service to a consumer.
There's nothing in that definition that necessitates a formal agreement, support commitment, etc.
So far enterprises are the only ones willing to pay for their "software supply chain", if you want this kind of "solution" just use what they use.
That's just whining from people who host their stuff on other people's servers and sites.
If other people package your software into a bigger system and then later improve their standards so that the bar is above your stuff now (e.g. you break it left and right with careless, untested changes, or whatever, whereas downstream is doing actual QA) that's downstream's prerogative. You don't have to do anything; they can fork your stuff and develop it themselves, or stick with the old version from you that worked, ignoring the newer releases.
In the past I've forked at the sign of any resistance.
I have a fork of GNU Awk called Enhanced GNU Awk (egawk).
https://www.kylheku.com/cgit/egawk/about/
It has a let construct for lexical variables.
Chances are it's not packaged for your favorite OS, so you'd have to build it yourself.
The next thing on the plate for that project is to support the GNU C preprocessor line annotations so that if Egawk is handed preprocessed code, it will report errors against the original file and line number.
I'd argue that the problem isn't that "software supply chains do not exist", but "you using a program or library without pay does not and should not mean the software's author is now responsible for fulfilling your use cases and paperwork requirements".
I had a running joke at a place I used to work that I should just quit and start my own US based company that "sold" open-source software so that it'd have a US-flagged company behind it. IMO easy way to branch out to commercial instead of just government contracts would be to audit the source code. Probably save a few companies from a left-pad incident (or worse) since they'd have to pull from my servers (or cache mine) and I'm smart enough to recognize that updating leftpad to be nothing would not be helpful to any client
That is the business model for Redhat, Ubuntu, SUSE, and other GNU/Linux companies. You can download it for free or pay a subscription to have emails and phonecalls answered.
Although I'd probably need to not be completely blunt about it. It'd just be mostly an open secret where any of the dev teams would know that's exactly what I'm doing (and any sort of LICENSE file would make it dead obvious) but to procurement and whatnot it just looks like you're funneling money to me in exchange for some software.
I also take full Responsibility and Accountability for that code, when it is running in my execution thread.
The result of this policy, is that I have written almost every single dependency that I use.
It has garnered me a lot of sneers from the "in crowd."
I don't particularly care.
WFM. YMMV.
[EDIT] Also, I like this:
my name is always spelled lowercase and is pronounced /ɪliˈɑnə ɛˈtiːn/.
please consider making the internet a weirder place today
note: if you send me unsolicited email based on my github profile:
i will screenshot it and share in my discord server
we will make fun of you
you will not receive a response*Like I said, I know what works for me. I'm not gonna argue about it. I do know that some find my stance offensive, which I don't actually understand. I'm not attacking anyone. I'm just saying how I do it, in my own work.
I write lots of packages that I integrate into my own work. Each package is a standalone project, with tests and docs. Because they are written and maintained by a single person, I know that some folks believe this makes them "worthless," but not to me.
It's the way that I work, on my own stuff. I'm not competing with anyone here, so I fail to see the problem. If I'm just some clueless old boomer, making a hash of things, then God bless my little heart, and I'll just be tilting at windmills, over here.
If I use the SL, then I should also have no problem using some lashed-up chimera that has a dependency hierarchy that spans three continents.
Like I said, I'll do things my way.
For the record, here's the Swift Foundation Library: https://github.com/apple/swift-corelibs-foundation (I mostly write in Swift).
It has plenty of open issues: https://github.com/apple/swift-corelibs-foundation/issues?q=...
Also, for the record, here's a peek at some of the "worthless" packages that I use in my own work: https://github.com/RiftValleySoftware
If every dependency chain can match these, then I'll be open to considering them.
As it is, I do use the occasional external package, but I'm picky.
The metaphor in this case is pretty straightforward as are its limits, and it doesn’t appear to be causing confusion with people misidentifying attributes of the metaphor that don’t map to the software domain.
(And to boot: there are actual software supply chains with a traditional supplier/vendor relationship…)
One of the central points that the author makes is that this is in his opinion not true, and he provides evidence for this.
Me, too. Which is why I pretty much abandoned the formal "OSS" world and don't use Github and the like. I still make software, I still provide a free license for anyone to use it, and I'll even maintain it for as long as it interests me.
But it's a hobby, not a job. If someone uses my software commercially, it's on them to make sure that it's properly maintained. They have the source, they can do it.
K. The rest of us are gonna keep using the phrase though.
>I just want to publish software that I think is neat so that other hobbyists can use and learn from it, and I otherwise want to be left the hell alone.
Don't use GitHub. I don't. Random dipshits aren't even aware of me.
>To continue the inclusive nature of open source, we need to be able to trust a wide range of identities, but still with verified integrity. This implies a federated model for identities, perhaps similar to how we support federated SSL certificates today
Oh yes, of course Google supports TLS-flavoured snake oil to match the TLS snake oil.
I'm shocked MicroSoft is extending open source after embracing it so well, shocked.
I'm now recycling a previous comment of mine on this topic:
Companies used to have employees write code, rather than stitch together random garbage written by random dipshits who could be tricked into using loose licenses. That's one cause for concern. The only reason open source receives support is because it helps corporations defang Free Software and get gratis labor.
All of this new security theatre is always about trust and reputation, and not trusting those disgusting lone programmers such as me or other silly things; it's always really about doing anything but truly auditing that yucky code.
I also think this battle is lost as GH and others will always yeld to the needs of folks that atually bring money to their business.
"Pure hobbist FOSS" mantainers likely need to start a Wikipedia ish non profit for software hosting or think about something else.
If the bet could be objectively settled I'd happily stake my entire savings on the question, because I've already talked to the folks making those decisions.
It's the opening question in PG's essay linked today.
That's the point of the article, yes. It's about the mismatch of expectations and behaviors between C suite, which expects every business to do its duty on pains of lawsuit, and hobbyist FOSS programmers, who don't see themselves as in business and who have less than zero duty to FrobnitzWare/Invisible Corporation kabushiki gaisha LLC Limited AG despite the fact that August Corporate Personage decided to base Essential Business Processes on their work.
I mean, fundamentally, the programmers are in the right, in that the company cannot, in point of fact, rely on a contract it never drew up or got anyone to sign, but the companies will do the legal equivalent of holding their breath and stamping their feet and this is something people will, every so often, write blog posts about. Heck, maybe, just this once, a blog post will inform some C suite suits about the power gradient here (corporations: none, rando programmers: quite a bit, in fact) and maybe nudge some of them to, maybe, not make software that collapses when left-pad's author has another Big Idea.
> If you are a multi billion dollar company and are concerned about log4j, why not just email OSS authors you never paid anything and demand a response for free within 24 hours with lots of info? (company name redacted for my peace of mind)
https://twitter.com/bagder/status/1484672924036616195
https://daniel.haxx.se/blog/2022/01/24/logj4-security-inquir...
Maybe "Stamping their feet" is a bit hyperbolic for this case, but it demonstrates the cultural mismatch very well: The email assumes the Log4J package comes from the "Log4J Company" such that a business requesting the results of an internal audit would be met with something other than an annoyed FOSS developer Tweeting about this clueless moo sending a form email to a random programmer they have no prior relationship with.
As Daniel himself says in the follow-up Tweet:
> I replied saying I'd be happy to answer all the questions as soon as we have a support contract.
There was, deep inside the company, an internal process which blindly assumed a contract would be present, and acted accordingly.
> https://news.ycombinator.com/item?id=30059404
> https://www.reddit.com/r/opensource/comments/sboj1l/entitled...
This complaint is in my opinion rather specific to GitHub (and could thus likely be solved by switching to a more repository provider, or hosting the repository oneself), but not being able to disable PRs at your own repositories is like having a blog hosted by a provider that does not offer the option to disable blog comments.
I see this differently: you have some own content
* a blog post
* a Git repository
that is hosted by some provider. "Everybody" can add their own "scribblings" next to it:
* comments
* PRs
In both cases it is clear that these "scribblings" are not your own content.
Why do you thus make the difference that in one case one is some kind of responsible for moderating, but not in the other case?
Also, blogs are on your own domain name usually, which at least gives the illusion that it's your own website rather than just your little tenancy on a blog platform. Some platforms don't give you a domain technically but a / instead, like FB Pages and Reddit, and they explicitly assign you the responsibility of moderating your own page. If their own moderators have to step in, it often leads to deletion.
Liability disclaimers are a legal grey area. Even the US pulls aside some developers at customs because of their work on OSS projects. Imagine what some of the least tolerant countries do.
Looking at it strictly from an open source perspective is having too narrow of a focus. If I purchase O365 licenses from a reseller and they buy it from Microsoft, that's a supply chain. If I have an embedded system that has a proprietary OS made by a 3 person company who use a proprietary firmware from other vendors, that is a supply chain.
The npm example is a good example of a convoluted and unreliable supply chain. OSS is an example of a supply chain that is by design unreliable, the suppliers disavow all responsibility to support or guarantee its features. OSS supply chain being unreliable does not mean the supply chain does not exist, it just means you have to compensate for the unreliability yourself. The real life equivalent would be sourcing materials from guys standing at a street corner selling stuff that "fell off a truck" or "made in their garage".
I fully agree with the author though; it was even funnier when NPM sarcastically blamed me for losing my password on the reset screen, since it was THEM who lost their users passwords and force-reset them all, so now I had to reset my password, forced to use 2FA and being mocked while doing all of that ONLY to be able to share my work for free with the world. I was very, very close to say fuck NPM I'm out, unfortunately they have a de-facto monopoly.
don't expect to get to publish on someone else's platform then not play by their rules regarding how published material is handled.
I don't think the metaphor is apt. Dependencies are fungible you can branch, tag, freeze, lock, etc whatever your requirements are and continue trucking along. Physical Supply chains are not that at all. In some cases you literally cannot go back to the previous revision ever again.
Such products are what the word "fungible" was first used to describe.
I think this comic demonstrates the "problem": https://xkcd.com/2347/
We all use FOSS - its one of the most unique aspects of software engineering compared to other industries IMO. The world would literally not have advanced nearly as far without it.
I'm incredibly grateful to work in a field that views FOSS maintainers so well and thankful to the FOSS for all the incredible and fun technology I get to use everyday.
I like to think of it this way...
- we download code
- from the internet
- written by unknown individuals
- that we haven't read
- which we execute
- with full permissions
- on our trusted devices
- where we keep our most important data
(someone wrote this that I found in the SSC space)
Given a supply chain attack can happen at many of those steps above, but is also extended to source code, build systems, and production services, it's a great metaphor for visualizing the complex "software supply chain".
I think this speaks to the fundamental weirdness of the information economy -- it's highly deflationary, with some extreme cases where hobbyists outperform paid software companies
Hobbyists + open groups beat companies in cases where IP concerns makes the software worse, like if they're making something a saas that should be a 100-line library. And also in cases where everyone in industry needs a thing and they're happy to have a great standard free one rather than paying three shitty vendors.
I don't know this, but I think a chunk of major OSS projects are maintained by people at big companies or universities. (Postgres may have been associated with fujitsu for a while, guido van rossum + linus torvalds parked in various companies, some fraction of kube committers are at bigcos or corporate-backed CNCF). I have nadia eghbal's 'building in public' on my list to read, guessing she knows the answer to this question.
Once you put stuff on someone else’s servers, they can do whatever they want to the package as long as it doesn’t break the license. That includes marking the package as critical or even making a fork the canonical repo. If you want that kind of control, you need to become your own package provider. Or distribute your code with a more restrictive license.
That's pretty much true of commercial software as well.
There is:
* "open source as a reference"
* "open source as a line item"
* "open source as a funnel"
It seems like our industry favors line item OSS. Personally, I think the other two are far more useful.
Line item OSS productizes a high-value project inside a company and releases it publicly for no charge. The goal of open sourcing is generally to steer the industry in a direction compatible with your company's architecture. Often times companies are trying to:
* Cultivate engineering talent trained on their architecture that they can hire in the future
* Increase maturity of their software stack by exposing it to environments outside their current architecture
* Recruit maintainers to drive down the cost of maintaining an internal pattern
* Cultivate an ecosystem of plugins and modules they can rely on for future engineering projects
* Cultivate good will in the developer community as a recruiting/dev-rel tactic
* etc.
Line-item OSS projects typically carry a high cost, they don't just become successful when you push to a public repository. These projects require devrel presence to meet 3rd party engineers where they are (conferences, blogs, documentation, branding, logos, news aggregator sites, etc.). They require maintainers to mentor 3rd party engineers through pull requests and issues. You have to navigate community building with codes of conduct, boards, foundations, etc. This all requires salaried employee's time. This high cost raises the bar for contributing to the commons: the return has to cover the salaried time spent on open sourcing the project. If the return doesn't cover your investment, ROI is negative and you're expecting companies to contribute for charity. What's worse is that failing on a line item OSS project often results in the opposite of what you want: a perception that your company is bad at OSS or not a great community participant. Every line item OSS project is a high-cost bet carrying risk to your reputation.
Open source as a reference, in contrast, is closer to publishing a conference paper. You describe how you are doing something, publish the source code as a reference, and call it a day. No community building. You don't commit to merging random PRs. You don't commit to fielding questions on issue trackers. It's a statement that what you are doing is working for you, that other engineers might benefit from similar patterns, and a jump start for motivated engineers to figure out how to apply the pattern themselves. You're contributing to the commons under a permissive license for the cost of a conference road show and a "cleanup" of the code to ensure you aren't leaking anything sensitive in your SCM history. I feel a lot more software would make it into the commons if we normalized this form of OSS.
"Open source as a funnel" is probably the most sustainable OSS model IMH(umble)O. "OSS funnel projects are associated with a vendor that sells services and products in that ecosystem. The services are particularly useful for companies building on top of OSS. If you are a small-ish org, every dependency you're bringing in is a pretty big risk. You have to have internal talent capable of maintaining the entire software stack. "OSS as a line item" and "OSS as a reference" do not give you any guarantees from the project. The maintainers aren't your employees and they owe you nothing. You often have no way to pay maintainers to solve your problems. A show stopper bug in those projects can bury your company. In contrast, "OSS as a funnel" has a vendor you can reach out to for consulting/contracting services to elastically scale out your engineering org for bugs in the stack. In the Node.js ecosystem, NearForm is a great example IMHO. They are maintainers on a pretty large catalogue of software (i.e. fastify). You can bring that into your architecture knowing that, if you end up blocked on a bug your team doesn't have the expertise to solve, you have a vendor you can reach out to and pay to solve your problem.
Sometimes line item OSS projects foster an ecosystem of vendors which in turn checks the OSS as a funnel box. For example, with GraphQL and React, you can't pay Facebook but someone out there will take your money to solve your problems.
I do consulting/contracting in the Node.js space and, when I bring modules into my client's architecture, having a vendor associated with the module is front-of-mind for me. I know I'm not leaving them with technical debt they have no way out of if something goes sideways. They always have me and they have a 3rd party vendor they can reach out to as well.