Hacker News Reader

Comment by cxr | Hacker News Reader

Imagine if we had a system where you could just deposit the source code for a program you work on into a "depository". You could set it up so your team could "admit" the changes that have your approval, but it doesn't allow third parties to modify what's in your depository (even if it's a library that you're using that they wrote). When you build/deploy your program, you only compile/run third-party versions that have been admitted to the depository, and you never just eagerly fetch other versions that purport to be updates right before build time. If there is an update, you can download a copy and admit it to your repo at the normal time that you verify that your program actually needs the update. Even if it sounds far-fetched, I imagine we could get by with a system like this.

View on HN

You're describing a custom registry. These exist IRL (eg jFrog Artifactory). Useful for managing allow-listed packages which have met whatever criteria you might have (eg CVE-free based on your security tool of choice). Use of a custom registry, and a sane package manager (pnpm, not npm), and its lockfile, will significantly enhance your supply-chain security.

No. I am literally describing bog standard use of an ordinary VCS/SCM where the code for e.g. Skia, sqlite, libpng, etc. is placed in a "third-party/" subdirectory. Except I'm deliberately using the words "admit" and "depository" here instead of "commit" and "repository" in keeping with the theme—of the widespread failure of people to use SCMs to manage the corresponding source code required to build their product/project.

Overlay version control systems like NPM, Cargo, etc. and their harebrained schemes involving "lockfiles" to paper over their deficiencies have evidently totally destroyed not just folks' ability to conceive of just using an SCM like Git or Mercurial to manage source the way that they're made for without introducing a second, half-assed, "registry"-dependent VCS into the mix, but also destroyed the ability to recognize when a comment on the subject is dripping in the most obvious, easily detectable irony.

Does the lockfile not solve this?

What is "this"?

this = deps getting updated when you don't want or don't expect them to

We do not appear to have a shared understanding of the problem to be solved.

This is because you have redefined the problem—partly as a way of allowing you to avoid addressing it, and partly to allow you to speak of lockfiles as a solution to that problem. See <https://news.ycombinator.com/item?id=45824392>.

Lockfiles do not solve the problem. They are the problem.

This is what I wrote:

> Overlay version control systems like NPM, Cargo, etc. and their harebrained schemes involving "lockfiles" to paper over their deficiencies have evidently totally destroyed […] folks' ability to conceive of just using an SCM like Git

That's the problem that I'm talking about—lockfiles. Or, more specifically: the insistence on practicing a form of version control (i.e. this style of dependency management that the current crop of package managers tell people is the Right Way to do things) that leads to the use of lockfiles—for the sole purpose of papering over the issues that were only introduced by this kind of package manager—and for people to be totally unaware of the water they're swimming in under this arrangement.

Everyone is familiar with the concept of "a solution in need of a problem". That's exactly what lockfiles are.

not really, because you can't easily see what changed when you get a new version. When you check in the third_party repo to your VSC, then when you get a new version, everything that changed is easily visible `git diff` before you commit the new changes. With a lockfile, the only diff is the hash changed.

The problem is that not enough people care about reviewing dependencies’ code. Adding what they consider noise to the diff doesn’t help much (especially if what you end up diffing is actually build output).

Not if you use git submodules, which is how most people would end up using such a scheme in practice (and the handful of people that do this have ended up using submodules).

Go-style vendoring does dump everything into a directory but that has other downsides. I also question how effectively you can audit dependencies this way -- C developers don't have to do this unless there's a problem they're debugging, and at least for C it is maybe a tractible problem to audit your entire dependency graph for every release (of which there are relatively few).

Unfortunately IMHO the core issue is that making the packaging and shipping of libraries easy necessarily leads to an explosion of libraries with no mechanism to review them -- you cannot solve the latter without sacrificing the former. There were some attempts to crowd-source auditing as plugins for these package managers but none of them bore fruit AFAIK (there is cargo-audit but that only solves one part of the puzzle -- there really needs to be a way to mark packages as "probably trustworthy" and "really untrustworthy" based on ratings in a hard-to-gamify way).

Huh? "Just use git" is kind of nonsensical in the context of this discussion.

Oh, okay.

Yeah, people invented the concept of packages and package management because they couldn’t conceive of vendoring (which is weird considering basically all package managers make use of it themselves) and surely not because package management has actual benefits.

Maybe in a perfect world, we’d all use a better VCS whose equivalent of submodules actually could do that job. We are not in that world yet.

Do you understand the reasons, and are you able to clearly articulate them? Are you able to describe the tangible benefits in the form of a set of falsifiable claims—without resorting to hand-waving or appeals to the perceived status quo or scoffing as if the reasons are self-evident and not in question or subject to scrutiny?

I'm not altogether surprised at the negative reaction to this comment, but I am at a loss to really get into the head of the reader who is so unhappy with it. Let's give it another shot:

You wrote—alluding to, but without actually stating—the reasons why registries and package managers for out-of-tree packages that subvert the base-level VCS were created:

> Yeah, people invented the concept of packages and package management because they couldn’t conceive of vendoring (which is weird considering basically all package managers make use of it themselves) and surely not because package management has actual benefits.

This is a sarcastic comment. It using irony to make the case that the aforementioned trio (packages, package managers, package registries), etc. were created for good reason (their "actual benefits").

Do you know what the reasons are? Can you reply here stating those reasons? Be explicit. Preferably, putting it into words in a way that can be tested (falsified)—like the way the claim, "We can reduce the size of our assets on $PROJECT_Z by storing the image data as PNG instead of raw bitmaps" is a claim that lends itself to being tested/falsified—and not just merely alluding to the good reasons for doing $X vs $Y.

What, specifically, are the reasons that make these out-of-tree, never-committed packages (and the associated infrastructure involving package registries, etc.) a good strategy? What problem does this solve? Again: please be specific. Can it be measured quantitatively?

This is exactly what Swift Package Manager does. No drama in the Swift Package world AFAIK.

Now you have the opposite problem, where a vulnerability could be found in one of your dependencies but you don't get the fix until the next "normal time that you verify that your program actually needs the update".

If a security issue is found that creates the "normal time".

That is, when a security issue is found, regardless of supply chain tooling one would update.

That there is a little cache/mirror thing in the middle is of little consequence in that case.

And for all other cases the blessed versions in your mirror are better even if not latest.

This is how most software used to work before internet package managers, and it turns out that the same people who aren't good at checking their dependencies before automatically upgrading are also not good at constantly monitoring their dependencies for vulnerabilities.

You are describing BSD ports from the 90's. FreeBSD ports date back to 1993.

Also, Gentoo dating back to 2003.

And today.

So, vendoring?

That's a bingo.

Well in the Java world, Maven had custom repositories which did this for the last 20+ years.

That is exactly what I do.

Reply on news.ycombinator.com