Package managers need to cool down | Hacker News Reader

Package managers need to cool down

nesbitt.io

63 pointszdw55 comments

I’ve mentioned this before, but at my previous employer we set up staged Artifactory so tha production couldn’t pull from anything that hadn’t been through the test stage, and test couldn’t pull from anything that hadn’t been through CI.

Because releases were relatively slow (weekly) compared to other places I worked (continuous), we had a reasonable lead time to have third party packages scanned for vulns before they made it to production.

The setup was very minimal, really just a script to link one stage’s artifacts to the next stage’s repo. But the end effect was production never pulled from the internet and never pulled packages that hadn’t been deployed to the previous stage.

Reminds me of my favorite setup for a React project. We had rotating deployed builds (like three or more of them) for pull requests, the React UI pointed to a specific system so you always had consistent data, devs would immediately be able to test the UI, QA could follow immediately after.

>> But the end effect was production never pulled from the internet

Having production ever pull from the interwebs just seems bonkers to me. Even if you (for some reason?) want to stay up-to-date on every single dependency release you should be pulling to your own repo with some sort of gated workflow. If you're doing continuous deployment you definitely want to put extra control around your external dependencies, and releasing your product quickly after they change is probably the rare exception.

> Having production ever pull from the interwebs just seems bonkers to me.

Is it really that big of an issue if your package manager pins dependencies by hash?

I guess, public package registry can be down an brake your pipeline, that's a risk. But I don't see how it introduces any new security problems.

Pinning dependencies by hash is completely undermined by automation that then updates the pins without meaningful review, as is common.

I think that this is the issue then, not pulling dependencies from the internet directly.

> meaningful review No that I think about it, maybe for the first time in history it's actually feasible to review all the code in the repos using LLMs. Before LLMs were a thing, for any big project that would be way too much work to realistically do it.

Also, someone can provide code review of publicly available dependencies as a service, to avoid wasting tokens of reviewing same code again and again by each dev locally on their machine.

U wonder if anyone is already working on such service...

Most (all?) of the solutions offered are not providing a "code review service" but rather a "curated registry" one: download from us and we guarantee some things.

It's definitely more widely known/used for container images than individual software packages.

You can still lock versions, or even hashes. But it still leaves you open to "denial of service" if the "interwebs" acts up or someone unpublishes a package.

Something similar is easy with docker. Build the image when releasing to your first stage env, deploy the very same image to the next stage, until it reaches production. Nothing can break in between and enough time to test.

The complexity (and therefore the effort) of implementing such a setup varies greatly, depending on the ecosystem you're working on: Java, Python, Rust, Go, Node, ...

If everyone only starts using a package after 7d, it feels like it just means we don't find out about problematic packages until 7d later . The reason 7d works is because it is the "don't go first" effect (I assert with no evidence). But if everyone does the same delay, there's no longer a benefit to you delaying. This feels like a prisoners dilemma of no one upgrading.

I do think there is some sense in having some cool down. Automated review systems having some time to sound alarms would be good.

I'm not sure what the reporting mechanisms look like for various ecosystems. Being able to declare that there should be a hold is Serious Business, and going through with a hold or removal is a very human-costly decision to make for repo maintainers. With signficiant lag. So we are up to at least 2d, if centralized.

Ideally I'd like to see something on atprotocol, where individuals can create records on their PDS's that declare dangers. This can form a reputation system, that disincentivizes bad actors (false reports), and which can let anyone on the net quickly see incoming dangers, in a distributed fashion, real time.

(Hire me, I'll build it.)

IIUC the recent high-profile npm backdoors were mostly detected by supply-chain-security firms that ingest all package updates from the registry and look for suspicious code using automated or semi-automated analysis. Dependency cooldowns work great with this kind of thing. I agree that, if malicious packages were mostly detected via user reports, dependency cooldowns would create a prisoners' dilemma.

I don't understand what you're saying about reporting mechanisms; is there something wrong with how this is currently done?

Maybe a better way would be to allow third-parties to certify releases, and you can specify only to pull the package once they've given it the green light.

Cathedral or Bazaar on and on. We vary in opinion.

IMO we should be using the best easiest information syndication we have for all, that's as decentralized as we can be. That's why I suggested atproto. I believe the Bazaar approach here would be more interesting, and would avoid pressure points of only specific people having the relationships to pull the oh shit alarm.

What prevents the malicious piece of a package from just staying dormant for the cooldown period so nobody notices? What happens if everyone waits for a week to cool down, so the exact same problem happens, only a week later?

This may work in the case the maintainer becomes aware of the compromised account/credentials/package before the cooldown period, otherwise if it's about vendors and "other people", it's a roll of the dice. They may have longer cooldown periods than you.

Then we absolutely need an override for the cooldown period when we have to pull a zero-day patch the moment it's released.

The only reason cooldown periods haven't been exploited it's because they're not widely utilized. If they become mainstream it'll take a couple weeks to be rendered useless (yes this last part is 100% futurology)

It is too bad pkgsrc from NetBSD did not become a thing, a few Linux distros allow for its use and I think Minux also uses it.

Using pkgin(1) you get to know what happens before anything is done. Creating packages is hard, but as an end user pkgsrc shows me what it will do before it does anything.

This example for installing a binary package is from an already active NetBSD workstation, all items needed will be shown for the package you want to install.

    # pkgin install gnumeric-1.12.59nb2
    calculating dependencies...done.

    4 packages to install:
      gnumeric-1.12.59nb2 goffice0.10-0.10.59nb2 lasem-0.6.0nb1 libgsf-1.14.54

    0 to remove, 0 to refresh, 0 to upgrade, 4 to install
    16M to download, 76M of additional disk space will be used

    proceed ? [Y/n] n

How would that tell you if, say, the lasem-0.6.0nb1 package contains malware?

I would guess the same way Linux does. I am not a pkgsrc developer.

It's a fine balancing act between getting the latest updates and avoiding supply chain attacks.

I completely understand the author here, because I'm actually also leaning more towards avoiding supply chain attacks than jumping on the latest CVEs.

It's just a gut feeling, rooted in 25 years of experience as a sysadmin, but I feel like a supply chain attack can do a lot more damage in general than most unpatched known vulnerabilities.

Just based on my own personal experiences, no real data.

I'll try to put words to it, but a supply chain attack is more focused, higher chance of infilitration. While a CVE very rarely is exploited en masse, and exploitation often comes with many caveats.

That combined with the current state of the world, where supply chain attacks seem to be a very high profile target for state actors.

and those rare zero-days can be treated as the exception, and dealt with quickly. It seems backwards to optimize for dependency change reaction time these days with the supply chain such an attractive target.

CVEs are conceptually not the worst thing, but when every time anyone uses a regex gets a CVE it stops being a useful metric real fast.

In my previous company, we "simply" used fixed versions for our dependencies. And we had our own NPM registry that only had already approved packages for specific version. Approval required a security review by someone from the Security team… At first I was super annoyed by this. But I started to like this approach. It also reduced surprises while developing in a team… "it works on my machine" was rare since everyone was using the exact same versions. And moving to a newer version was done on a regular basis but it was an intentional thing we did.

How did the security team conduct a security review of a non trivial package

they run it throuh a tool that checks online whether any cves relate to that version. They don't care whether you actually hit the vuln, if there's a cve it's "bad". That's usually the level i see.

I'm a little bit amused by the inclusion of .NET in this article.

The last time I pulled in more than Dapper I was using .NET Framework 4.8. Batteries are very included now. Perhaps a cooldown on dapper and maybe two other things would protect me to some degree, but when you have 3rd party dependencies you can literally count on one hand, it's hard to lose track of this stuff over time. I'd notice any dependency upgrade like a flashing neon sign because it happens so rarely. It's a high signal event. I've got a lot of time to audit them when they occur.

I think there's a lot more than you might initially realize. A few of the top of my head (beyond your ORM): automapper, 3rd-part json or Polly, logging, server-side validation, many more. Another vector: unlikely a lot of other languages I've found way more .net libraries for 3rd party connectors or systems are community-based.

.NET definitely includes more these days, including lots of the things I've mentioned above, but they're often not as good and you likely have legacy dependencies.

I disagree. I also believe you might be equating your preferences with necessities. Much of what you listed is absolutely not needed, and often just brings in more external dependencies of questionable benefit. Take AutoMapper, for example, I cannot stand the tool, but many people enjoy it. I (sarcastically) bet many were thrilled when Jimmy, the maintainer decided to change to a paid license years after its release and heavy adoption.

I would argue .NET is roughly just as feature packed as many other frameworks, if not more so. I am glad many popular 3rd-party libraries are not baked into .NET. I do not want frameworks to be rife with garbage no one needs. Keep them lean and keep them mean.

Though, I will agree with you on logging. I do believe that the 3rd-party solutions are better.

This is theater. I say this every time it comes up, but the solution here IS NOT to just add vain and silly friction between potentially-malicious upstreams and package users. You'll never win that war.

The solution is independent audit. "Package managers" need to be (as they are in the Linux world) human beings responsible for integrating, validating and testing the upstream software for the benefit of their users.

NPM, PyPI, Cargo et. al. continue to think they can short circuit that process and still ship safe software, and they verifiably cannot.

Oh the lengths we go to avoid the obvious solution out of laziness...

How much would it cost to have at least an AI review? It eoild be better than nothing...

I agree with the author that changes are needed to stymie the proliferation in malicious packages (almost certainly potentiated by the ubiquity of LLMs), but I wonder if this is the best way to go about it.

The author posits two scenarios allowing for bad actors to take control over packages to add malicious code: stealing the original maintainers credentials, and taking ownership over dormant packages.

I would guess the second is much more likely. At least, it's the only one I've witnessed myself (in Arch Linux - I now view AURs are somewhat risky). I acknowledge this is argument based on anecdote - but what isn't up for debate is that it's far easier to identity dormant packages with high fan-out than stealing someone's credentials.

The cool down approach addresses the symptom; it would be better to address the bad actors who are the cause: dormant packages, recently taken over by maintainers lacking prior reputations and with high (transitive) usage, should be viewed with high levels of suspicion by the package managers. We should assume the new maintainers are malicious, and require them to prove they are not.

Package manager should be pinning exact versions and never updating anything unless my manifest changing. Ever.

Cooldown should be feature of dependabot/renovate/etc

Bun added `trustedDependencies` [1] to package.json and only executes postInstall scripts coming from these dependencies. I think this is something that should be supported across all JS package managers, even more than version cooldowns.

[1]: https://bun.com/docs/guides/install/trusted

Can you help me understand why one would ever need a post-install script in the first place, please?

Ime the most reasonable case is an optional compilation of native components when prebuilt ones are not compatible. See also node-gyp

Some tools also install pre-commit hooks. I don't like this practice, but I get why people are using it.

To restart a service, or run ldconfig?

Compiling native extensions that link against libraries that can’t be included in the package for license reasons. That’s probably the one reason that simply can’t be removed.

That's security theater. The package can still run arbitrary code the moment it's actually used.

That could probably be solved by opting in to the permission model of Node. But that won't work for everybody, especially in legacy applications.

Having trusted dependencies at least drastically reduces the risk that 'git clone && npm install' takes over the entire system.

Cooling down dependencies would certainly help, also.

How can you know that a dependency you trust won't be hacked? At best it slightly reduces the risk, but it's not even close to the effectiveness of version cooldowns that just block 100% of fresh updates.

Has any notable package manager tried the staggered rollout model that approximately every one of us uses when deploying changes at our day jobs? The complexity is, of course, version compatibility/mismatch, but it seems like a solvable problem. Then you could have an automatic canary system.

The closest I've seen to this are opt-in early release channels.

Does the timer start at upload, or does the timer start at mass exploitation?

Not a bad idea, but we'd need to have evidence of the former case to make it mandatory and widespread.

IIRC, many times it's compromised credentials of maintainers, which are caught very quickly, so that's evidence of the former case.

I think the premise is that modern scanners are really good at finding malicious code (and are run by dozens of companies in the industry), but when it gets pushed and installed inside of that 7 day window, the spread is uncontrolled. This basically gives you opportunity to let the machinery in the package ecosystem do it's job.

Surprised not to see nix mentioned in connection with this topic!

If you use nix (especially nix flakes), this consideration falls out naturally from the nixpkgs repository reference (SHA, branch, etc) you choose to track. Nixpkgs has various branches for various appetites for the "cutting edge".

Can you explain how this works? Is it different from what's described in the "Language vs. system package managers" section of the post?

Yeah, definitely. Think of it as what the post calls a "system" package manager. The difference between nix and the SPMs mentioned by the post is that in the former case, control over dependencies lies with you, not with the package manager.

In other words, with nix you decide the spec of the software you want installed on your machine, not the maintainer of your chosen package manager. Depending on your use case and knowledge/experience level, either choice may be preferable.

Also, nixpkgs is definitively the widest-spanning "package manager" on the "market"; see link.

https://en.wikipedia.org/wiki/Nix_(package_manager)#Nixpkgs

Yes! Nixpkgs straddles both worlds, like a system package manager it provides a way to install packages and their dependencies. However, like most language package managers it also imposes a locking mechanism so that every input into the nix expression* is locked to a hash, the mechanism is recursive and handles multiple versions of the same package in parallel.

The recent(ish) concept of "nix flakes" means there are 2 related but different mechanisms for achieving this but the end result is the same.

* In the land of NixOS everything is a nix expression including the system packages, configuration files, install image. It's all locked to hashes of upstream sources and is in theory, fully byte-identical reproducible.

This is security through obscurity (which is to say: it's not security). Malware already waits dormant for years in many different attack vectors. There are known, simple fixes for the attacks on package managers. They need to implement those fixes, not ineffective hacks.

A few points/qs:

- Could you explain what you mean by "security through obscurity"? The mechanism is well explained in the blog.yossarian.net posts linked within. It is simply adding a time filter on a client.

- Also, I'm not sure if package registries (e.g. server) and package managers (e.g. client) are being conflated here regarding "attacks on package managers", this seems to be more of a mitigation a client could do when the upstream content in a registry is compromised.

- Lastly, I agree with the sentiment that this is not a full solution. But I think it can be useful nevertheless, a la Swiss Cheese Safety Model. [1]

[1]https://en.wikipedia.org/wiki/Swiss_cheese_model

> Malware already waits dormant for years in many different attack vectors

And some malware doesn't wait. Sure, some supply chain attacks like the one in Notepad++ are much more sophisticated, but some untargeted ones (like the recent Cline CLI one) rely on package managers doing thousands of downloads before it's noticed and stopped.

The argument "all these vulnerabilities in other people's technology that we depend on have really simple fixes, so we don't need to do anything" is not a compelling one for most businesses.

If solving supply chain attacks from package managers is a simple security problem for you, you could make a lot of money though!

This seems like it would cause the https://xkcd.com/989/ effect.

I've been working on a project lately as my bachelor's dissertation which I later plan on working on long term on this issue.

The basic premise is a secure package registry as an alternative to NPM/PyPi/etc where we use a bunch of different methods to try to minimize risk. So e.g. reproducible builds, tracing execution and finding behavioral differences between release and source, historical behavioral anomalies, behavioral differences with baseline safe package, etc. And then rather than having to install any client side software, just do a `npm config set registry https://reg.example.com/api/packages/secure/npm/`

eBPF traces of high level behavior like network requests & file accesses should catch the most basic mass supply chain attacks like Shai Hulud. The more difficult one is xz-utils style attacks where it's a subtle backdoor. That requires tests that we can run reproducibly across versions & tracing exact behavior.

Hopefully by automating as much as possible, we can make this generally accessible rather than expensive enterprise-only like most security products (really annoys me). Still definitely need a layer of human reviews for anything it flags though since a false positive might as well be defamation.

Won't know if this is the right direction until things are done & we can benchmark against actual case studies, but at least one startup accelerator is interested in funding.

This is silly. Critical security and bug fixes come out and you are going to wait because you think older must be safer just to avoid a supply chain attack issue? Just secure the supply chain. Be critical about your dependencies and update strategy before updating. If you got some 200 translative dependencies and you don't know everything your build, that is a problem and you probably should look into that because assuming waiting is a solution is not going to stop you from getting hurt with that risk of a surface area.

In the age of AI, I reduced my load on small utility libraries and just have the bigger ones that I'll follow semver and update to manager versions when it make sense and always take small patches but still look at the release notes for what changed.

Reply on news.ycombinator.com