Compromised PyTorch-nightly dependency chain between December 25th – December 30
pytorch.org
pytorch.org
My recollection is that Java libraries were larger, higher-quality, more stable, and better-maintained, and you didn't need as many of them. A Java jar was not a "package" but contained dozens of "packages" developed together. Jars tended to be self-contained or mostly self-contained; small dependencies would shipped inside. The idea of making npm packages as small as possible, like practically putting each file in a separate git repo, and publishing it as a separate artifact, emerged shortly after NPM itself, and it was radical, and not really particularly good. Java also has a much larger standard library, and between the packages that come with Java itself, the packages that aren't technically part of the standard library but were written by Sun/Oracle, and well-known third-party utilities, you didn't need a lot of third-party packages. And if you needed something tiny like left-pad and didn't have it, you'd probably just copy and paste it.
Maven doesn't allow execution of arbitrary code at install-time, which curbs a large number of potential supply-chain attacks.
Because of the JVM and JARs being mostly self-contained Maven doesn't really need to worry about system or runtime dependencies (unless you're using Scala...). This allows Maven to be a 'dumb' package manager that relies on simple semantics (no hidden specially-generated indices, for example) and be fairly successful. Of course, there's an internal battle of whether Gradle or Maven is superior, but they both rely on the same distribution and packaging specifications.
I have many gripes with Sonatype but Maven Central isn't really one of them. The fact you can publish a packages to the likes of PyPI, npm Registry, or Docker Hub with 0 friction makes those places very attractive to spammers and bad actors. Maven Central having a higher barrier of entry is a feature.
IIRC Brian Fox, the CTO of Sonatype, was actively involved with Maven in the early days and was part of the decision for Maven packages to use domains for namespaces. Namespaces are another valuable feature of Maven that makes supply-chain attacks like typo-squatting harder to pull off.
At the same time, some people genuinely shit on Maven Central and think that it's inferior to other registries.
I think golang has the best answer for this, where it's easy to impersonate but it has to be explicit.
I think Go's approach is interesting, though it does rely on some magic that isn't immediately obvious. I agree that being explicit is a tremendous benefit: it avoids the attack used here, and makes it less likely for typo-squatting to succeed (e.g., `npm install axiod`).
I don't do any heavy computations except for running test suites of various kinds, and these seem to perform the same on raw hardware as they do in the Kernel-based VMs.
With such container, we can catch the compromised supply-chain attach easily, right?
Does anyone know such container exist?
But, say you had such a container, there’s an important distinction between “you captured a log showing the smoking gun evidence of the supply chain attack”, and “you successfully picked that log out of all of the log data you generated and classified it with high confidence as an attack”.
Speaking from experience, the second problem is the hard problem for a multitude of reasons. So while you would have the data, you’d probably have trouble getting good precision/recall on when to actually sound the alarms vs. when it’s some SRE who needed to troubleshoot some network connectivity issues.
The suspect application doesn't need the privileges, so I'm not sure how much of a problem that is?
> there’s an important distinction between “you captured a log showing the smoking gun evidence of the supply chain attack”, and “you successfully picked that log out of all of the log data you generated and classified it with high confidence as an attack”.
Assuming that you're talking about the signal:noise problem, that's hard in the general case but I feel like you could easily pick off really obvious cases like trying to access private SSH/GPG keys and still get a lot of value.
Probably. I’d agree that it’s worth trying at the very least. I’ve run into enough “should be easy” cases that turn out to be not that easy that my default is to get the data and see if the hypothesis really pans out.
1. https://github.com/ossillate-inc/packj
It DOES NOT require a VM/Container; uses strace. It shows you a preview of file system changes that installation will make and can also block arbitrary network communication during installation (uses an allow-list).
Also a Little Snitch-like host-based firewall, which would request explicit permission to connect.
Generally a VM jail is preferable - firecracker or cloud-hypervisor(virtiofs & gpu passthrough) recommended.
A proper namespace jail (eg. bwrap) is sufficient for 99.9% of cases. To break out of a properly configured namespace jail you would need to sacrifice a 0day.
This is an oversimplification, there may be other protocols that are passed through or utilized, they would add to the surface.
Zero, essentially. State actors profit massively from such systemic weaknesses, so it is not in their interest to eliminate them for the population at large (they do of course want to eliminate them for themselves, but they already have extremely strict supply chain policies so that's mostly a solved problem).
Hell, we have state-sponsored institutions working hard to actively create vulnerabilities in software that previously didn't have them. Security vulnerabilities are a tool through which power is exercised. They're not going away as long as governments have any say in it.
Do we want all those packages or do we want the functionality of them? Dependency hell happens for reasons of deficient first-party support. Notably, languages that lack a sufficient standard library and a blessed toolchain should be considered a pre-existing condition to this decease.
I get that language maintainers have legit reasons to exclude something like HTTP from std, but there has to be some middle ground here. For instance, Golang provides experimental packages that have high quality but a lower level of support. To me, this is a win. It centers the community in a common direction, and delivers real world value in the meantime, with minimal maintenance upkeep compared to the 100s of packages we see in some ecosystems like JS and to a lesser extent Rust.
It still contains only a tiny fraction of the functionality needed by any large project. The world is just too complex for a standard library to ever realistically cover a meaningful portion of the problem space.
We all need tensor algebra, video decoding, cutting-edge network protocols, compression, cryptography, dozens of data exchange formats, syscall bindings for three or more platforms, fuzzing, containers, and I don't know what else. This isn't going to all fit into a standard library. "Dependency hell" is here to stay.
Libraries refuse to shift away from this culture thing, and decided to just blacklist the known exploit path while keeping the vulnerability intact. It's not that they do this for backward-compatibility, it's new libraries still being designed this way.
The log4j 2 bug is slightly different, but IMO as a design issue it has root in the same culture thing described above.
Log4j 2 refused to shift away from it since they are paid to add that kind of bloat, yes this wasn't an accident caused by a bad culture, this was a feature someone requested. As far as I understood they are completely separate from the original log4j and it is just another, mostly compatible logging implementation.
> Java is unique as the culture encourages doing magic convention over configuration by reflection AND it-just-works serialization
As opposed to what? Python? The language that culturally refuses to fix the GIL and as workaround provides multiprocessing, which requires that everything is serialized?
Also I may not have seen many Python code bases, but from what I have seen ClassLoader/Import abuse is alive in both languages.
Python at least does not pretend their it-just-works serialization is secure. The documentation [1] actively discourages deserializing untrusted data and suggests to use pure data format (e.g. JSON) as possible.
In contrast, the documentation on ObjectInputStream before Java 11 does not warn about this at all. Even then [2], it suggests to implement blacklist/whitelist filters, and it is pretty hard to get right. The same filtering can also be done for pickle, but the usual consensus in Python developers seems to be "don't do this even if it's possible".
> but from what I have seen ClassLoader/Import abuse is alive in both languages.
Yeah, should have said that my rant is mainly against endless deserialization/OGNL injection/whatever-popular-expression-language injection bugs in frameworks, not abusing ClassLoader. These features, just like log4j2's code-execution-disguised-as-string-interpolation feature [3], shouldn't exist.
[1] https://docs.python.org/3/library/pickle.html
[2] https://docs.oracle.com/en/java/javase/11/docs/api/java.base...
[3] I'd argue that's the real bug, instead of the obvious JNDI class loader blah blah stuff. Luckily log4j didn't refuse to fix it and completely removed "message lookup".
In my opinion, Java’s standard library is good, but not as good as the one in Python and then Golang. I also prefer the one in C#.
There is a size limit for standard libraries?
I think Java introduced modules and its dependency model so you could package only what your software needed.
There is a size limit to what language curators are able to maintain, yes.
The idea of the standard library is that it will select components that are general enough to cover a wide range of use cases; therefore the language builders ned time to review every new proposal, and their time is limited.
You may dump everything under the sun in the standard library, but then it won't be better than what you get from a library marketplace.
There are thousands, if not tens of thousands, of packages available to install via apt or yum. But most of those packages are packaged by a dedicated maintainer - not any rando. The bar isn't much higher, but there _is_ a bar. Python's (well, pip) practice of letting any vermin with no prior vetting publish is root of the problem in my opinion.
Which leads me to wonder: do we really always want the very latest version or the packages? Any slightly older version will be immune to dependency-poisoning, thanks of the scrutiny of users over several weeks.
Not really, but then someone needs to decide what version to use and when/how this version is updated. For every single dependency. I agree that keeping track of your dependencies and actually managing them may be better engineering but it's a lot of work not many people want to do
PyPI probably needs a vetting system for new contributors of some kind.
I suspect a prime reason is the absence of package management on Windows which a lot of developers (and users) use, and secondly the desire by developers to always use cutting edge library features when writing code, but nobody wanting to upgrade any dependencies after. There used to be a lot discipline about being compatible with many libraries versions IMO, nowadays people just specify the latest version of each library in their requirements.txt file (or equivalent in other languages).
Yet, every main Linux distribution has its own packaging format (deb, rpm, etc) , package naming convention, dependency resolver, package release strategy (rolling, fixed, etc) , package build & deployment system (source, binary, per arch binary, etc), and package install peculiarities (custom, upstream focused, system wide, in a chroot, in a snap, etc), reproducibility constraints, etc...
So it's not like it's a _solved_ problem for Linux distributions.
Not to mention that most distribution package managers are system wide, while language package managers are often environment based.
> Yet, every main Linux distribution has its own packaging format (deb, rpm, etc) , package naming convention, dependency resolver, package release strategy (rolling, fixed, etc) , package build & deployment system (source, binary, per arch binary, etc), and package install peculiarities (custom, upstream focused, system wide, in a chroot, in a snap, etc), reproducibility constraints, etc...
> So it's not like it's a _solved_ problem for Linux distributions.
Just because someone reinvents the wheel does not mean it wasn't invented (solved before hand. I would also argue each distro package manager is miles ahead of any language one.
> Not to mention that most distribution package managers are system wide, while language package managers are often environment based.
That's the thing I was alluding to in the second part of my post, if developers would be more careful about backwards compatibility we wouldn't have to use environments. I do admit that packages for apps are more of an issue, it would be nice to upgrade those without having to upgrade the rest of the system.
Certainly not. A recent example: I wanted to try a KDE distribution, so I installed Neon which has three dependency updaters by default: pkcon, apt, and snap. If I use Python or Node, then I usually need to use their distribution management systems as well.
Dependency management is one area that completely fragments Linux into different distro universes.
I am not saying Windows is better (it isn’t) but if Linux had solved the problem, then most of us would share one solution.
The state of linux software distribution is completely abyssimal and ridiculous
1. pytorch publishes their nightly package on their repo which depends on a custom triton build provided on their repo, but using a package name they don't own in pypi. This has been mitigated by them by renaming the dependency from torchtriton to pytorch-triton, reserving pytorch-triton package in pypi, and changing the dependency name on the newer nightly builds 20221231 forward to point to pytorch-triton package instead.
2. pytorch installation instruction for nightly using pip in (https://pytorch.org/get-started/locally/#start-locally), make use of the --extra-index-url option. This is a known vector of dependency confusion attacks, and is an inherently insecure method of installing packages from private repositories. The recommended approach of distributing wheels in private repositories is by using a repository server that allows proxying/redirecting the public pypi packages to pypi, and users should be using a single --index-url pointed to that private repository (assuming the maintainer of that private repository is to be trusted). --extra-index-url is meant to provide mirror urls (serving the same set of packages as the main one), rather than to combine repos with different sets of packages.
Alternatively, keep a separate requirements_private.txt around for private dependencies and add a line --index-url <my private repository>.
Thousands.
He was able to steal packages names for Python (PyPi), JS (npm), Ruby (Gem), where these various companies have their own private package repositories with private modules where they don't control the package names in the default repositories.
The main requirements for this attack are
1. Having private repositories, where the package names in the default repositories is not owned by the owner of the package.
2. Package manager allows downloading packages from multiple repositories (the default and private repositories) without being able to pin specific package to only be downloaded from the private repositories.
What's notable on his findings is the omission of Facebook and Google, which I believe due to their usage of Buck/Bazel and monorepos for their internal codes. Another thing to note was that Alex mainly target companies internal private packages, while this particular instance affects an open source project which was providing package on their own repository and make use of a package (which I was not able to find any bug bounty program for).
There's another post by Kjäll et.al [2] that explains how this particular vulnerabilities affects other package managers (PHP, Java, .Net, ObjC/Swift, Docker), in what conditions it's vulnerable, and how to mitigate the risk. Two notable language package managers that were not affected are
1. Rust, mainly because you have to explicitly select the private registry for each private packages.
2. Go, mentioned as unlikely, due to the use of FQDNs in the package names and hash verification by default.
I think anyone adding non default package repositories or providing one (their own private repo in enterprise setup, or 3rd party provided repositories), need to be aware of this particular class of vulnerability, and implement policy to mitigate it. I would say, individual devs installing on their dev machines or CI/CD systems based on shell commands (rather than secured package manager setup) would be the main area of attacks mainly due to the relative difficulty of auditing those scenarios.
[1]. https://medium.com/@alex.birsan/dependency-confusion-4a5d60f...
[2]. https://schibsted.com/blog/dependency-confusion-how-we-prote...
And from the engineering blog, "[...] PyTorch 1.0, the next version of our open source AI framework."[1] (emphasis mine)
[0] https://www.facebook.com/whitehat/
[1] https://engineering.fb.com/2018/05/02/ai-research/announcing...
However Meta has since ditched it[2], and a careful keyword search of pytorch.org, linuxfoundation.org, suggests there is not any current official bug bounties for PyTorch.
We show alerts in GitHub pull requests, or the CLI, if you add a dependency with a supply chain risk.
I couldn’t bring myself to use a wildcard for a long time. I always specified the exact version and incremented manually to feel like I was at least trying to maintain control.
I still think it’s an insane practice, but with software engineering ever increasingly being the art of composing dependencies with bits of glue code, I get it.
Linux's security model would work well on multi-user mainframes in 80s but doesn't make much sense for an Internet-connected computer with a single user. The threats now are completely different.
For comparison, iOS and Android's security model would not allow a Python package to steal SSH keys. But the best solution would be to implement least privileges principle and do not grant unnecessary privileges to programs.
Another solution which could protect from such attacks would be to hire someone to inspect the packages. But it seems thatnobody is providing such services. Is it because people got used to everything being free?
I think even in iOS and Android, the same thing could still happen if the user keep secrets (say private keys) in their file directories, and then the user give permission to the app to access their file directories, which could be legitimate need of common applications like file manager, text editors, etc.
I think the main change to solve this would be to not keep secrets in files that are readable by users (or program run by users) implicitly, instead using a secret provider that makes it explicit to the user whenever an application needs to access such secrets.
- Flatpak is terrible because it doesn't follow a file system hierarchy that was invented in the 70ies.
- Flatpak is terrible because my early 90ies package manager is the pinnacle of packaging.
- Flatpak is terrible because I only trust my distribution's packages.
- Flatpak is a security nightmare because it doesn't isolate every application now. (Which is not really possible, because applications/toolkits need to be adapted).
- Flatpak is terrible because now my applications cannot open arbitrary files anymore (including ~/.ssh).
Conservatism is what holds the Linux ecosystem back. We have seen this story before with systemd. This is sad, because Red Hat and others are doing fantastic work modernizing Linux (see Flatpak, Fedora Silverblue, etc.).
…but the usage of MAC in Linux is so awkward that it might as well not exist, since the majority of distributions, packages and users ignore it completely to avoid all the headaches that come with it.
It’s tricky. It would be nice if the maintainers of big distributions/packages took MAC seriously and defaulted to restrictive SELinux policies or whatever, but history has shown this to be _very_ brittle and annoying for users. That’s why you see this stuff enabled in iOS and Android but not general Linux distributions — it’s much easier to lock things down with MAC when you’re developing a singular OS from scratch with apps that run in a sandboxed environment that’s isolated from the “system”. But if you’re maintaining something like python or pip, it’s basically impossible to do that.
This is the case for Rust, it seems. Or any GitHub-based project that pulls recursively submodules.
Unless you have a curated, audited package repository, i can't see how you easily avoid it. Hence it is reasonable to ask how the OS can help.
The problem does not stem from the operating system level, and so it should not be fixed there, and it likely can't be fixed there, at least not without possibly breaking a whole bunch of stuff that is working just fine today.
You mention some mechanisms for preventing these issues at packaging level. But so long as bad actors can sneak malicious commits, it's an open problem. Here a whole package was replaced, in another scenario a single line of malicious code can be sneaked in, released, rubber stamped and shipped via bona fide channels. Ensuring no single bad commit is made in any package or its dependency doesn't seem like a realistic task.
As other posters suggest, this could be somewhat mitigated by mobile-style finer-grained permissions, even if the problem doesn't stem from an OS failure per se.
If you try to fix a problem at the wrong layer all you will end up doing is create a bunch of false positives which will result in those protections being disabled. Then, by the time you need them they will no longer be active. A problem like this should be dealt with as close to the source as possible. A defense-in-depth strategy is a sound one but only if it does not add burdens that will lead to the defenses being disarmed in peaceful times.
Finally, I fail to see which finer grained permission would have stopped this, though maybe someone can make some suggestions. Not allowing the use of Node, Python, Go or Rust would seem to be a bit harsh and nothing short of that would seem to solve the problem without the OS having to gain massive awareness of what is going on at the application level.
Microsoft effectively fell for this by dragging everything and the kitchensink into the operating system layer, the only result of that was an even larger attack surface at a level where it mattered even more.
This is not an easy problem to solve. For starters you would have to review the way all of these dependency resolution mechanisms work and flag the ones that you think contain risk. Then you need to get the suppliers of those systems - and others like it - to agree on a standardized way of vetting the import process and finally you would have to make the OS aware of a set of signatures or certificates that would stop such an attack in its tracks.
The alternatives are doing away with easy dependency resolution or taking charge of the root cause and fixing it: the admission that decentralized source code control is a great thing to have but unsuitable for the distribution of packages for immediate introduction to production systems based on the say so of whoever controls the namespaces. The kind of centralized control that would require is something that the likes of Apple and Microsoft can provide, no Linux distribution is dominant enough to be able to solve this (nor are they wealthy enough). Distros have a hard enough time vetting their own output.
Perhaps a simpler starting point is a runner script that enforces such rules on an opt-in basis from the user, and tooling that makes it easy to run binaries through this. Like a reverse sudo sort of thing.
I'm certainly not saying it's easy, but I'm not sure it is impossible either. It's a bit like an internal firewall.
Even if apps request blanket permissions, that's a win already. On mobile, when apps do it, it is a big red flag for me and I skip.
I took a look and the vulnerability that was exploited is warned about and described in the documentation.
> Warning > Using this option to search for packages which are not in the main repository (such as private packages) is unsafe, per a security vulnerability called dependency confusion: an attacker can claim the package on the public repository in a way that will ensure it gets chosen over the private package.
>Hello, if you stumbled on this in your logs, then this is likely because your Python was misconfigured and was vulnerable to a dependency confusion attack. To identify companies that are vulnerable the script sends the metadata about the host (such as its hostname and current working directory) to me. After I've identified who is vulnerable and repoterted the finding all of the metadata about your server will be deleted.
I of course don't believe it, but interesting to see something like this.
It would be better if packages could specify a cryptographic hashes of the packages they depend on and/or specify the package repo that a package should come from. There is some prior art for this in e.g. the Javascript and Rust ecosystems.
This problem did not occur simply because pytorch has many dependencies and one of them got compromised. If pip allowed you to specify dependencies in a more secure way, this would not have happened.
--index-url https://user:pass@path/to/pytorch/nightly/repoand all dependencies need to be pinned to specific versions as well. hence this would only work when users are making use of venvs, instead of user install / site install setup.
python3 -c "import pathlib;import importlib.util;s=importlib.util.find_spec('triton'); affected=any(x.name == 'triton' for x in (pathlib.Path(s.submodule_search_locations[0] if s is not None else '/' ) / 'runtime').glob('*'));print('You are {}affected'.format('' if affected else 'not '))"Judging from the package installation stats this was installed around 2,500 times
But an ssh server (or any other server) should not also be used as a development environment that is pulling dependencies (if you are developing a server, then you should be using dummy keys and doing the development in a non-production environment).
It does, but very few people use it.
I hate to “give props” to the bad guy, but as a former pentester this area is fascinating. The cat and mouse game is eternal, and both sides evolve. This malware is pretty standard, but it’s also smart — you can tell it was developed by a dev to target devs.
Any progress on blocking exfiltration via DNS? It seems only malicious programs make heavy queries like this, and it should be detectable. Even a small ML network could probably be trained to tell the difference between legit queries and malware uploads.
But the packages they ask you to uninstall have the same names. How does one install the nightly package? If I just installed "torch" can I assume that I don't have the nightly version?
If you installed `torch` via the instructions to install the nightly version specifically, then you get the nightly version. By default, you get the stable version of `torch`.
This is documented on the installation page, assuming you instead by their method. You specify a different source URL when invoking the package manager, so you would need to deliberately install the nightly release rather than get it accidentally. If you installed it by other means (perhaps you OS's package manager) this is also likely to be the case.
> If I just installed "torch" can I assume that I don't have the nightly version?
Probably, but it depends how you installed it, if you are using your OS repository then check with their documentation. The linked page for this thread includes a section entitled "HOW TO CHECK IF YOUR PYTHON ENVIRONMENT IS AFFECTED" (their capitalisation), follow the instruction there if you are not 100% sure what version you currently have installed.
I recently wanted to contribute a feature I needed to an NPM package. The package is fairly popular (400k+ downloads per week) but is unmaintained with the last release about 3 years ago. There are no practical alternatives to this package. So my only option was to fork it.
Unfortunately, one of its dependencies was found to be malware between that last release date and now. That dependency was nuked and no longer is available (not that you would install it anyways), so neither my fork nor the project can be built. The dependency tree is, of course, very complicated and it would take a lot of work to figure out a replacement for that dependency and integrate it.
I ended up directly patching the last release's compiled bundle to do what I needed to do. In this case, this very useful (but unmaintained) package is now more or less dead and further development work will be very difficult, its source code a probably useless relic.
there are no good implementations of this yet to my knowledge. encrypted fuse with snitch like rules and prompts is probably the play.
any network snitch program would have caught this and prevented exfill. a prompt would have been displayed for a new connection to sketchy.io udp 53. from pytorch or some child process. most eyebrows would rise. many requests would be denied.
on mac you have little snitch and at least one foss alternative. on linux you have opensnitch, or can roll your own on libnetfilter_queue[1] or lsm hooks[2].
hopefully soon somebody implements encrypted fuse snitch. until and even after then, wrap yourself in network snitch safety blankets and feel warm and cozy.
ufw preventing all udp 53 to anywhere but 1.1.1.1 might have worked.
it’s the raised eyebrows that are the most important part. ttre, time to raised eyebrow. random network nonsense should not pass invisibly.
This is not the real torchtriton package but uploaded here to discover dependency confusion vulnerabilities
the compromised version is still available in pypi as version 2.0.0+0d7e753227 https://pypi.org/project/torchtriton/#history.
So technically, if you are pulling the older version of pytorch-nightly (specifically 2.0.0.dev20221230), it will still pull that compromised dependency (because torch have explicit version lock to it).
All PyTorch nightlies with this dependency have been deleted
pip3 download torch==2.0.0.dev20221230+cpu --extra-index-url https://download.pytorch.org/whl/nightly/cpu
Requires-Dist: torchtriton (==2.0.0+0d7e753227) ; extra == 'dynamo'
Although I guess this is low risk, because people normally would download nightlies without pinning to a particular version/date.
But in case there are people that does pin their version, and cache those vulnerable versions (locally or on their own proxies/private repositories), they could still be affected.
I recommend to get PyPA to yank the 2.0.0.dev20221230 version in pypi, and possibly amend the post to remind people to purge their caches not just on their local but also on their proxies/private repos/mirrors (mainly for the torchtriton package) and to immediately stop using any pytorch nightlies dated before Dec 31 2022 (mainly any pytorch nighlies that has a pin on torchtriton==2.0.0+0d7e753227, not just between 25 Dec to 30 Dec).
the nightly pytorch moved to depend on our own secured `pytorch-triton` now, secured on PyPI and our nightly channel.
seems like a higher priority task than generating artificial art.
let's turn the GPT's toward something like a softwaresupplyGPT that can be trained on all the software on all the individual package/dependency managers that are in existence.
1. https://github.com/ossillate-inc/packj flags malicious/risky packages.
No other package distribution system is that bad. BSD ports are better, Homebrew is better.
Things were a little bit slower to get new versions but things were safe and reliable.
Now, cool kids on the block arrived a few years ago, and decided that they were a lot smarter to have always the last version of everything, in direct and instant way, and build on moving sands. They were thinking that old farts were dinosaurs doing legacy software dev.
All we can say now is that by forgetting the past, you are doomed to make the same mistakes again and again.
They would only build from source for example, and rebuilt by the distribution infrastructure. And in addition, the slow pace of having things like stable, testing, unstable was to give the opportunity to serious users to ensure to used package that were stable, safe and battle tested.
Then if someone does try to creep into your package list with a new GPG key or modifies the content with a new key, it would call attention to this much faster and prevent these kind of situations.
Thus; if you trust an author, then add their GPG key to your vault. This will most likely cover multiple packages of the same project, as well as well known groups that provide packages on these infrastructure.
Then npm/pypi/cargo/whatever are given a list of the parties you trust and the number of certifications each bit needs before it can be downloaded.
Edit: of all the responses, none provide any mechanism for the software-verifier to be compensated. Says a lot about how much we value that kind of work as an industry.
Not familiar with Python's ecosystem at all, but judging from this commit[1] it doesn't seem to have anything it(?)
[1]: https://github.com/pytorch/pytorch/commit/bc92444b34dfbe3841...
Regardless - I want to be protected even when the Author of the package themselves is compromised (either by their tokens being stolen or them manually compromising the project)
That sounds like some combination of Crev[0], Sigstore[1], and Trillian[2].
[0] https://github.com/crev-dev/cargo-crev/blob/master/cargo-cre...
1. https://github.com/ossillate-inc/packj flags malicious/risky packages.
FRSCA: Factory for Repeatable Secure Creation of Artifacts provides a simple-to-install solution that aims to help secure the supply chain by building secure pipelines. It also provides abstractions and definitions with security guardrails ensuring all builds follow supply chain security best practices.
GUAC: Graph for Understanding Artifact Composition leverages metadata to provide deeper visibility and enable users to quickly understand security issues throughout the software supply chain, their blast radius, and how to remediate their root causes. Learn more as we build this out in the open source with our blog article.
https://www.kusari.dev/products/securesupplychain/
See also: https://slsa.dev/
Regarding manually reviewing software, it would be very easy to create a malicious majority of verifiers who certify malware, because malicious “verification“ takes much less resources than truthful verification. Rather than verifiers being payed, verifiers would need to themselves pay for the right to verify, in order to throttle malicious “verification”.
But realistically, you need verifiers that you trust in the first place, and if those exist, digital signatures are sufficient.
You'd think people would know better by now.
That’s… not great.
We're working on this at Chainguard by starting at the lowest levels (a containerized Linux distro) where we can deliver packages safely. Then we can use that layer to safely deliver language packages, like in this case.
This is all being done in the open, with projects like Sigstore (sigstore.dev) and our Linix distro Wolfi (wolfi.dev).
This is an incredibly complicated space and there's no silver bullet. We need to build safe delivery mechanisms for trustworthy software.
Linux distros have traditionally done this very well, but they've struggled to solve language package manager distribution, leading developers to shy away from and work around them when installing things like PyTorch.
Our hope is that we can bring the trustworthiness of Linux distros to language package managers, and the ease of language package managers to Linux distros.
sigh Why didn't anyone think of this attack vector till now?
Maybe this is the kick that lang devs needed to take dependency security seriously and finally make peer review mandatory for third party packages by default. If it doesn't, I hope we continue getting kicked in the balls until we do.
The root problem is "I'm using software from hundreds of authors, how do I know I can trust all of them?". And that problem is indeed unsolvable. It's not even a technology problem.
And no, peer review is not the solution. Scientific fraud is still widespread despite peer review being the standard for many decades. Granted, there may be better approaches than the Wild West that is PyPI and NPM, but this problem will never go away completely.
You will never have a theoretically perfect solution to this problem, but guess what: turns out you don't have to. Even single-reviewer systems like linux repositories have proven to be vastly more secure than this crap.
This problem is absolutely solvable and eventually it will be. I just hope they solve it the right way.
Yes? Especially if it's pulling in dependencies written by a bunch of other people because then it doesn't require malicious action by those three reputable devs, but merely negligence on their part in how those dependencies (and any updates etc. to them) are managed.
I encourage you to do more research on this topic.