Hacker News Reader

Comment by hadlock | Hacker News Reader

I believe this is a system where a human/system builds a package and uploads and cryptographically signs it, verifying end to end that the code uploaded to github for widget-package 3.2.1 is the code you're downloading to your laptop for widget-package 3.2.1 and there's no chance it is modified/signed by a adversarial third party

View on HN

That's my understanding also, but I still feel like 'who cares' about that attack scenario. Am I just insufficiently paranoid? Is this kind of attack really likely? (How is it done, other than evil people at pypi?)

Yes, it is likely. It is done by evil intermediaries on hosts that are used to create and upload the package. It is possible for example if the package is created and uploaded on the developer laptop which is compromised.

---

From the docs:

> PyPI's support for digital attestations defines a strong and verifiable association between a file on PyPI and the source repository, workflow, and even the commit hash that produced and uploaded the file.

It still doesn’t protect against rogue commits to packages by bad actors. Which, IMO, is the larger threat (and one that’s been actively exploited). So while a step in the right direction, it certainly doesn’t completely solve the supply chain risk.

I'm not sure there is any way to completely solve supply chain risk. All you can do is raise the bar for a successful attack. Right now, we hardly have any controls at all.

It’s honestly a bit nuts that in 2024 a system as foundational as PyPI just accepts totally arbitrary, locally built archives for its “packages”.

I appreciate that it’s a community effort and compute isn’t free, but Launchpad did this correctly from the very beginning — dput your signed dsc and it will build and sign binary debs for you.

Build farms are expensive though. The PSF probably doesn’t have this kind of money and if they do, there’s a whole lot of other Python issues to fix.

https://xkcd.com/2347/ all over again.

Why not? It also accepts the totally arbitrary, locally typed "code" from the author.

I'm also happy with the Launchpad approach, but even happier with debian's Reproducible Builds approach; you _can_ provide a binary package that you built on your own machine, but several other people will _also_ build it. With a reproducible build, they will all get _exactly_ the same result for a given architecture. If they don't... we've found a problem.

I'd prefer to trust:

* the creator of the package over the distributor of the package

* a multitude of independent distributors who can all vouch the package builds the same for them, over a single distributor who builds it themselves and says "trust me bro"

Could you explain why you think it is a likely risk? Has this attack happened frequently?

It is likely in the same way that aliens are statistically likely. There is no evidence so far afaik, but you will likely never find out when it happens, and it can compromise the whole world if it were to happen to a widely used package. It is not worth the risk to not have the feature. I even think it should ideally eventually become mandatory.

If it happens, you won’t notice it even with PEP740 because all those “trusted publishers” already are or may easily be compromised by state actors. It’s all smoke and mirrors without reproducible builds.

Yes. Solarwinds.

Code checked out from the repository was not the same that was used to build the actual release. These are not high-likelyhood incidents but when they do occur, they tend to have high impact.

And more recently, semantically similar code-vs-build mismatch: CrowdStrike. The broken release was modified after the actual build step, as per their own build process, and the modified artifact is what was released.

To which CrowdStrike incident are you refering to? The global impact CrowdStrike incident was caused due to a driver defect which wasn't caught by quality assurance processes. It had nothing to do with malicious actors which were interfering with the code repository or the software deploment process.

The same incident, because the root causes are even more messed up than just shoddy QA.

Yes, the thing wasn't caught because of missing QA. What I find even worse is that the build process for their "channel files" involved:

    * building a release in CI, for which tests were run
    * modifying the built artifact as a post-process step, and
    * uploading this modified end result into their CDN infrastructure

In effect, what they actually built from their sources in the CI pipeline was not what was delivered to end users. You are correct in that attestations wouldn't help against such flagrant lies. And it wasn't a malicious act (although maliciously incompetent might qualify).

That post-build modification step would fly in the face of the attestation concept. It wouldn't help against having an empty set of tests, but an attestation-friendly build process at least would discourage messing around with the artifacts prior to release.

I'd encourage you to read the Verizon DBIR before making statements about whether a given attack is likely or not. Hijacking build systems is not likely: https://www.verizon.com/business/resources/reports/dbir/

A directed attack against the homegrown build system of a small company is unlikely. An attack against a high profile, centralized system or a commonly used package in such system is something to be prepared against.

You are correct. Start distributing and requiring hashes with your Python dependencies instead.

This thing is a non-solution to a non-problem. (The adversaries won't be MiTM'ing Github or Pypi.)

The actual problem is developers installing "foopackage" by referencing the latest version instead of vetting their software supply chain. Fixing this implies a culture change where people stop using version numbers for software at all.

Nothing stopping version numbers from being 512 bits long. It’s nice if they can be obviously correlated with time of release and their predecessor, which hashes alone can’t do.

It’s like HTTPS vs HTTP for your packages. It’s fine if you don’t care but having more secure standards helps us all and hopefully doesn’t add too much of a headache to provides while being mostly semi invisible for end users.

If I were a state actor trying to introduce vulnerabilities in an adversary's system, handing the guy who runs sqlalchemy (or whatever, pytorch, logger take your pick) $250,000 to modify their package with my vulnerability would probably be step 0 or step 1

1- Why not compile it? 2- does pip install x not guarantee that?

Yeah, because the problem with Python packaging is a lack of cryptographic signatures.

/"Rolls eyes" would be an understatement/

Reply on news.ycombinator.com