Hacker News Reader

> I agree, but I don't think this is PyPI's problem to solve. From a security perspective, PyPI should prioritize the platforms where the traffic is.

To me that's a bit of a weird statement, PyPI is part of the Python foundation, making sure that the project remains true to its open-source nature is reasonable?

My concern is that these type of things ultimately play out as "we are doing the right thing to limit supply chain attacks" which is good an defendable, but in ~5 years PyPI will have an announcement that they are sunsetting PyPI package upload in favor of the trusted provider system. pip (or other tooling) will add warnings whenever I install a package that is not "trusted". Maybe I am simply pessimistic.

That being said we can agree to disagree, I am not part of the PSF and I did preface my first comment with "I guess I am an idealist".

> making sure that the project remains true to its open-source nature is reasonable?

What about this, in your estimation, undermines the open-source nature of PyPI? Nothing about this is proprietary, and I can't think of any sane definition of OSS in which PyPI choosing to verify OIDC tokens from GitHub (among other IdPs!) meaningfully subverts PyPI's OSS committment.

> PyPI package upload in favor of the trusted provider system. pip (or other tooling) will add warnings whenever I install a package that is not "trusted". Maybe I am simply pessimistic.

Let me put it this way: if PyPI disables API tokens in favor of mandatory Trusted Publishing, I will eat my shoe on a livestream.

(I was the one of the engineers for both API tokens and Trusted Publishing on PyPI. They're complementary, and neither can replace the other.)

> What about this, in your estimation, undermines the open-source nature of PyPI?

Absence of support for self-hosting, in the spirit of freedom 0 = OSD 5&6? Or, for that matter, for any provider whose code is fully open source?

> Absence of support for self-hosting, or for that matter for any non-proprietary service?

This has nothing to do with self-hosting, whatsoever. You can upload to PyPI with an API token; that will always work and will not do anything related to Trusted Publishing, which exists entirely because it makes sense for large services.

PyPI isn't required to federate with the server in my basement through OpenID Connect to be considered open source.

I believe you that token uploads will continue to be possible, but it seems likely that in a couple of years trusted publishing & attestations will be effectively required for all but the tiniest project. You'll get issues and PRs to publish this way, and either you accept them, or you have to repeatedly justify what you've got against security.

And maybe that's a good thing? I'm not against security, and supply chain attacks are real. But it's still kind of sad that the amazing machines we all own are more and more just portals to the 'trusted' corporate clouds. And I think there are things that could be done to improve security with local uploads, but all the effort seems to go into the cloud path.

> or you have to repeatedly justify what you've got against security.

The only reason I started using PyPI was because I had a package on my website that someone else uploaded to PyPI, and I started getting support questions about it. The person did transfer control over to me - he was just trying to be helpful.

I stopped caring about PyPI with the 2FA requirement since I only have one device - my laptop - while they seem to expect that everyone is willing to buy a hardware device or has a smartphone, and I frankly don't care enough to figure it out since I didn't want to be there in the first place and no one paid me enough to care.

Which means there is a security issue whenever I make a new package available only on my website should someone decide to upload it to PyPI, perhaps along with a certain something extra, since people seem to think PyPI is authoritative and doesn't need checking.

The 2FA requirement doesn't need a smartphone. You can generate the same one time passwords on a laptop. I know Bitwarden has this functionality, and there are other apps out there if that's not your cup of tea. Sorry that you feel pressured, but it is significantly easier to express a dependency on a package if it's on PyPI than a download on your own site.

Sure. But PyPI provides zero details on the process, I don't use 2FA for anything else in my life, no one is paying me to care, I find making PyPI releases tedious because I inevitably make mistakes in my release process, I have a strong aversion to centralization and dependencies[1][2].

I tell people to "pip install -i $MY_SITE $MY_PACKAGE". I can tell from my download logs that this is open to dependency confusion attacks as I can see all the 404s from attempts to, for example, install NumPy from my server. To be clear, the switch to 2FA was only the triggering straw - I was already migrating my packages off of PyPI.

Finally, I sell a source license for a commercial product (which is not the one which got me started with PyPI). My customers install it via their internally-hosted PyPI mirrors.

I provide a binary package with a license manager for evaluation purposes, and as a marketing promotion. As such, I really want them to come to my web site, see the documentation and licensing options, and contact me. I think making it easier to express as a dependency via PyPI does not help my sales, and actually believe the extra intermediation likely hinders my sales.

[1] I dislike dependencies so much that I figured out how to make a PEP 517 compatible version that doesn't need to contact PyPI simply to install a local package. Clearly I will not become a Rust developer.

[2] PyPI support depends on GitHub issues. I regard Microsoft as a deeply immoral company, and a threat to personal and national data sovereignty, which means I will not sign up for a GitHub account. When MS provides IT support for the upcoming forced mass deportations, I will have already walked away from Omelas.

Have you maybe documented what you have done, so that others who want to follow the same path can look up some information?

No, I haven't. The main idea is to create your own in-tree build backend, described at https://peps.python.org/pep-0517/#in-tree-build-backends .

In short, use "backend-path" to include a subdirectory which contains your local copies of setuptools, wheel, etc. Create a file with the build hooks appropriate for "backend-path". Have that those hooks import the actual hooks in setuptools. Finally, set your "requires" to [].

Doing this means taking on a support burden of maintaining setuptools, wheels, etc. yourself. You'll also need to include their copyright statements in any distribution, even though the installed code doesn't use them.

(As I recall, that "etc" is hiding some effort to track down and install the full list of packages dragged in, but right now I don't have ready access to that code base.)

Thanks for the info.

Thank you for being the first person to make a non-conspiratorial argument here! I agree with your estimation: PyPI is not going to mandate this, but it’s possible that there will be social pressure from individual package consumers to adopt attestations.

This is an unfortunate double effect, and one that I’m aware of. That’s why the emphasis has been on enabling them by default for as many people as possible.

I also agree about the need for a local/self-hosted story. We’ve been thinking about how to enable similar attestations with email and domain identities, since PyPI does or could have the ability to verify both.

If there is time for someone to work on local uploads, a good starting point would be a nicer workflow for uploading with 2FA. At present you either have to store a long lived token somewhere to use for many uploads, and risk that it is stolen, or fiddle about creating & then removing a token to use for each release.

> I believe you that token uploads will continue to be possible, but it seems likely that in a couple of years trusted publishing & attestations will be effectively required for all but the tiniest project.

That's what I think will happen.

> And maybe that's a good thing? I'm not against security, and supply chain attacks are real.

The problem is the attestation is only for part of the supply chain. You can say "this artifact was built with GitHub Actions" and that's it.

If I'm using Gitea and Drone or self-hosted GitLab, I'm not going to get trusted publisher attestations even though I stick to best practices everywhere.

Contrast that with someone that runs as admin on the same PC they use for pirating software, has a passwordless GPG key that signs all their commits, and pushes to GitHub (Actions) for builds and deployments. That person will have more "verified" badges than me and, because of that, would out-compete me if we had similar looking projects.

The point being that knowing how part of the supply chain works isn't sufficient. Security considerations need to start the second your finger touches the power button on your PC. The build tool at the end of the development process is the tip of the iceberg and shouldn't be relied on as a primary indicator of trust. It can definitely be part of it, but only a small part IMO.

The only way a trusted publisher (aka platform) can reliably attest to the security of the supply chain is if they have complete control over your development environment which would include a boot-locked PC without admin rights, forced MFA with a trustworthy (aka their) authenticator, and development happening 100% on their cloud platform or with tools that come off a safe-list.

Even if everyone gets onboard with that idea it's not going to stop bad actors. It'll be exactly the same as bad actors setting up companies and buying EV code signing certificates. Anyone with enough money to buy into the platform will immediately be viewed with a baseline of trust that isn't justified.

As I understand it, the point of these attestations is that you can see what goes into a build on GitHub - if you look at the recorded commit on the recorded repo, you can be confident that the packages are made from that (unless your threat model is GitHub itself doing a supply chain attack). And the flip side of that is that if attestations become the norm, it's harder to slip malicious code into a package without it being noticed.

That's not everything, but it is a pretty big step. I don't love the way it reinforces dependence on a few big platforms, but I also don't have a great alternative to suggest.

Yeah, if the commit record acts like an audit log I think there’s a lot of value. I wonder how hard it is to get the exact environment used to build an artifact.

I’m a big fan of this style [1] of building base containers and think that keeping the container where you’ve stacked 4 layers (up to resources) makes sense. Call it a build container and keep it forever.

1. https://phauer.com/2019/no-fat-jar-in-docker-image/

>> PyPI package upload in favor of the trusted provider system. pip (or other tooling) will add warnings whenever I install a package that is not "trusted". Maybe I am simply pessimistic.

> Let me put it this way: if PyPI disables API tokens in favor of mandatory Trusted Publishing, I will eat my shoe on a livestream.

Yeah, sure. But parent poster also mentioned pip changes. Will you commit to eating a shoe if pip verifies attestations? Of course not, you know better than I do that those changes to pip are in the works and experimental implementations already available. You must have forgotten to mention that. PyPI doesn't have to be the bad guy on making sure its a PITA to use those packages. Insert Futurama "technically correct, the best kind of correct" meme.

> You must have forgotten to mention that.

Insinuating dishonesty is rude, especially when it’s baseless: the post linked in this one is explicit about experimenting with verification. There’s no point in signing without verification.

pip may or may not verify attestations; I don’t control what they do. But even if they do, they will never be able to mandate them for every project: as I have repeatedly said, there is no plan (much less desire) to mandate attestation generation.

Regardless of your intent, you didn't acknowledging the parent poster's concern about python overall. Just exculpated PyPI.

I believe you would have received a much less pushback if you hadn't been coy about the obviously impending quiet deprecation (for lack of a better phrase) you finally acknowledged elsewhere in the thread.

There is no quiet deprecation. Literally nothing is deprecated in this announcement.

I was reaching for a descriptor and ended up riffing off 'quiet quitting' and 'quiet firing'.

I understand you're not deprecating or recommending anything, and have learned in the meantime just how ugly python infighting is in and around package tooling. I can see the motivation to keep external messaging limited to saying a feature was added and everything else remains constant.

I work with a bunch of normies that think python packing starts and ends at "pip install"-ing systemwide on Windows. One day in the future the maintainers of the packages they use will likely be encouraged to use this optional new feature (and/or they already have been publicly). Even later in the future those end users themselves might be encouraged by warnings or errors not to install packages without this extra feature.

PyPI has deprecated nothing. Bureaucrat Conrad, you are technically correct. The best kind of correct.

> I understand you're not deprecating or recommending anything, and have learned in the meantime just how ugly python infighting is in and around package tooling. I can see the motivation to keep external messaging limited to saying a feature was added and everything else remains constant.

You're insinuating an ulterior motive. Please don't do that; assume good faith[1].

(You can find trivial counterexamples that falsify this: PyPI was loud and explicit about deprecating and disabling PGP; there's no reason to believe it would be any less loud and explicit about a change here.)

[1]: https://news.ycombinator.com/newsguidelines.html

I'm with @belval on this one, it's ok to prioritize github, but people that want the standard to implement an alternative should not feel like they are doing something that may not be supported.

It kinda feels like that right now.

Again, to be clear: the standard does not stipulate GitHub or any other specific identity providers. The plan is to enable GitLab and the other Trusted Publisher providers in short order.

This is exactly the same as Trusted Publishing, where people accused the feature of being a MSFT trojan horse because GitHub was enabled first. I think it would behoove everybody to assume the best intentions here and remember that the goal is to secure the most people by default.

I think the point is that this needs to be made clearer in the official docs from the get go.

It's said explicitly in the second sentence in the usage docs[1].

> Attestations are currently only supported when uploading with Trusted Publishing, and currently only with GitHub-based Trusted Publishers. Support for other Trusted Publishers is planned. See #17001 for additional information.

[1]: https://docs.pypi.org/attestations/producing-attestations/

At this point it should be fairly obvious that if you have to defend the phrasing in multiple threads here on HN, get some folks to help rephrase the current document instead so you can comment with "we updated the text to make it clear this is a first pass and more options are getting added to the doc soon".

If you draw an ugly cat, and someone tells you it's ugly, it doesn't matter how much you insist that it isn't, and the same is true for docs. It doesn't matter what your intention was: if people keep falling over the same phrasing, just rephrase it. You're not your writing, it's just text to help support your product, and if that text is causing problems just change it (with the help of some reviewers, because it's clear you think this is phrased well enough, but you're not the target audience for this document, and the target audience is complaining).

Anyone can run an OIDC system if they want. But PyPI is not under an obligation to trust an OIDC provider running on a random rpi3 in your basement. More than that, GitHub is "trusted" because we can be pretty sure they have an on-call staff to handle incidents, that they can reliably say "This token was provided on behalf of this user at this time for this build", etc.

Even if you standardized the more technical parts like OIDC claim metadata (which is 100% provider specific), it wouldn't really change the thrust of any of this — PyPI is literally trusting the publisher in a social sense, not in some "They comply with RFC standards and therefore I can plug in my random favorite thing" sense.

This whole discussion is basically a non-issue, IMO. If you want to publish stuff from your super-duper-secret underground airgapped base buried a mile underneath the Himalayas, you can use an API token like you have been able to. It will be far less hassle than running your own OIDC solution for this stuff.

If I can't build on a rpi3 in my basement and am forced to use GitHub that's exactly against the spirit of open source

You still can. You just use an API token with PyPI.

Please improve your reading comprehension. I swear, this website is embarassing sometimes. You can still do this with an API Token. You can upload from a C64 with an API token. What you cannot do is run some random OIDC provider on your random useless domain and have PyPI magically respect it and include as part of the Trusted Publishers program. There is no point in it, because the program itself is constrained by design because it only provides any benefit at "large scale." Your random dumb server providing a login for you alone does not provide any benefits over you just using an API Token.

Any pathway to provide trusted attestations for random individual Hacker News users like yourself will, in fact, require a different design.

> error-prone cryptographic bits if they don't need to

They can't. Because you wouldn't accept their key anyway.

I think you're missing something. The key in question is a short-lived ECDSA key that lives inside a publishing workflow and is destroyed after signing; neither GitHub nor the Sigstore CA generates a signing key for you.

PyPI will accept any key bound to an identity, provided we know how to verify that identity. Right now that means we accept Trusted Publishing identities, and GitHub identities in particular, since that's where the overwhelming majority of Python package publishing traffic comes from. Like what happened Trusted Publishing, this will be expanded to other identities (like GitLab repositories) as we roll it out.

How does pypi know I'm not github? Because I can sign with my keys and not with github's key.

Never mind all the low level details of the temporary keys and hashes and all of that. This is an high level comment not a university book about security.