Publish a Python Wheel to GCP Artifact Registry with Poetry | Hacker News Reader

Publish a Python Wheel to GCP Artifact Registry with Poetry

sergiolema.dev

25 pointssergiolema19 comments

GCP Artifact Registry is an OCI Container Image Registry.

It looks like there there are a few GitHub Actions for pushing container image artifacts to GCP Artifact Registry: https://github.com/marketplace?query=artifact+registry&type=...

FWIW, though it may not be necessary for a plain Python package, "pypa/cibuildwheel" is the easy way to build a Python package for various platforms in CI.

SLSA.dev, Sigstore;

GitHub supports artifact attestations and storing attestations in an OCI Image store FWIU. Does GCP Artifact Registry support attestations?

"Using artifact attestations to establish provenance for builds" https://docs.github.com/en/actions/security-for-github-actio...

> GCP Artifact Registry is an OCI Container Image Registry.

That is one of the supported formats (and maybe most common), but not the only one.

https://cloud.google.com/artifact-registry/docs/supported-fo...

The Python one behaves just like PyPI, you just need to specify the URL provide credentials.

GitHub specifically doesn't have Python package index (PEP 503, PEP 740) support on their roadmap: https://github.com/github/roadmap/issues/94#issuecomment-158...

GitLab has Python package registry support (503?): https://docs.gitlab.com/user/packages/pypi_repository/

Gitea has Python package registry support (503?): https://docs.gitea.com/usage/packages/pypi

PyPI supports attestations for Python packages when built by Trusted Publishers: https://docs.pypi.org/attestations/ :

> PyPI uses the in-toto Attestation Framework for the attestations it accepts. [ in-toto/attestation spec: https://github.com/in-toto/attestation/blob/main/spec/README... ]

> Currently, PyPI allows the following attestation predicates:

> SLSA Provenance, PyPI Publish

Artifact Registry > Artifact Registry documentation > Guides > Manage Python packages: https://cloud.google.com/artifact-registry/docs/python/manag... :

> [Artifact Registry] private repositories use the canonical Python repository implementation, the simple repository API (PEP 503), and work with installation tools like pip.

PEP 503 – Simple Repository API: https://peps.python.org/pep-0503/

PEP 740 – Index support for digital attestations: https://peps.python.org/pep-0740/

Recently, I've been building a project in Python that doesn't have a Docker image as output. Instead, I need a runnable file. Why? Because I need to talk to the machine directly and not to Docker. Because I need to talk to the GPU drivers directly and not to another abstraction layer.

So, I needed to create a runnable file in Python. I needed to create a wheel file. All this with Poetry and integrate it into my CI/CD pipeline.

Poetry has a command[0] to set a version in a pyproject.toml file. Why did you choose sed over that?

[0] https://python-poetry.org/docs/cli/#version

How many of you are publishing python packages to registries other than PyPI? I'm curious as I've only ever published to PyPI.

At my job, we use AWS CodeArtifact to host a couple dozen internal libraries we use for Python and TypeScript projects. I suspect that this is a common use case for these kinds of artifact repositories.

what's your experience with AWS CodeArtifact? We are migrating to AWS and we are in doubt about using it or our internal Nexus server.

To access a private CodeArtifact repository, you have to first fetch a short-lived token, then supply that as the password when you access it via npm/yarn, poetry, etc. In most cases, this is an inconvenience that can mostly be paved over with the AWS CLI or a shell alias.

This quickly get messy though. We use AWS CDK and build our assets in a Docker container. Each time the token changes, Docker invalidates a bunch of layers and rebuilds the image. AWS CDK sees that and uploads a new .zip to S3 or an image to ECR. Then Security Hub sees a new Lambda function or image, scans it, and carpet bombs my email whenever a CVE is found.

It's ... not ideal.

At my company we're using GCP Artifact Registry for an internal PyPI proxy as well as to publish (proprietary) Python wheels to.

That’s cool! Maybe I’ll do that at my next workplace.

I've used https://github.com/gorilla-co/s3pypi or some variant of it in a few places. Basically cloudfront + s3 instead of a dedicated artifact registry.

I've been doing it to Azure (private package repo) since their instructions were ok. I would like some of these to publish instructions for uv as well (instead of just pip/twine).

I've published them to Artifactory's internal pypi, and also to Gemfury. It all works decently.

Someone needs to make a site like You Might Not Need jQuery but You Might Not Need Docker. It's unfortunate to me these days that putting everything into containers seems to be the default.

These aren’t normal containers, they are “artifacts” which are just the manifest and a single layer and a mime type.

OCI artifacts are one of the most under hyped technologies. Almost everyone already has an image registry, now that registry supports basically any kind of package and with nice versioning semantics.

Why wouldn’t you use it? I don’t want to auth again into something else and deal with different registries for every language I have.

Definitely. OCI artifact repos are much closer to the Artifactories of yore (and today) than they are to "it's all docker".

Most major tech companies I've worked with have an internal OCI repo of some kind maintained; internal packages are available and external packages are cached (or blocklisted, in certain situations, which is its own flavor of helpful for security risks) and everything from Docker to Maven to Pip/Poetry/etc. Just Works.

The CUE configuration language uses OCI artifacts as the storage/distribution mechanism of its package manager

There is no docker here.

https://cloud.google.com/artifact-registry/docs/python

This is just a private PEP503 repository, your own private PyPI.

Reply on news.ycombinator.com