Hacker News Reader

Comment by zokier | Hacker News Reader

I know python packaging has it's issues, but so far I personally have gotten pretty far with plain pip. The biggest shift to me was switching from original virtualenv to built-in venv module. On the other hand, if I wanted to be really serious about dependency management, I'd steal a page from FAANG, and build a monorepo and avoid all this hassle with package managers.

View on HN

I would really encourage you to try your hand at at a monorepo. I manage a python monorepo in prod and dependency management is hell. Poetry has some newer features that I am looking at trying to implement, but the state of the ecosystem wrt big monorepos is horrible.

Poetry support for mono repos is really horrible and all the maintainers are saying is „it’s a tool indented to publish packages, not manage your dev environment“.

Poetry has a non-package mode, though…

What is this non-package mode you are referring to?

I was talking about the maintainers not wanting to include features a lot of the userbase would like (like monorepo stuff), because they are saying their target audience is package authors, while in fact most of the users aren't.

I’ve been working on a thing [1] to make monorepo (“workspace”) builds easier, works well with Rye/uv. Doesn’t do anything during dev, just removes the need for hacks and scripts at build-time.

Would be curious if anyone thinks this is a useful direction, ultimately hope uv/hatch include something like this.

[1] https://github.com/carderne/una

I'm in a similar boat - uv's workspaces look tantalising.

yeah. its not great, we had to build out a poetry plugin that worked for our cases to support a mono repo, https://github.com/cloud-custodian/poetry-plugin-freeze?tab=...

the uv support on workspaces (virtual and concrete) has me intrigued.

> I manage a python monorepo in prod and dependency management is hell.

Why does it work for FAANG but not for you?

FAANGs vendor everything so there's only one version of a given dependency, and they pay people to make this work.

Bingo, this is what I was referring to in my comment. And yes, I assume it is quite a lot of labor. But I feel that lot of this perceived friction is from people trying to cut corners and avoid doing that labor, while still getting the benefits.

the overarching dream of computing in general is for a few people to do the labour and then for everyone to benefit through the magic of free replication.

if some smart and dedicated engineers can do the work to build a tool that lets everyone trivially manage a monorepo, that is certainly the best possible situation to end up in.

Why does that work for FAANG but not for drawnwren? How difficult is it to support “one version of a given dependency”?

It's as difficult as:

Package A depends on C 1.0 and B depends on C 2.0. How much work it is to get down to one version of C in your dependencies is up to how different 1.0 & 2.0 is, and how A & B use it. But if you want them resolved, it's up to you to do the engineering to A or B.

But pip doesn’t and can’t solve that issue, right?

Correct. The solution is to modify packages A and or B, which is a high cost approach (hence why FAANGs could throw warm bodies at it, but most everyone else throws their hands up in exasperation).

So the canonical solution to this is to use Bazel. The reason I did not end up using Bazel is because it is a pita to manage and I didn't have enough time to allocate to this problem to implement the FAANG solution.

The shell scripts I wrote are a painful, but less painful than dealing with Bazel.

FAANG might have more revenue per line of code

Are you managing your deps in the monorepo too, or do you have some sort of half-mono setup?

I would really encourage you to think more on this. Good enough for you... is not really the goal. We want standard tools that help with packages and virtual environments that can scale to an organization of 2 python devs... all the way up to hundreds or thousands of devs. Otherwise, it fragments the ecosystem and encourages bugs and difficult documentation that prevents the language from continuing to evolve effectively.

It would be great if that tool existed, but it doesn’t seem to right now. I can appreciate the instinct to improve packaging, but from an occasional Python developer’s perspective things are getting worse. I published a few packages before the pandemic that had compiled extensions. I tried to do the same at my new job and got so lost in the new tools, I eventually just gave up.

One of Python’s great strengths is the belief there should be one, obvious right way to things. This lack of unity in the packing environment is ruining my zen.

From ecosystem point of view I think all language specific package managers are crap. Instead of fragmenting the wider ecosystem on language boundaries, we should be looking more for language independent solutions. Nix is cool, as are buck2/bazel.

Normally I’d agree with you, but they’re talking about the built in tooling. That isn’t going away anytime soon. If it’s good enough for their purposes they should continue using it, especially with no clear consensus on a “better way”.

Nobody is saying that they shouldn’t continue using it?

Maybe I’m reading it wrong but that was the impression I was getting from that comment. I’m happy if I’m incorrect though.

As long as you don’t care about Python versions I too find pip + venv sufficient.

But there is no way for you to define which Python version your project is built against.

If you’re building a package you probably need to test against multiple versions. If you are building a “project” that isn’t an installable, distributed package but a bunch of code that is shared with a few developers and does something (run a machine learning model, push itself to a cloud function, generate a report) you probably want to target 1 and only 1 Python version.

As for the mono repo approach are you suggesting to copy paste numpy and pandas into the repo?

What you mean by "project" is a service. A service includes the Python interpreter. Choosing and provisioning the interpreter seems out of scope for pip or any other Python-specific tooling IMO. What if your service needs another interpreter like Ruby, or a database, message broker etc? Your Python tool won't help with that. This is what docker, and higher-level orchestrators like docker compose and kubernetes, and pyinstaller are for.

> Choosing and provisioning the interpreter seems out of scope for pip or any other Python-specific tooling IMO.

It's in-scope for Rye and hatch.

And I agree with that choice. Choosing a python version is as integral a dependency as choosing a version of numpy.

Sure, but you can only document that choice in your pyproject.toml, which is well-established. You have no idea how someone is going to provide that interpreter. You don't know what OS or how to provision an interpreter. If you are the person provisioning that interpreter then just use the tools you already know, not yet another tool to do basic stuff like install an interpreter.

That has been entirely my experience too: pip is good enough for me.

Reply on news.ycombinator.com