Python Practical Package Packing 2024 | Hacker News Reader

Python Practical Package Packing 2024

matt.sh

35 pointskr0bat35 comments

While the author enumerates the things they think you should do, I think it was very light on the _why_ you would want this setup. I think the argument of "you are doing it the old way" have been really ineffective at convincing people to do anything different, particularly around python tooling.

The official docs recommend doing many of the things the author cautions against or calls bad. https://packaging.python.org/en/latest/guides/distributing-p...

Yeah I gotta agree here, I'm not sure I've seen someone so opinionated with python. I tend to try to use things built into python or official recs until I hit an actual problem that needs to be solved.

This is why I'm not posting this article to my team's slack. There's some good stuff in the article, but my team also has some well-meaning over-optimizers that will see this and start throwing in orjson even though json (de)serialization is an insignificant impact to our execution time, or loguru even though our code base has stupidly simple logging requirements, and they'll do it because they'll see this article stating authoritatively that those are the correct choices without considering the value of simplicity, the longevity of standard libraries, or whether the problems those libraries are trying to solve actually apply to us. And they'll do that because the author of this article failed to address or even acknowledge ANY of those concerns.

> using loguru when you could be using structlog

Oof. Yikes. Outdated much? I kid, but it's way too easy to be dogmatic about stuff like libs.

As you scroll down, it gets worse.

"Bonus: Legacy Outdated Python Usage Warning Signs" was a helpful list of things to look at, but I disagree with many as best practices. I like when code from the year 2000 still works, and you do that by using standard packages.

The chase for the latest / greatest means bitrot and is the sign of a junior developer. You save a bit of time in the short term, but a new latest / greatest comes out a few years later.

The conclusion is a bit weird too. That kind of rant is what leads to no-hire decisions. Be careful what you put online.

Matt needs to stop this "I know better than you" attitude.

Seriously, first C and now Python.

I can't wait for the 20th revised version of this blog post where Matt has slowly realises he doesn't know half as much as he claims to and has added 50 side notes explaining all the cases where his blanket statement is not quite as broadly applicable as he ignorantly assumed.

Poetry is so slow, please use uv, for some of my projects, poetry takes 20-30 seconds to update/add new packages while uv completes it in seconds. But I can agree that's better to use a package manager than plain old `requirements.txt` file since you can pin the version you want to use for development or production.

Does uv support platform independent lockfiles now?

Yeah the `--universal` flag should do that.

  uv pip compile --universal requirements.in -o requirements.txt

Given how much the author talks about professional standards, one would think they would write professionally.

Also, is the author aware of why people use Anaconda? Conda environments can make it significantly easier to link CUDA or Fortran libraries properly, which are quite prevalent in scientific computing. Many people who use such code bases are professional scientists and not professional programmers, so I understand seeing that generally conda-based packages are poorly architected.

It's almost like other people sometimes do things differently because they have different needs or have thought of things you haven't, not just because they're too stupid or uneducated to know better.

I've been liking pdm as opposed to poetry, as poetry uses some non-standard syntax in the pyproject.toml file. I had a lot of trouble getting hatch environments to handle changes to default python versions and the like. I like how pdm played nicely out of the box with pyenv if you're using it, otherwise manages environments directly.

Nothing I have used so far in the Python ecosystem comes even close to Rye. Everything is incredibly fast, environments work across multiple operating systems, etc. For me, Poetry has been a massive headache (try installing PyQt6 Windows).

Just feel this needs to be solved higher up the pipeline than where they're trying to solve it. The fact multiple versions of the same library can't coexist in a project meaning you end up having to find this plateaux where every dependency is within range if they cross over. Node/NPM can do this, don't really see why Python couldn't eventually and read dependencies from within the project folder rather than this whole fake idea that it exists at a system level which just hasn't been the reality of how people are using this language for a long time.

For someone that is from the Ruby world and uses rbenv / bundler confortably, what would be the canonical way of using / managing Python in MacOS?

There seems to be a sea of alternatives and I see every tutorial mention `pip install` while I don't even have that running in my CLI (only pip3). Do people assume an alias here of I have somehow messed up my environment?

I think people assume an alias exists. Same with `python` being an alias for `python3`. Some linux distros default to creating that alias when you install python3.

I like Python, but those version suffixes are a very common problem. But in 2024 python should be python 3 and pip should be python3-pip.

This situation is also partially why there is so much python tooling around versions, environments and dependencies.

I would love to have a little less dynamic Python, that could be compiled to not huge binaries.

I use pyenv, seems to work very well.

Pixi [1]. A lot of people recommend UV, which pixi uses under the hood to install pip packages [2].

The difference between UV and pixi is that pixi is better tooling for working with Conda repositories, and UV focuses in working with pip packages. Pixi allows installing not only pip packages but anything that has been packaged for Conda, which includes a lot of non-python stuff (C/C++ libraries and more).

For instance, I use it to install the Go compiler per-project, since using a global installation tends to grow "polluted" with every single go package that you have ever installed. You could even use it to install Node, NPM or ruby per-project, it just works. It also supports lock files [3].

1: https://pixi.sh/latest/

2: https://prefix.dev/blog/uv_in_pixi

3: https://pixi.sh/latest/features/lockfile/#why-a-lock-file

Tangent: to run things with pixi, you need to either 1) `pixi run thing` 2) `pixi shell` to spawn a new shell with adjusted env or 3) `eval "$(pixi shell-hook -s bash)"` (or similar for other supported shells) to modify your current session.

I found that direnv [1] really improved the ergonomics for me. There's some misunderstanding that direnv requires Nix but that's not true. It does one thing and does it well: load .envrc files when you cd that path.

I have a lil `.envrc` that looks like this:

    # Load pixi shell settings.
    eval "$(pixi shell-hook -s bash)"

After that, you can just run `python` and it will run whatever python you asked it to install (`pixi add python` adds the latest python 3 in conda by default).

1: https://direnv.net/

I think it's fine to have high-concept dependency management for the Serious Python Professionals, but what about for people who just want to put a project together and not have their life taken over by figuring out YAML syntax, new-hotness build tools, and dependency graph structures?

Ironically, the same author has a blog post deploring the evil of Too Much Technology: https://matt.sh/matt-curve

Due to supply-chain attacks, for your cybersecurity, I strongly encourage everyone to use a container for each project for development purposes. I do this using devcontainer.json in VSCode which is free. PyCharm too offers devcontainer integration but not for free, so I moved away from PyCharm. You can alternatively use a plain docker container if your IDE integrates with it, but devcontainer is more suited for this purpose. All this is assuming that you don't use a web IDE which already offers project isolation.

Please let the magical pi version of py, 3.14, get python to a suitable, standardized vessel for packaging.

I realize the Venn of packaging has copious disjoint functionality slices.

Too, what one does for a bit of one-off, non-typed cloud manipulation is different than for a full-on, re-usable library.

But the fabulous disaster that is the python packaging landscape was tiresome a decade ago.

Maybe the PSF can sponsor a kickstarter so that peanut gallery mouths like mine can contribute funds where our skills are tapering off.

I will change my projects to use pyproject.toml and remove all the requirements.txt. Looking forward to changing it again for py_requirements.yatxml in 5 years

Enjoyed the article, I think it mentions some good practices, although it’s missing some topics that may or may not be worth including, depending on your view of what constitutes ‘packaging’ in python.

This bit towards the end certainly came out of left field, but made me laugh:

dawn your peepers on some of these fun adventures in hiring i’ve had recently:

rejected in interviews by a 25 year old senior hiring manager rejected in interviews by a 28-something who had worked at google for 5 years and had a PhD from stanford rejected for arguing, akshually, i made more 10 years ago than what your scale is rated at rejected by some founders who got rich by selling an ios fart app to apple ten years ago rejected via HR/email/auto-reject a couple places at least nobody has ever accused me of not being sufficiently candid.

you can put lists into hn like so, with two newlines

- item 1

- item 2

> BAD: you use the built-in python logging module. It is a complete disaster of an API. You should use loguru everywhere instead.

???

Isn't `src`, as in `src/packagename/__init__.py`, a thing any more?

It is. That is more common than the author's suggestion.

tl;dr use poetry and write decent code

uv rules and poetry drools; I like the idea of poetry but once you see how fast uv can build your environment whether or not the files are in cache I think you could never go back to poetry.

Is the UX comperable? Might try this

uv feels a lot like pip but unlike pip it has a correct resolution algorithm. I understand that people are working on a higher-level product that incorporates uv.

I haven't used uv but it looks like its trying to be a drop-in replacement for pip which would therefore lack everything that makes poetry great.

Rye on the other hand seems to be extremely poetry-like while being written in rust for speed: https://rye.astral.sh/

But personally, I can't justify switching my employer to experimental python dependency management over something as inconsequential as taking a minute to install dependencies so I'm sticking to poetry until Rye has some years of development behind it.

Same here... not gonna switch tools every 6 months for such a small benefit. I like that it downloads its own interpreters though, makes it easy to start a project.

Reply on news.ycombinator.com