Hacker News Reader

> I am sick and tired of the python packaging space pushing this godawful tool for Python. Please just use pip-tools or poetry instead.

As an outsider, Python packaging seems like a shit show, sorry. This bug in pipenv is bad, yes. But poetry was deliberately non-deterministically failing CI builds that used a deprecated feature.

Maybe pip-tools is reliable and has a reasonable maintainer, but having to manually create venvs is mediocre UX.

PDM sounds interesting because it doesn't need venvs under the hood, thanks to a recent PEP.

> Python packaging seems like a shit show

It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third.

The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.

>primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third.

That never stopped npm to be way better than all of them...

>The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.

So, the "solution" is "just give up"...

Npm doesn't do environments. You have the choice of "in this directory" or "globally" and pray there are no symlinks involved or things fail mysteriously. It doesn't quite do packaging either.

So it lands in a similar space to what the parent describes - mostly a good dependency manager.

>Npm doesn't do environments.

"This directory" (which can be overriden with env setting or similar advise for where to lookup) works fine for me, and never have been a problem in quite complex setups. Not sure why "symlinks" would be involved.

In any case, what NPM does is leaps and bounds ahead of any shit Python offering we have to deal with, which is a constant source of trouble.

IIRC, it's also the model ("this directory") that there's a PEP to imitatate, not sure what happened to that...

Back in the days we had the .eggs directory in python it worked fairly well except for binary packages so we got virtual environments.

What was the problem with binary packages for .eggs?

Because, iirc, npm does handle e.g. .so files in packages that are based on C/C++ code, and puts them and everything else in the same, relocatable even (another pain point with virtualenvs), .node_modules directory.

> which can be overriden with env setting

Support for this is not great. Lots of packages still don't support this properly. My experience matches the 2015 comment https://github.com/npm/npm/issues/775#issuecomment-71294085

> Not sure why "symlinks" would be involved.

If you make your node_modules a symlink, multiple packages will fail. Even if you're not interested in doing that, others are. I tried the be symlinks because the env var option is not reliable.

> What NPM does is leaps and bounds ahead

Unless you change your node / gyp version. It doesn't really have a concept of runtime version. You can restrict it, but not have two concurrent versions if they conflict.

What is the reason for using a symlink for your node_modules directory?

I was trying to quickly switch between installed node_modules without a full install run each time. Also, wanted to do some clever caching of a large module between projects.

You meant to write „I am more used to npm and intuitively work around the quieks while in python i run into these“

No, I meant to write "I've worked with Python for 20+ years, and Node for 8, and Python has always been a shit show on that front, whereas npm haven't given me any trouble (1)".

But thanks for the uncharitable strawman. Couldn't be that I know both and don't like it, must be that I'm not familiar enough with the wonderland that is the Python package management...

(1) At least not any trouble of that kind. I've had some issues regarding codependencies for add-ons, but that's a problem far beyond the kind of problems the Python package management has. If only Python's packaging stuff only had the same kind of issues...

What makes NPM a cut above Python package managers is that I can clone a JavaScript project, run npm install, and it will just work in such a way that it will not conflict with other projects.

With python I have to work out which package manager it's using (the JavaScript ecosystem also has multiple package managers - yarn and pnpm - but they're compatible with each other!), and there's a significant chance it's using pip which installs packages globally unless I manually create and manage a virtualenv.

The gold standard is probably Rust's "cargo" (and "rustup" for toolchain version management), which broadly follows the NPM model but with a better implementation with fewer quirks. But NPM has gotten much better (it uses lock files by default now for example), and is mostly painless at this point.

> But NPM has gotten much better (it uses lock files by default now for example)

Last time I checked, "npm install" will still overwrite your lock file, you have to explicitly use "npm ci" for that not to happen.

Or maybe that changed recently?

Looks like that's still the case. It does still use lockfiles, it just updates them by default. There is also as you mention a built-in command that does the right thing.

Right, it's just one of these things that inexperienced people (or people coming from other languages) would get wrong, so no idea why they have this behaviour.

It depends on the context.

* If the package.json file can be satisfied by the lock file (i.e. if the two files match) then the lock file will be used unchanged, even if there are newer versions available. This is (mostly) identical to the `npm ci` command.

* If the package.json requires more recent versions of a particular package than are available in the lock file then this package will be updated (but other packages will remain the same) and a new lock file will be written. This happens if, for example, the package.json file has been changed by hand, and so the lock file is no longer in date. In contrast, the `npm ci` command exits and fails to install the dependencies.

* Similarly, if the package.json file has new dependencies that aren't in the lock file, it if dependencies exist in the lock file that are no longer present in package.json, then these packages will be installed or deleted and the lock file updated. Theoretically, only the relevant dependency and its dependencies will be affected, but because if the deduping algorithm, NPM may make other changes if it thinks it can simplify the dependency tree. Again, in this situation, `npm ci` will just fail.

There is one other minor exception. The `npm ci` command will never make any changes to the lock file, but `npm install` may still choose to update the lock file with additional metadata (for example if the lock file was created with an older version of npm). This will never change the versions, but it may look a bit messy in the commit history, particularly if different people on a team are using different versions of npm. As a result, I tend to use npm ci as my default anyway.

However, as a role of thumb, if `npm ci` is able to successfully install dependencies, then running `npm install` will always install the same dependencies. The only difference is when the package.json file disagrees with the lock file, in which case `npm install` will probably do what you want, unless you want some very specific results in which case you still have the npm ci command available.

(This has been true since at least npm v5, although I think there were some bugs in the initial implementation.)

Why wouldn't one wish to update the lock file by default?

Because you want reproducible builds. Updating dependencies should be an explicit operation, distinct from installing from a known state.

what? npm is, for a lack of better word, shit. not worse than Python situation (coming from somebody who was excited when new-style classes were introduced), but that is a very low bar. ($DAYJOB makes me use npm _and_ pnpm, nx, nvm, corepack and I guess some other tools I don't know about just to make the goddamn thing build.)

Any argument why is it shit?

it's so slow there's a whole ecosystem to avoid using it, edited my comment above.

I think it's not npm/yarn that is slow, it's that Node project have tons of dependencies and transitive dependencies and such (leftpad, anyone?)

But that's more a fault of the library ecosystem and culture, than the design of npm/yarn

> That never stopped npm to be way better than all of them...

npm isn't my favourite, but it's definitely improved with the competition from yarn and pnpm.

I think that Stack (Haskell) and Cargo (Rust) are state of the art, for ease of use and reproducibility. The other user would describe these as cramming three functionalities into one, but they do it so well.

Stack is good, but the Haskell community seems split between Stack and pure Cabal users... plus there's also ghcup and it's not easy to understand whether one should have stack manage GHC versions or no... every source tells you something different.

I also hate that there's no good way to install a depedency globally so you can play around with it (you kinda can, but it's very hard to remove it afterwards).

But the real shitshow is the HLS that just crashes on me all the time and doesn't always work correctly with Stack.

Alas, wheels are terrible for providing anything but a Python API. If you have headers, compiled tools, or some other complicated SDK, good luck (NumPy is excepted here because it only ships headers this way and chooses to live with wheel limitations). Wheels depending on you also ship copies of your library (if you use PyPI), so the lackluster dependency management is just waiting to mix things up and give your users "fun" linker/runtime issues.

> It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third.

Many build tools in other languages also do all of these three things - for example Ruby's Bundler or maven/gradle for JVM languages. And they're pretty reliable at all of these tasks.

(I guess, "virtual environments" is maybe not the right description. What these tools all do is isolating dependencies of different projects from each other, although they might do it in slightly different ways.)

Package management in Python is mostly solved to me with conda+conda devenv+conda lock.

Sometimes I still need to get packages from pip in that scenario, but conda makes package management a lot better than pip+alternatives for me.

PDM in my experience has been the best, plus it's rapidly growing, and is more PEP compliant than poetry. My go-to choice, been advocating it since I found out about it!

I have used python for 15 years but less frequently nowadays because engineering manager. What irks me out about poetry is that it tries to be too magic and in the end just is oddly surprising. Will pdm feel „more solid“ to me?

> But poetry was deliberately non-deterministically failing CI builds that used a deprecated feature.

I’m having difficulty finding a reference on this, could you elaborate?

https://github.com/python-poetry/poetry/blob/a613347bf0a9fd7...

Interesting that you call out venv creation as a UX problem. I never considered it any worse than having to `cargo new` a project. I find having to activate and deactivate vs trust a 3rd party project that watches my every `cd` a much bigger issue.

There is no good alternative. Maybe poetry is the least worst, but it is a fractal trash fire like the rest of the python ecosystem.

To name some issues i've had with poetry(and keep having):

- Updating packages does not properly take effect sometimes. It will sometimes keep using old code, and only deleting the entire underlying venv fixed it. In this instance, I was using a wheel package locally. - It has incredibly poor UX and will just vomit raw python errors when you use it slightly incorrectly. - There's so many bugs in its handling of CLI args and getting a typical python None error is very common - It is ridiculously slow and doesn't cache things properly

I could probably go on but I'd have to sit and think a bit about it, and that's pretty much the opposite of what I want to do: forget all about python and poetry.

> pipenv's dependency resolution is the slowest among all the tools in this space including poetry, pip-tools, hatch and so on

"Slow" doesn't just mean "slightly annoying", mind you.

I was once tasked to properly package up an Airflow installation. Not being a Python guy, I tried doing it with pipenv because we were using it for other projects.

It was impossible to resolve the dependencies because the resolver would routinely throw a timeout error. In the end I gave up and switched to Poetry. That was still a bit slow (I think Airflow just has a messed up dependency graph...), but it wrote the lockfile in a reasonable amount of time and just worked thereafter...

Personally i've never really seen the advantages any of these tools over having a requirements.txt, or slightly fancier and more modern, using a pyproject.toml file.

e.g. in a recent project the entire package configuration is this pyproject.toml file:

    [project]
    name = "vera"
    version = "0.0.0"
    requires-python = ">=3.10"
    authors = [
        { name = "Doxin" }
    ]
    dependencies = ["moderngl==5.7.3", "pysdl2==0.9.14", etc etc]
    
    [project.optional-dependencies]
    dev = ["pre-commit==2.20.0"]
    
    [tool.setuptools]
    packages = ["vera"]

Getting started with development is as easy as

    python3.10 -m venv venv
    source venv/bin/activate
    pip install -e vera[dev]

Updating packages is just rerunning the pip install command.

Not to mention the horrible configuration anti-pattern where everything depends on environment variables.

Environment variable tweaks are for local-only specific fixes, not for reproducible/distributable configuration.

> not for reproducible/distributable configuration

At my work, we use environment variables all the time for configurations, e.g. for telling a kubernetes pod where to find the database. In your opinion, what is the better option(s)?

Config files, and command line arguments as a very distant second

Secret managers. All cloud providers have them built in, and on other platforms you can easily whip up a self-hosted one. See https://blog.diogomonica.com//2017/03/27/why-you-shouldnt-us...

I recommend against with using secrets managers as opposed to config files for storing configuration. Ideally, your configuration is versioned and code-reviewed. Config files are optimal. Even secrets used by your deployment should be stored in configuration - encrypted, of course. The benefit of versioned configuration files, secrets included, cannot be overstated.

You can encrypt sensitive configuration fields with tools like `mozilla/sops` [1], which will reach out to your KMS or secrets manager of choice to encrypt/decrypt sensitive fields on the fly.

This way, you are minimizing the splitting of state across secrets managers and your code. Your configuration is stored at code, code reviewed, and versioned, which has devops and security benefits of its own.

[1]: https://github.com/mozilla/sops

The problem with environment variables as configuration is that it's unstructured, hardly documented, and overall hard to reproduce and inspect.

Nothing is worse than trying to understand an issue with a program that heavily relies on environment variables for configuration, as environment variables are designed to be short-lived, memory only.

A good old configuration file is the best. You can version it, distribute it, it's explicit, possibly structured, easily documented.

That doesn't mean that there is no room for environment variables, but these should be for local-only hacks and tweaks.

> Nothing is worse than trying to understand an issue with a program that heavily relies on environment variables for configuration

I totally agree with this.

> A good old configuration file is the best.

But the issue with this is that you often don't know what your configuration is ahead of time. Sometimes it is only generated just before execution of your code.

This is partially addressed by .env files (many projects support them) and an accompanying file with empty values or defaults, that lists and documents the variables (the .env file itself is then in .gitignore)

Every solution in the space is bad and it's a wonder that any python software is maintainable and I think the only real answer is everyone gets fucked all the time with these bugs and rough edges

I say this as someone who has done an unfortunately large amount of work in the python packaging and python package distribution space and still get bitten every few months

Plus pipenv falls down the stairs whenever I have tried to use it with packages that rely on native extensions like NumPy and GDAL. Not sure about pip-tools or poetry. My path of least resistance has been to use a local package manager (MacPorts, Homebrew before that), and then use Conda in CICD to test compatibility with different versions of Python.

> exactly like how `npm install` updates the package-lock.json and requires additional flags to use the lockfile instead

`npm install` doesn't do that anymore since at least a year