Problems with Python dependency management

Never had a problem making reproducible builds doing so.

To really pin everything you'd need to use something like asdf, on top of poetry or a manual virtualenv.

Otherwise you get your colleagues complaining that pip install failed with mysterious errors.

People end up committing either one or the other, not both, but:

- You need the source code, else your project is hard to update ("why did they pick these versions exactly?" - the answer is the source code).

- You need the compiled pinned versions in the lock file, else if dependencies are complicated or fast-moving or a project goes unmaintained, installing it becomes a huge mindless boring timesink (hello machine learning, all three counts).

Whenever I see people complaining about python dependencies, most of the time it seems just that somebody lacked this concept, or didn't know how to do it with python, or are put off by too many choices? That plus that ML projects are moving quickly and may have heavy "system" dependencies (CUDA).

    #!/usr/bin/env -S uv run
    # /// script
    # requires-python = ">=3.12"
    # dependencies = [
    #     "google-api-python-client",
    #     "google-auth-httplib2",
    #     "google-auth-oauthlib",
    #     "selenium",
    #     "webdriver_manager",
    #     "pydantic",
    # ]
    # ///
    import argparse
    import datetime

Edit: Huh, two spaces works too:

  #!/usr/bin/env -S uv run
  # /// script
  # requires-python = ">=3.12"

Yes, that format is specified by PEP 723 "Inline script metadata" (https://peps.python.org/pep-0723/). The cause was originally championed by Paul Moore, one of the core developers of Pip, who authored the competing PEP 722 (see also https://discuss.python.org/t/_/29905) and was advocating for the general idea for long before that.

It's also supported by Pipx, via the `run` subcommand. There's at least one pull request to put it directly into Pip, but Moore doesn't seem to think it belongs there.

  [project.scripts]
  mytool = "path.to.script:main"

  uvx --from mylib mytool

https://github.com/langroid/langroid-examples?tab=readme-ov-...

E.g. to chat with an LLM

  uvx --from langroid-examples chat --model ollama/qwen2.5-coder:32b

  uvx --from langroid-examples chatsearch --model groq/llama-3.3-70b-versatile

It was originally not a bazaar but "batteries included" where the thing you wanted to do had an obvious best way of doing it. An extremely difficult property to maintain over the decades.

pip, pipenv, poetry, conda, setuptools, hatch, micropipenv, PDM, pip-tools, egg, uv, ActiveState platform, homebrew, your operating system's package manager, and many others . . .

Relevant xkcd: https://xkcd.com/1987/

New workflow tools like Poetry, PDM, Hatch, uv etc. tend to do a lot of wheel reinvention, in large part because the foundational tools are flawed. In principle, you can do everything with single-purpose tools. The real essentials look like:

* Pip to install packages

* venv to create environments

* `build` as a build frontend to create your own distributions

* a build backend (generally specified by the package, and set up automatically by the frontend) to create sdists and wheels for distribution

* `twine` to put sdists and wheels on PyPI

The problems are:

* Determining what to install is hard and people want another tool to do that, and track/update/lock/garbage-collect dependencies

* Keeping track of what venvs you made, and which contains what, is apparently hard for some users; they want a tool to help make them, and use the right one when you run the code, and have an opinion about where to keep them

* Pip has a lot of idiosyncrasies; its scope is both too wide in some places and too narrow in others, it's clunky to use (the UI has never had any real design behind it, and the "bootstrap Pip into each venv" model causes more problems), and it's way too eager to build sdists that won't end up getting installed (which apparently is hard to fix because of the internal structure of the code)

* Setuptools, the default build backend, has a legacy of trying to be an entire workflow management tool, except targeting very old ideas of what that should entail; now it's an absurdly large pile of backwards-compatibility wrappers in order to keep supporting old-fashioned ways of doing things. And yet, it actually does very little in a modern project that uses it: directly calling `setup.py` is deprecated, and most of what you would pass to the `setup` call can be described in `pyproject.toml` instead; yet when you just properly use it as a build backend, it has to obtain a separate package (`wheel`) to actually build a wheel

* Project metadata is atrocious, proximately a standardization issue, but ultimately because legacy `setup.py`-exclusive approaches are still supported

The old ways of doing things have existed for much longer than the new ways, and become well established. Everyone just accepts the idea of copying Pip into every new virtual environment, even though it's a) totally unnecessary (even before the `--python` option was introduced two years ago, you could sort of get by with options like `--target` and `--python-version` and `--prefix` and `--platform` and `--abi`) and b) utterly insane (slow, wasteful, makes it more confusing when your PATH gets messed up, leads to misconceptions...). And that's before considering the things people try to do that aren't officially blessed use cases - like putting `sudo` and `--break-system-packages` on the same command line without a second thought, or putting code in `setup.py` that actually tries to copy the Python files to specific locations for the user, or trying to run Pip explicitly via its non-existent "API" by calling undocumented stuff instead of just specifying dependencies (including "extras" lists) properly. (The Pip documentation explicitly recommends invoking Pip via `subprocess` instead; but you're still probably doing something wrong if this isn't part of your own custom alternative to Poetry etc., and it won't help you if Pip isn't available in the environment - which it doesn't have to be, except to support that kind of insane use case).

Another part is that people just don't want to learn. Yes, you get a 'Giant full page warning saying “use a venv you idiot”'. Yes, the distro gets to customize that warning and tell you exactly what to do. Users will still give over a million hits to the corresponding Stack Overflow question (https://stackoverflow.com/questions/75608323), which will collect dozens of terrible answers, many of them suggesting complete circumvention of the package-management lock. It was over a year before anything significant was done about the top answer there (disclosure: I contributed quite a bit after that point; I have a personal policy of not writing new answers for Stack Overflow, but having a proper answer at the top of this question was far too important for me to ignore), which only happened because the matter was brought to the attention of the Python Discourse forum community (https://discuss.python.org/t/_/56900).

The proliferation of requirements.txt files is a massive reason for why Python dependency management sucks.

The author would be well served by using the first person and not including us in his uncertainty.

The lack of good namespacing practice is a problem. Part of the reason for it, in my estimation, is that developers have cargo-culted around a mistaken understanding of `__init__.py`.

One can emulate it with tools like poetry and uv but that incurs a performance penalty that every script has to go through `poetry run` and `uv run` which is often a few hundret ms and unsuitable for performant CLIs.

There were many problems with the proposal. The corresponding discussion (https://discuss.python.org/t/_/963) is worth looking through, despite the length.

Installers like Pip could help by offering to install `--in-new-environment` for the first install, and Brett Cannon (a core developer) has done some work on a universal (i.e. not just Windows) "launcher" (https://github.com/brettcannon/python-launcher) which can automatically detect and use the project's venv if you create it in the right place (i.e., what you'd have to do with __pypackages__ anyway).

Invoking SAT with clause count: 9661561

Invoking SAT with clause count: 5164645

You could try running

  python -m pip check

  python -m pip inspect 

Or, update stuff automatically:

  python -m pip install --upgrade  

  python -m pip install --no-deps

pip install --ignore-installed --dry-run --quiet --report

And, there's a lot more than that. Pip is pretty powerful and I'm surprised everyone dislikes it so much.

Hope this helps. Cheers!

Indeed, the novice shouldn't have to make all kinds of intricate choices about the build system to get something running. The language designers should have provided a good set of default choices here. The problem with python is that the default choices aren't actually the good ones.

No, the main problems with this are at the language level. Languages like Java can do it because the import is resolved statically. Python's `import` statement is incredibly dynamic, offering a variety of rarely-used hooks (start at https://docs.python.org/3/library/sys.html#sys.meta_path and follow the links, if you dare) that ultimately (by default) put imported modules into a single global dictionary, keyed by name. If you want to work around that, you can, but you'll definitely have to roll up your sleeves a bit.

Even if you just want to choose between two different versions at runtime, the base `import` syntax doesn't have a way to specify a version. If the package version isn't explicitly namespaced (which causes more pain for everyone who doesn't care about the exact version) then you need to make special arrangements so that the one you want will be found first (normally, by manipulating `sys.path`).

But if you want to use both during the same run, you'll encounter far more problems. The global dict strategy is very deliberate. Of course it offers a performance benefit (caching) but it's also required for correctness in many normal cases. It makes sure that different parts of the code importing the same module see the same module object, and therefore the same module-level globals.

That is: in the Python world, it's common that the design of C involves mutating its global state. If you "let B use its older version" then it would have to be an entirely separate object with its own state, and mutating that would result in changes not seen by A. Whether or not that's desirable would depend on your individual use case. A lot of the time, you would want the change to be shared. But then, even if you could propagate the changes automatically, you'd have to deal with the fact that the two versions of C don't necessarily represent their state identically.

> by letting B use its older version but re-package everything under a non-colliding name. Or perhaps I depend on library B and B uses C but I don't use the parts of B that interact with C. I could build a fat-jar for my project, and effectively drop the parts of B that I don't need.

In principle, the B author can make the dependency on C optional ("extra"), for those users who do need that part of B. Or even for B to optionally depend on B-prime, which encapsulates the C-using parts.

This is actually not very difficult (and the above-described dynamic nature of Python is very helpful here), but in practice it isn't done very much (perhaps Python developers are wary of creating another left-pad situation). But I do wish e.g. Numpy were more like that.

But the "fat-jar" analog is also possible in Python, as long as it isn't a problem that you aren't sharing state between C-old and C-new. It's called "vendoring"; Pip does a lot of it; and the tooling that Pip uses to enable it is also made available (https://pypi.org/project/vendoring/) by one of the main Pip devs (Pradyun Gedam).

Poetry was a wonderful breath of fresh air. uv is the same, but 100x faster.

Are you referring to something else?

Django users are using it natively, and cherryPy/etc. Python full-stack maybe, but they'll have DB drivers and JSON parsers probably from C.