Distributions vs. Releases: Why Python Packaging Is Hard
pydist.com
pydist.com
Some years ago, pip started distrusting HTTP mirrors, and while you could add some options to force it use HTTP, these options weren't present in previous pip versions. Which meant that you now had to provide options depending on the pip version -- which is harder if you don't call pip directly (for example through dh-virtaualenv).
We switched to HTTPS, but one with a TLS cert signed by a company-internal CA. Getting pip to trust another CA is non-obvious, and again depends on the pip version. So another version-dependent treatment necessary.
Then pip suddenly moved all of its implementation from pip.something to pip._internal.something, stating that it was documented all along that you shouldn't import anything from the pip.* namespace. But, the package names all didn't start with an underscore, so, by python convention, looked like perfectly fine public classes and methods.
Moreover there simply isn't any public API to do things you really don't want to reimplement yourself, like parsing `requirements.txt` files.
As soon as you want to do anything slightly non-standard with pip/distutils/setuptools, you find a solution pretty quickly, and that solution turns into a nightmare when you upgrade any of the components involved.
Also, finding a local pypi mirror that lets you inject your own packages and deals with wheels correctly and doesn't generally suck and is somewhat easy to use... not easy.
My solution at a small company, where we couldn't waste time with this crap, was to vendor everything: when a developer updated the requirements.txt, they would also install the packages in a project directory (using "pip install -r requirements.txt -t ./dependencies") and commit that. That way, the rest of the process (CI, packaging, etc) could just checkout the repo, set PYTHONPATH=./dependencies and ignore pip completely.
[1]:
https://nixos.org/~eelco/pubs/iscsd-scm11-final.pdf
[2]:
Or any other piece of C software with macros that depend on the build environment. Same .c file in, different binary out.
% ldd $(which nc) | grep ssl
libtls.so.18 => /nix/store/s6j0yd68cnfb4mv76lyrb413qhhac57g-libressl-2.8.3/lib/libtls.so.18 (0x00007f8875f6d000)
libssl.so.46 => /nix/store/s6j0yd68cnfb4mv76lyrb413qhhac57g-libressl-2.8.3/lib/libssl.so.46 (0x00007f8875f1a000)
libcrypto.so.44 => /nix/store/s6j0yd68cnfb4mv76lyrb413qhhac57g-libressl-2.8.3/lib/libcrypto.so.44 (0x00007f8875d3b000)
Consequently, a Python derivation that depends on readline would have a different hash than one that does not depend on readline.
What about client/server components where there's dbus in between, like the network manager or volume control?
These are solvable problems, but it's been enough of a challenge building a server OS from containers (RancherOS), and building a desktop OS that way is a significantly harder problem.
I don't really see the problem there, I see the plugins as essentially just data (i.e. stored in a mounted volume), for which updating and versioning is in the domain of the application itself or maybe some standardized library it uses.
desktop widgets: essentially the same thing, it's a plugin to the desktop environment and can be stored as a volume mount on the DE container.
dbus is probably something that would require an evolution on the container side, or alternatively it would need to be all abstracted into network interfaces. another possible way to look at it is to have a layer between kernel and containerized userland that is responsible for manipulating all the physical host things in the traditional way, and the examples you give are exactly that. maybe this sort of thing should continue to be distributed tightly together with the kernel.
It seems that things must have recently regressed; the page currently says 40% but it used to say 98%.
But to get there, you need sane build system, in order to explain these dependencies outside of python/java/whatever... And obviously the source code (although precompiled static libs should work in this scheme too)...
The goal again - just to have one thing, and most importantly no .so, .dll, .dylib lying around,
And zipimport pip package
Same thing for java, dart, etc. This makes it much easier to deploy, use, download, as it's basically "zero"-config install, no need for scripts to tweak your PATH, etc.
(but it has limitations clearly, if you use proprietary code, or you can't compile given open source library bundled with you - e.g. LGPL code, that you have to dynamically link, then again if you are project is open source this is no longer that much of a limit (but would not know that for sure))
I feel like, as a representative of the tool it’s easy for readers to default to understanding any criticism (even the slightest) of user behaviour & no discussion about tool behaviour as being defensive of the tools.
I know the Python package tools community value their users and would never dream of suggesting that they’re using the tools the wrong way. How important do you feel it is to recognise/appreciate those users in user-facing messaging?
The proper fix would be to make publishing a release a separate operation. But that breaks all existing tooling and workflows.
So, both points are incorrect.
- The package manager can't tell whether the package will be able to support your system or not, so if this is e.g. an optional dependency your install will probably break instead of the package being skipped.
- This results in an unexpected and hard-to-foresee dependency on some other non-npm server, which could disappear, be blocked by a firewall, etc.
- Standard tools will not understand what this package is doing and their assumptions may be broken, e.g. when trying to make builds repeatable.
Now answering your notes:
- It is not very different in python from practical point of view. I understand that while from theoretical point of view it should be different but it is not. E.g. when I was working on Windows I met situations where package is supported but there is no binary package. In such situations it falls back to compilation from source code and for some packages you need to perform "ancient Zulu dance" to get it compiled on Windows. Sometimes I was in the mood to do that sometimes I was not. However from user perspective even if package manager can tell that it is supported it does not help much. In practice, both python and node packages compile successfully quite frequently and in the end package is almost always supported.
Node supports optional dependencies and failure to install package either as prebuilt or built from source code will simply skip it and will not fail the whole install.
- In practice node packages usually use github and sometimes amazon s3 for prebuilts (at least based on quick analysis). There are still at least two systems but github and amazon s3 both seems to be good enough.
- I don't see problem with repeatable builds in node's case. Could you elaborate more here?
I see one problem with node's approach however. In case you want to have your own prebuilts (e.g. you have your own server) then you will end up with one big problem. You will need to have override each package with prebuilts separately.
Overall I think node did good thing by not trying to do everything and allowing community to figure out prebuilt solution. Problem is that there are multiple solutions. Python approach most probably will work out as better solution in the long run. In the end, most probably both solutions will be equally good and will look quite similar from user perspective.
This seems like reasonable suggestion.
If reproducibility matters, you can also use Microsoft R Open to get your packages from a frozen snapshot of CRAN : https://mran.microsoft.com/
Maybe 2 years ago there was still room for honest debate among Python package tooling. There just sincerely isn’t anymore.
[0]: https://conda.io/projects/conda-build/en/latest/resources/va...
That is way out of line.
I'm not going to try to talk you out of conda, but for the benefit of anyone reading your comment who doesn't know better I have to point out that your opinion is far from universal.
I’ve been working day in and day out with packaging and environments in Python for 12 years, and nothing has come close to being as serious of a general solution as conda. I count my blessings every day I get to use it, remembering the bad old days with Python’s native packaging, wheels & eggs, and then also the pipfile / pipenv mess.
How have those approaches failed?
- https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...
https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...
That is the author of that blog post, replying to a comment there.
This seems like the biggest downside I've seen mentioned thus far. Can you (or anyone) offer more detail as to how using Conda resulted in corrupting your Python installation? It it similar to what can happen with game mods, or is there something else going on?
Probably Conda is fine if you just commit to it, but I think the users who got into these messes weren't knowledgable enough to oversee the consequences of their actions.
So you could argue that the problem isn't with conda, but with the users. However, it seems that for naive users the consequences of using conda aren't clear (enough), which can get them into a mess.
I hate to hear that you had a bad experience. If it’s at all possible, please provide something on our issue tracker (github.com/conda/conda) we can replicate and write regression tests against. Help us improve on what we’ve missed.
Using the miniconda installer often helps keep it lightweight too, no need for Anaconda in many use cases.