I had to set up Python projects for some machine learning classes in college and it was a complete mess.
C has shared libraries and static compilation (bundling of dependencies) Java has .jar files which probably contain all your dependencies, go and Rust statically compile to something that has minimal dependencies and python has... no clue. Some packages you have to install via your package manager (in linux), because they have to be compiled against your local libraries, some other dependencies are somewhat included... and if you then try to also develop on the same machine you can choose between your system python libraries or installing them via pip (which makes your system weirder to deploy).
Having said that, I'm no expert in python but I have done some work in it and the deployment question is something that always seemed weird and unexplained to me.
- There's one tool to use, and it comes pre-installed
- You don't have to deal with virtualenvs or packages from different projects conflicting.
- Native code dependencies "just work".
That's only because you're basically assured there will be conflicts inside any single project. Every time I install anything via npm I get a boatload of warnings about insecure dependencies, which are effectively impossible to fix without breaking the whole mountain of hacks.
There are good deployment stories out there. JS is not one of them.
That's not an issue with the package management though, that's an issue with the quality of the packages themselves. The JS ecosystem does have issues there. But the tooling itself is good.
Though, as the sibling comments say, it's debatable whether it should.
There are tradeoffs for sure, like the ridiculous sizes of those directories on development machines. But a giant SSD is cheaper than my time resolving dependency issues on user machines.
No language should replicate anything JavaScript does when it comes to package management
On the other hand, deployment of Python applications is still unresolved, and there is no single standard method to ensure both compatibility with and isolation from the underlying system in a universal way. There are different methods, from virtual environments (both standard and nonstandard ones like conda), to version managers like pyenv and asdf, to "jailers" like pipx and pyinstaller, to containers; but none of them has yet clearly won.
The candidates for distribution are things like pyinstaller or pex (still not fully mature) but I don't think they are as easy to use as Java jars or Go binaries.
> All of the above are suitable for deployment on your cloud servers but not as distribution mechanisms to end users.
virtualenv's are also not a distribution mechanism because they aren't portable (unless that has changed). I can't "distribute" a virtualenv to a host. I can only use virtualenv's to change how pip (or poetry) installs artifacts. Pip is still your distribution mechanism, virtualenv is a runtime customization to change how your distributed artifacts are loaded.
The current situation is actually not that bad. With my students, the topic is quickly explained and mastered. But it's not a common occurrence.
However, it is true that it's still not as easy as it could be. Some reasons.
- Python is older than java.
- Python is shipped as part of MacOS and Linux, but using the system Python can cause issues, so it's a trap.
- Python deals with a lot of compiled extensions. Scipy embeds C, fortran and assembly. It's a hard problem.
- The Python project has *tremendously* less resources than java.
- Politics. Lots of it.
- Debian et Red Had packagers made a mess by splitting the python setup in many small packages.
- Python 2/3 transition took attention span from the problem during a decade.
- You often install many python versions on the same machine, which makes things harder.
- Python is used by a huge number of non programmers. Much more than java. This shows in the reported difficulties.
- There are way too many ways to install python. So many combinations of failure modes.
- Microsoft screw up with their store not packaging "py" or the core dev by not packaging it on Unix.
- Anaconda brings as many problems as it solves, and is popular because of enterprise limitations.
Python is glue language for C/C++/Fortran/Rust/etc. projects. Most of the valuable packages in Python in fact C++ projects (like Tensorflow, Pandas, etc.). When you are talking about Python dependency management you are talking about the Cartesian product of the packages you are depending on, their dependency in terms of compilers and header files. It is not hard to run into missing .h files while trying to install these dependencies. This situation was improved significantly a while back with the WHL files.
Anyways this whole article does not make too much sense, I am using Python for 10 years professionally and never run into the concept of nested venvs let alone somebody trying to use that in production.
They would be if python provided the .h files in the packages rather than expecting them to already exist on the system.
(Nb I’m a maintainer of a python project that’s difficult to build, and the number of issues we get that start: can’t install because foo.h can’t be found is very high, even though there’s other explanations closer to the end of the failed build)
I'd say maybe 25-50% of developer linux workstations have the appropriate packages and will compile out of the box, and if they don't, it's almost always a one line command to install them with a package manager. We've got instructions or docker images for testing for most major linux distros.
MacOS is probably in the 5-10% range for successfully compiling out of the box, some of the headers ship with MacOS, but there are a couple steps to go through. M1 is still a WIP in the python packaging community.
Windows... 0% compile out of the box. You have to set it up, download dependencies and so on. There are at any one time a handful of people who can compile for the windows platform, and a smaller handful of people who can debug problems.
Now, to be fair, I think this project is probably in the most complicated 1% of python projects from a packaging point of view. There are probably more complicated ones, and the only two complications I'm aware of that we don't have to deal with are 1) The chicken and egg of if we break an update we lose the ability to update all the other packages including our own (pip) and 2) GPU drivers.
One reason people like PEP 582 is that, by being something built into the interpreter, it will reduce a lot of the confusion over which isolation tool to use.
Part of the problem, which local installs won't completely solve, is that the python interpreter is still changing fast enough that modules are having trouble keeping up. E.g. tensorflow is still working on python3.9 support [1], and now we'll see python3.10 in the next couple weeks. Even with local installs we'll have to be careful to use the right interpreter.
Another piece of the problem, which is also kind of a good thing, is the huge size of the python module ecosystem. I've got python scripts with over a hundred requirements. It saves me from writing a lot of code, but resolving the requirements is a challenge. The pip package manager has recently added a dependency resolver, which has helped a little make sure everything can work together.
The only surefire ways I can think of to make this huge module ecosystem interoperate cleanly are (1) have every module support every other version of every other module and interpreter (insane amount of work, won't happen) or (2) have simpler interop interfaces like the unix shell convention of only passing strings between programs, but that would take away a lot of the ease-of-use of passing rich objects around between modules.
Did you use conda (anaconda/miniconda) or pip?
I ultimately fixed this with going to the conda files list, and choosing exactly what I needed. Not straightforward at all.
"Machine learning" here is crucial I suppose. Sklearn, keras are big beasts.
Would you install something like django/flask site than I don't think you would had any problems.
It's super easy, barely an inconvenience
Gradle is a hot mess and JavaScript is the just a complete shit show