Why not tell people to “simply” use pyenv, poetry or anaconda
bitecode.substack.com
bitecode.substack.com
It’s so thoroughly broken, every day a dev on some team gets their poetry env entangled with some system installed python, or numpy suddenly decides all the CI builds will now compile it from scratch on every build, or.. Today it was segfaults on poetry version X on the M1 Mac’s, that went away in version Y but of course version Y broke pandas for the windows devs..
Maybe you are an accidental accelerationist? Using the system Python is fastest way to a broken system and a broken Python.
I always use the system Python plus virtualenv and that works fine, it's certainly never broken my system...
I don't even know by what mechanism that breakage would happen.
If you just run "pip install" doesn't literally tell you to use the --user flag or virtualenv?
I guess it's probably an easy mistake to make.
Debian is particularly bad because they'd comingle python 2 and 3 packages in the same filesystem locations if you used pip out of the box.
That was a horrid discovery on my part. I assumed, coming from java where this sort of thing has been a long solved problem, that no one would be daft enough to do something like that.
Most projects state something like "run pip install whatever", which people will then do. If you're lucky, it will ask for sudo. If you're even luckier, you stop to think before entering your password.
That default Python env means I don't ever modify the system python for any reason. I type `python` and I get my own Python3 from the base.env
If you use homebrew, there are two settings that help keep your old python binaries around so that venvs that reference those old envs don't break.
HOMEBREW_NO_CLEANUP_FORMULAE, this takes a lists of packages to not cleanup
HOMEBREW_NO_INSTALL_CLEANUP, prevents old packages from getting removed
I actually far prefer the Python method to, say, Go's GOPATH. But, I far prefer Rust/Cargo to either.
I wish Python got a standard integrated solution for package management that works.
pip doesn't cut it in my book since it doesn't let you specify dependencies reproducibly. It either doesn't support lockfiles or ONLY supports having a lockfile, without dependencies.
venv also doesn't cut it since you have to remember to explicitly activate it and keep track of which venv is activated right now.
If we take a look at the much-maligned Node.js+npm, it's still far better than what built-in tools in Python let you have. Yes, Node.js doesn't provide full isolation from globally-installed node_modules, but at least it supports a local node_modules directory and lets it take preference. Notably Node.js+npm, with all its warts, doesn't suffer from the the two issues I've mentioned above.
I think this should be emphasized more, I don't think it's just a matter of preference, not separating packages between projects using virtualenvs will land you in a world of hurt as soon as you want to update or uninstall any of them and they're hopelessly entangled with system packages and other projects.
What is impressive, is how so many in the industry have forgotten about them.
Save in requirements.txt. Includes sub dependencies that are system specific.
Install from requirements.txt on different machine and get errors.
Uninstall dependency and save to requirements.txt.
Look at requirements.txt and see that sub decencies are still there.
What is the right way to avoid these issues on Python?
Keep your high-level dependencies in requirements.in.
Create a virtual environment using venv.
Run `pip-compile resolver=backtracking`. It autogenerates a requirements.txt file with all dependencies and sub-dependencies and their versions. This essentially acts as a lock file from other languages/frameworks.
Install from the autogenerated requirements.txt file in the venv virtual environment.
If a dependency changes, change in requirements.in, recompile the .txt file, and reinstall.
This is new for me. I used `pip freeze > requirements.txt`
* use a virtualenv, so that dependencies are installed per project and not globally
* only specify my top-level dependencies in a requirements.in file, and let pip-compile (a dependency, part of pip-tools) compile my requirements.txt
So far it has served me well and I've not encountered any error due to this method.
I cannot claim it is "the right way" though, just something that works for me.
Or junk all this and just use poetry, which manages both the abstract dependencies (pyproject.toml) and concrete ones (poetry.lock).
However: poetry still falls short in managing the python runtime, I am continuously having to divert time to help our data scientists untangle the messes poetry makes with virtualenvs and their local python setups.
Also, they broke backwards compat on the lock file format? So now devs running newer poetry versions break projects for devs on older versions because the lock files aren’t compatible?!
Once you set the version of python in the virtualenv and activate it, the right python will always be there. The virtualenv even has an instance of python in it's bin. When that python is run, whatever's installed in the virtualenv comes with it.
Poetry, however, does suck even though it's using virtualenv underneath. It's doing frankly too much and seems to have a lot of regressions. In the end it should only be looking at the contents of the virtualenv and using pip to install the correct packages into it.
Edited to add: With the last version of poetry causing issues in my development env, I'm about ready to ban it from our team.
Still, this means you can install multiple system Python versions and your venv will always contain an executable file of the version the venv was last installed/updated with.
╭─android@localhost ~/tmp/py
╰─$ python3 -m venv testvenv
╭─android@localhost ~/tmp/py
╰─$ file testvenv/bin/python3
testvenv/bin/python3: symbolic link to /data/data/com.termux/files/usr/bin/python3So to fix it you would need to change that, but then you would break a load of Python libraries which are most of the reason people use Python in the first place.
Deno doesn't really have that issue so badly because JS imports are ... not completely sane but they're quite a lot saner.
Anyway I think if you were going to break compatibility with Python by fixing the packaging and import system, it's not a much bigger step to fix the rest of Python too and then you have Nim or Lobster.
> What problem are you having with imports?
This is a Python function I wrote some years ago to import a module when given a path:
def module_from_path( name, path ):
### thx to https://stackoverflow.com/a/50395128/7568091 ###
### thx to https://stackoverflow.com/a/67692/7568091 ###
import importlib
import importlib.util
spec = importlib.util.spec_from_file_location( name, path )
module = importlib.util.module_from_spec( spec )
sys.modules[ spec.name ] = module
spec.loader.exec_module( module )
return importlib.import_module( name )
The dynamic module:
» cat sub/test_module.py
def hello(name):
print(f'Hello {name}')
» cat test_script.py
from importlib import import_module
full_module_path = 'sub.test_module' # no ".py"
module = import_module(full_module_path)
# optional, if you want to use across the project:
import sys
sys.modules[full_module_path] = module
print(sys.modules[full_module_path])
# end optional
# try it:
module.hello('there!')
» python3 test_script.py
<module 'sub.test_module' from '/home/foo/sub/test_module.py'>
Hello there!
https://docs.python.org/3/library/importlib.html#importlib.i...
In general, I don't see the intrinsic complexity being reduced much more than this:
from importlib import import_module
module = import_module('sub.test_module')> In general, I don't see the intrinsic complexity being reduced much more than this
you know in NodeJS I do `const tm = require('sub/test_module');` which I for a number of reasons do find inherently less complex.
I used to work at a company where they had a custom import system and it was a nightmare.
from foo.bar import Baz
And because of how Django's settings work (import django.settings and all your stuff is magically there), people in my company have gotten into the habit of putting code in settings/__init__.py to do the things they need, like retrieve secrets from AWS.
Which I don't want to have happen when I'm trying to import settings.bla_module to use a type in a unit test.
Yes, and it needs to—to find the variables there, which may/not be other packages.
> how Django's settings work
Right. Django is great and popular, and so I think many folks don't realize how downright clunky it is in a number of places. Many parts of it are over-engineered like this one. It has improved over time but some things are set in stone due to history. I'd like their settings redesigned myself.
> __init__.py to do the things they need, like retrieve secrets from AWS
No, no, no, no... NO! ;-)
Imports should only create variables, define functions, and/or alias a package. They should never, ever run lengthy computations or access the network. That's basic software design 101, unfortunately not taught everywhere.
I don't see this one as a clear flaw of Python. "Dumb" coding can happen anywhere. I suppose it could prohibit this kind of stuff, but that might prohibit other cool things that might be reasonable in limited situations. It is a "consenting adults" design, with those trade offs.
https://stackoverflow.com/questions/14132789/relative-import...
And you need (maybe used to?) an __init__.py file in each package.
But once you know that, it's straightforward, no? Certainly not, "only a few people in the world know this..." level. Don't think I'm some kinda level-10 genius here.
Yes because referring to the filesystem is logical and intuitive and referring to some abstract "package" that changes depending on how you invoke Python is insane.
> But once you know that, it's straightforward, no?
I wish!
Close to none of the notation of Python works the same as a shell. Even quotes, which are superficially similar but have significant differences.
> abstract "package"
Packages are not abstract, they are simply a folder with .py files inside. Sounds concrete enough to me.
> changes depending on how you invoke Python
I'm not sure what this refers to? I know you can change PYTHONPATH. And the current folder portion changes with it.
Fast forward 30 years and we really don't use most systems like that anymore, especially in production environments. We run things in VMs or containers and build them up from scratch with ease--it's all just cattle. Python hasn't really adapted to that new reality.
You might think this is silly but think about something like making a systemd service to run your python script in a venv--how do you do it? You have to activate the script or call its python bin, but it's not obvious how to do that in any python docs.
The Python docs do cover this topic, and do so rather well: https://packaging.python.org/en/latest/guides/installing-usi...
How do I ensure that at each customer site, my library has the set of dependencies it needs, with versions that it is known to work with? Each installation method has a different constraint solver, a different means of specifying dependencies.
If your question is "how can I ensure technical end-users have the same set of python packages as I do for the code they run using standard pip+venv?", the answer is to pin dependencies in a requirements.txt file.
If your question is "how do I stop end users from installing dependencies for my software cowboy-style?", the answer is to write installation and usage instructions, and/or include an AIO run script.
If your question is "how do I package my library so that end users' package managers know my library's downstream dependencies when they install it?", you build a wheel using `pip wheel`, which again relies on a requirements.txt. If I'm understanding you right, you're mistaken that you have to handle the package managers separately; they all use pip + wheels under the hood. Conda is a bit of an asterisk in that you can package things differently if you desire, but it plays nice with pip + wheel builds too.
https://docs.conda.io/projects/conda-build/en/latest/user-gu...
It doesn't matter which language you're using, any non-trivial code you have is going to have dependencies which you will need to pull in.
If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!" problems that used to plague people.
You disagree that "you shouldn't need a clean OS every time you want to use a new language version"!? ... I don't even know what to say, that's not something you can disagree with
> If you try to build it in a completely clean container, you can ensure that you have caught every dependency, and you will eliminate all of the "but it works on my machine!"
This is what testing is for. You don't need to cripple your development environment to have confidence your code works in other environments. Talk about the ends not justifying the means.
I built this resistance to docker about 5 years ago, is it (or was it) still correct? Is there a best practice for sharing GPUs with containers that's not a pain in the ass?
Edit: I should specify that I typically develop on Ubuntu, but occasionally will do so on Windows (without WSL).
On the one hand, it bothers me that software is so brittle that you need a container around it so that everything is just right. On the other hand, you can argue that containerisation is a clever way to make it all more robust.
If you put everything into an Uber Container, the same problem would surface.
Containers exist to solve the fragile dependency/dynamic linking problem.
Python doesn't need "something like deno". All programming languages need package management.
> And it works well on Windows. No idea about Mac and don't care.
Package management is a cross platform problem. You may not need it but that doesn’t make the current solution good.
This is where I kinda regret switching from and miss chef. Pick up an RPM/APT package from vendor, shove it into your package manager, the environment works. It developed a sufficient amount of weird behaviors later on, but that's besides the point.
Instead we have a really precise documentation of 4-5 steps to get everything installed in a way that should overall work, and each step is annotated with 4-5 ways that look deceptively good but end up being even more quirky. And when you do all of that, and review it twice, it still doesn't even work consistently across 6 workstations. 3 work the same, 2 have interpreter discovery anomalies, 1 has import conflicts with seemingly unrelated other installed stuff.
Whenever I have a working python VM / container build or installation, I kinda feel a need to shut up about my jenga tower because otherwise the universe will find a way specifically to ruin my day.
I hate how I have to feel like this about a language I really like.
And then
Python3 -m venv envdir
source envdir/bin/activate
bin/python3 -m pip install --upgrade pip
And similarly the rest of the packages needed for the project
This does not mess up the system library or other python projects
This step is the least intuitive until you've done it enough times that it's muscle memory.
Can it be better--yes. Is it worse than _any_ other language's packaging if it were to scale to the number/kind of cases Python supports/used--no.
"thoroughly broken" is useless. Are there bugs--most definitely yes. Do report them, or even help fixing them if you care.
There are many many different use-cases where different approaches are preferable (no size fits all).
Personally, most most of the time packaging is a non-issue e.g., pinning versions and cache are highly recommended for CI to avoid unnecessary surprises, flakiness.
- the guy who created C++
I always bring up Go in such arguments - according to the survey [0] done by the Go team, 93% of Go users are satisfied with the language.
In Python land one of those seems to come along every couple of years. They start over and implement a solution for some subset of the problem space. Then they realize the Python packaging mess is much, much bigger than anticipated and progress stalls. At that point there are 15 partial "standard" packaging solutions for Python, where there were 14 before. None of them are feature complete, and they don't really coexist well or at all.
Cue the obligatory https://xkcd.com/927/ and https://xkcd.com/1987/
From that presepective , most python libs are written in python only.
If you get any python job, you'd get a lot of side-eyes for not knowing of/how to use and initialize basic venv package sets. Even if the shop uses conda.
People were not impressed.
Hence the previous article (https://bitecode.substack.com/p/relieving-your-python-packag...) advocating to go back to the basics yourself.
See the reddit thread to witness exactly this in action:
https://www.reddit.com/r/Python/comments/124hktv/comment/je5...
Now we have a never ending train of hotshot developers that think they can "solve" Python packaging with another tool, without realizing that you can't design a tool to fix a broken 15-year-old system. It's not just a "there are now X competing standards" issue, it's also the fact that anyone who makes an earnest attempt is doomed. They will never be able to get it past the point of universally good enough.
I lament the fact that I'm dragged back into the Python ecosystem every time I need to do something with a PyTorch-based tool someone spent years of effort building. I would never recommend the language to a newcomer. Ruby pollutes the global namespace, and Gemfiles execute arbitrary code, but at least with Ruby there's Gem, and then there's Bundler, and nobody complains.
Will we forever be beholden to the mistaken decisions of the original developers in the early 90's, just because Python is simply too widespread? Because even people like me who detest the packaging experience are forced to go back there because nobody's going to rewrite the hottest ML project in Elixir or something? Even the simple mistakes like putting venv files in Scripts/ instead of bin/ forcing one to work around it in cross-platform CI make no sense in hindsight. Nobody remembers why it was made that way after so long. Now it just floats over you like a spectre.
Python's environment management is just the tip of the ice berg
Dynamic typing
Gil
Speed is slow at best
Async streams having its set of non tractable issues
Debugging by printing pdb all over and breaking
I spent about three hours trying to figure out how to setup python keyrings to work, to let me just get started using poetry. On a system I was ssh'ed I to. Gnome-keyring-daemom was up. I spent a while adding random pam rules suggested by archwiki in to inject more gnome-daemon stuff in my envs. Random gnome-keyring-unlock scripts, which quickly start talking about Clevis and tpm and fido 2-factor. Wading through hundreds of responses about Seahorse, a gui tool unsuitable for ssh. Many long miserable sad stories.
In the end I stumbled upon someone who suggested just nulling out & turning off keyring with some config to make it have a null provider. After this the poetry project just worked.
The tiny handful of deps this project has were already installed on my system, but poetry was also a task runner, instrumental for the usage of this single-file script.
There's been so many years of churn in the python world of tools. A fractal nesting doll of virtual-env, mkvirtualenv, & various offshoots. I hope some day there is a mature reasonable option folks generally find agreeable. Poetry eventually worked for me, but what a miserable gauntlet I had to walk, and the cries of so many who'd walked the path & utterly failed echoed out at me at every step.
For example, you can install oscclip to a temp dir with a dedicated venv and run it immediately:
$ pipx run --spec 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help
$ pipx install 'oscclip @ git+https://github.com/rumpelsepp/oscclip'
$ pipz runpkg 'oscclip @ git+https://github.com/rumpelsepp/oscclip' osc-copy --help
$ pipz install 'oscclip @ git+https://github.com/rumpelsepp/oscclip'You don't need poetry for that. My single CLI packages still use a setup.py that I haven't touched in five years and was simple enough to write.
Nothing in computing is "simply".
"Wellllll, that's non-trivial."
It’s not confusing if you short-circuit the conversation; if you need advice, don’t go down the rabbit hole, just use poetry and pyenv on MacOS, you will make your life easier than the alternatives. There are plenty of docs that will give the hand-holding that you are looking for, so nice you have made the decision.
If you know what you are doing there are arguments for other options, and data scientists have different tool chains. But I think we make it harder for new engineers by having five different ways of doing it, each with people arguing that actually their way is better.
And I remember custom build scripts with Cargo being a major pain at the time I used it, but the Cargo developers were able to sidestep major fundamental problems because they knew what didn't work with packaging in the decades before and managed to think about the design carefully enough from the start.
Rust is different because the choice of package manager is "simply" Cargo, not necessarily that Cargo itself is trivial to use.
which is the industry for which users are really the product. user's data in, profits out.
I would recommend to 'python -m venv' and thats all.
The only limitation I've encountered is when moving the environment or renaming one of the parent directories. In which case it's easy to create a new one:
# optional: freeze the environment if you don't already have a requirements.txt
source .venv/bin/activate
pip freeze > requirements.txt
deactivate
# remove the old environment
rm -rf .venv
# create a new one
python3.10 -m venv .venv
# activate it
source .venv/bin/activate
# install the requirements
pip install -r requirements.txt # install the requirements
npm install
# remove it
rm -rf node_modules
For those already familiar with venvs, the above just condenses to “remove the old venv and create a new one”.
Aside from needing to know to use it. Which is certainly a problem. But python blessing a single venv-system might be worse in the long run...?
The whole reason I posted this one is to justify the "just use -m pip and -m venv" stance I wrote in the article 4 days ago.
Because you will always have a very vocal minority of people that will fill the threads with counter arguments actually increasing complexity and risks of failure.
As the article says, I think these days it's pretty safe to just use Python's built-in venv and stay away from everything else.
sudo apt install python3.X
sudo apt install python3.X-venv
python3.X -m venv .venv"There should be one-- and preferably only one --obvious way to do it.", unless it is how to setup your environment.
I only use Python every few months, and it is always a struggle.
In comparison, "cargo build" works 98% of the time just after "git checkout"
conda solves it by packaging EVERYTHING, giving you atrocious 30GB environments, pip doesn't solve it at all and none of the challengers really have much to offer (in my opinion).
Personally I like Go. If someone wants to build my stuff, then I just say go here http://go.dev/dl and download Go, then set location to where the code is, and enter "go build". thats it. All languages should be that easy.
Go doesn't make a habit of interfacing with lots of legacy C,C++ and Fortran Code. There's probably some of that. But nothing like Numpy, Scipy, Tensorflow...
I'm not sure what you mean about Rust not extending into the same depth, could you elaborate?
And if not then that's not a packaging issue but rather a problem about portability because yes, some packages are system specific, and Windows is quite specific. One way around even those issues is to use either WSL2 or Docker desktop, maybe using VSC development containers or WSL remote.
There's always a fringe of packages in every package manager that deosn't have the greates compatibility among different dependencies and systems. Python's ecosystem is no exception to that.
What is going on in the Python ecosystem is people expect Python to interface to C/C++ modules on Windows. Reliably. And no other ecosystem even really tries.
As an example, Rust interfacing to Skia (C++) used to cause hideous build errors on various Windows installs. I needed to install and upgrade significant chunks of Visual Studio (not VSCode--actual Visual Studio) before the Rust system could compile it correctly. And the next Windows update it all broke again.
Also, Rust seems to indicate that being general purpose and interfacing with lots of legacy C (and some C++) aren't the issues.
Dynamic libraries can definitely be a problem, across languages. I've seen plenty of crates that rely on platform-specific .so/.dylib, so I'm not really nervous about that being supported, but it's absolutely possible that there may be Windows-specific issues that I haven't heard about.
edit Complete rewrite, let's hope nobody has responded yet :)
Just about the only question I've fielded every so often in the past several months have been people who found their way to an old tutorial based on the old system and them being confused about what happens, and a quick link to the current docs seems to resolve all their problems.
These are the languages and package managers I have experience with:
- JavaScript and npm
- Python and pip/conda
- Kotlin and Gradle
- Scala and mill
- Common Lisp and quicklisp
- Racket and raco
- Raku and zef
- Lua and luarocks
- Elixir and mix
- Haxe and haxelib
- Rust and cargo
- Nim and nimble
- OCaml and opam
- Smalltalk and Monti/Meta-cello; also Visual Works parcel manager
- and a few others
Out of all these, cargo, opam, mix, and zef are the only ones that are mostly working as intended. They're still a pain, but they're predictable and honest about their limitations. Everything else is just bad, to the point where it's a waste of time to think about which is the biggest and which are a bit less of a clusterfuck. For some of these I had to resort to building a chroot env with a copy of my system inside to convince them to compile and link against correct libraries. If that's not a clusterfuck, I don't know what is...
Unless you were lucky to see something better than "mainstream state of the art" solutions, you won't realize how bad it actually is. At that point you start inventing yarns or poetries, adding layers on top of layers to paper over simplistic, hardcoded assumptions made by original authors 20-30 years ago, and go into denial when someone tells you that's not going to fix anything.
Somehow we ended up in a state where step one of doing anything new with python is to fire up an empty docker container. And I'm awfully tired of folks in the "python community" blaming the victims of the mess they made.
This approach seems insane but correct.
It will be less hard than using docker.
My recent example: last week, I was installing a new program that uses some python scripts during its operation. I could not get those scripts to run. I knew it was because of the usual python problem of the scripts not matching the version of python that was executing them, but figuring out how to fix that took me a full day.
Just to make Python work. Not even as a dev.
Wearing my user hat, this is a clusterfuck. There is no other language that I am exposed to that presents this sort of problem, and this is a very common issue with Python.
This is why I start to get sweaty any time that I'm using software that involves Python. It turns into a crapshoot and half the time, it's going to cost me a lot of time and stress.
> perpetuates the false image.
You can believe it's a false image if you like, but there are a whole lot of end user experiences that indicate it's very real.
It will help with exactly that.
Python is the only language I've encountered that causes this sort of trouble for ordinary end-users. As a dev, I find this frustrating because the python scripts in question are clearly only "glue" in the first place. That's a lot of burden to put on the end user for just glue.
As for my issue, I did make it work in the end. I'm 90% sure I didn't do it in the right way and it will cause me problems at some point in the future, but that will be a problem for future me.
Ideally it should be fixed at the installer level.
I will write about this at the end of the week.
- easy_install, distutils and eggs were deprecated
- ensurepip and venv were introduced
- the whole wheel ecosystem flourished
- a new dependency resolver was introduced to python
But:
- a lot of noise has accumulated. Hence the procedure in the previous article, to avoid the traps from all that noise and go back to what usually works.
- we have a long way to go, but it's not something easily fixed (hence the conclusion of the current article explaining why it's going to take a long time to get better).
Hopefully, the direction of that long way is relegate python (and javascript!) to legacy languages.
Something, hopefully, will come out of webassembly. Otherwise, the last decade only saw the rise of two fundamentally flawed languages as the most popular ones.
Ruby losing to both was not a good thing. And I'm not even a Ruby programmer! I hated the language back in the Rails vs Java days. But it is impossible to deny that great software came out of Ruby.
Javascript and Python has produced no great software, just ponderous stacks of crappy code and poorly organized libraries. They are popular because they are used for throwaway web applications and throwaway scientific / non-programmer code, and it shows.
The fact that golang is what is producing new "good" software: a language that is intentionally bad, as opposed to python and javascript being worse because of cruft accumulation or Eich writing it a weekend, is also not a great sign but it's not the step backward that javascript and python are.
People writing in JS are usually techies who could/would migrate. People writing in Python are on a wide spectrum that goes from "one-step-above-matlab" to "know-it-all-c-wizard". You won't actually get all these people to move on any reasonable timeframe.
HN is biased because we tend to love tech and a lot of us know several languages, but Python is much more democratized. My friends in physics, biotech, mechanical engineering, basically everyone in STEM is using Python and to them it's a tool that took a lot of time to learn and they won't drop it all because dependencies are hard.
Besides, code at a large scale in any language is always messy. There are many ways to manage a growing codebase but if your only approach is relying a "good" language with static typing, verbose/explicit syntax, and a better built-in package manager, you are doomed to fail anyways.
Perhaps this is a case of correlation=/=causation? Python is more accessible so less adept programmers are more likely to use it.
When there's a new python version I'm interested in, I install it via Homebrew and update my zshrc to clobber everything else via $PATH. All my scripts and tools are broken? Just reinstall the packages globally. Whatever.
Since the big 3.x transition, it's pretty rare for forwards-compatibility to break (IME), and if something does, I can just try running prior python3x binaries until I find the last version that worked.
It's hideous, but honestly the least stressful way I've found to date.
If you "simply" (yes yes I know) download the python version you want directly and compile/install in a local folder, and use that with venv as a way to manage individual project dependencies, all problems go away.
Python packaging is fine for small local dev; problems arise in distribution, rollouts of upgrades and ensuring your apps work in the zoo of local setups your users have
On the most tightly managed lab machines, which are all in lockstep on a fixed configuration (latest Ubuntu LTS with a updated image pushed annually), we can provide a consistent Python setup (e.g. Python 3.10 and a fixed set of C-based modules like psycopg2). However, our staff and PhD desktops and laptops are more diverse - with the OS often only being upgraded when the distro is going out of support, they could be running n, n-1 or n-2. That, most likely, means three different Python versions.
We could use pyenv to let people install their own preferred version. Installing with pyenv requires building from source (slow on some of our oldest machines). This also means installing the Python build deps, which is fine for our departmental machines but not possible on the HPC cluster (operated by a different business unit) or the Physics shared servers. It's also less than ideal for our servers where students deploy their projects (where we want to minimise the packages installed, like the build-essentials meta package).
It's also a massive stumbling block for less experienced students with their own laptops which could be running any distro, of any age. Many CS101 or engineering/business/humanities students taking a programming class, who have never programmed before, would really struggle.
So, classes might tend towards teaching lowest common denominator Python (i.e. the oldest conceivable version a student might have installed on their machine).
Sure, we have in-person and remote lab machines students can use - but it's not always convenient (especially for the data science / ML students running Jupyter notebooks with their own GPU).
There are workarounds, but they all have serious downsides.
Compared with Node.js and Go, where users can just download the appropriate package and unzip/untar the runtime or compiler version of their choice, deploying the Python runtime has enormous friction (especially for less experienced users). This has the bonus of simplifying deployments elsewhere in our infrastructure (CI/CD, containers, etc).
And while we all complain about node_modules, Python venvs not being trivially relocatable is another huge frustration.
We've used Anaconda, but that comes with its own issues (most recently, finding that it ships with its own gio/gvfs binaries and libraries which fail to mount our CIFS DFS shares - causing confusion for users running `gio mount` from within a conda environment).
Disclaimer: I was using Fedora which has Python 3.11, using Fish which is clearly non-standard and I’m a sysadmin not a Python dev.
`${my-venv-path}/bin/python`
`${my-venv-path}/bin/${whatever-binary-you-need}`
`%my-venv-path%\Scripts\%whatever-binary-you-need%` (because Windows...)
We also haven't figured out how to incentivize maintaining backward-compatibility. 99.9% of the time when some library updates, and stops working with language version X, it's using some hot new feature of the language. Usually just because the library author thought it would be cool or more elegant. The entire software world needs to update now, because someone left-padded us for elegance.
How to transition a bazillion packages though I do not know.
It saddens me to say, but imports are indeed a complicated topic on Python. The PYTHONPATH is not something people know about, and once they do, it's not intuitive.
The weird handling of namespaces or relative paths doesn't help.
I'm just sticking all dev projects into a separate LXC and calling it a day. Don't want to deal with all the various separation models the various languages and package managers cooked up
https://gist.github.com/tom-pollak/8326cb9b9989e0326f0d2e19f...
>> I think you missed the point.
Maybe I did, but I've been using Docker as version management for pretty much every technology I employ for five or six years. Prior to that I sparsely used things like rbenv and virtualenv and I actually thought it was super dangerous and unreliable. Maybe it's gotten better in recent years, and certainly people who write python and ruby every day are going to know more about this than I do.
I don't install anything on my computer if I can just use Docker for it. OK, I do have go:latest, but I use docker images for various projects that might be on any version of go from 1.8 to 1.20. Your website still runs on PHP5.3? I can help you (I won't, but I could totally run it locally!).
Reasons I like docker better:
1. Any scripts or configs can explicitly refer to the version number. No guessing or assuming.
2. Our whole team uses the same version.
3. Only one dependency: docker.
Granted I'm more of a sysadmin than a developer and I'm sure that biases apply.
If you have a Docker image that builds with "pip install whatever" and this library gets updated with a breaking change, you won't be able to rebuild the image without changing the dependency or the code itself, for example.
I'm late to the party but AMA :)
Also, [even if it did], it installs binaries and doesn't compile.
You have already lost.
So if I had to chose one way for the next decade, I would bet on this.
But I will not pretend we can be sure about anything on this matter.
In fact, my hope is that we can improve the process tremendously by creating an unified way of bootstrapping Python for all OS. If this ever happens, the procedure would become mostly obsolete.
At the end of the week I will write about this, and the various experiments the community have done so far in that regard.
And it's often less about the package manager and more about the ecosystem: Can you find what you need? Is what you need stable enough? Does it break every few months with new versions of the runtime, either because of actual incompatibility (npm/node) or versioning shenanigans (nuget)?
But these are strictly package level issues, not package management.
The downside of node is the over reliance on flaky dependencies. The actual package management is good, especially compared to python.
It's not just "flakiness" of the package maintainers. The basic problem is that there is no single standard javascript package, and no package manager, especially npm can solve this.
That's not "being toxic", you just can't be everything to all people.
As tedious as javascript package management can be, python has consistently given me more trouble.
Saying anaconda is worthless because its maintainers don't expend a ton of effort on making your particular niche more convenient is the actual toxicity...
Saying anaconda is worthless because its maintainers
don't expend a ton of effort on making your particular
niche more convenient is the actual toxicity...
It's a ridiculous non-solution given that even in the data science context most stuff is POSIX compatible. If I can get R and Python on their own to work without hassle, there's no reason a mere package manager should throw up unnecessary walls.