How I think I want to drop modern Python packages into a single program
utcc.utoronto.ca
utcc.utoronto.ca
$ python3 -m venv --system-site-packages my_overlay_venv
$ ./my_overlay_venv/bin/python -m pip install whateverDatabases do it, Git does it, ZFS does it.
Ok the first two examples aren't filesystems but they can store files, so not totally irrelevant?
Relying on abstractions to simplify the architecture is a sign something might be right. Simply because simple things are less error-prone IMHO.
... oh, wait.
Java has been a safe place to deploy code to because of its xenophobia. You don't expect Java to be linked into endless C libraries, and Java often doesn't come pre-installed thanks to license issues. (Maybe Larry Ellison is rich because you read "INSTALL JAVA TO EXPERIENCE THE POWER OF JAVA". The consequence is that you got a clean copy of Java and that your clean copy of Java won't break all the crapplets that came with your OS that expected an ancient Java that came with the OS.
Python got a boon from being shipped w/ most Linux distros, but it was also a setback... On Linux "python" became a synonym for "python 2" because otherwise you'd break the system scripts. (Today it drives me nuts that Microsoft added "python" to the past on Windows because Windows was the one operating system where you didn't have to hide the system Python to get a good Python)
"venv" is genius but conda got so much mindshare that many people never realized "venv" could do everything "conda" does except for corrupting internal data structures that force a periodic reinstall.
The consequence is that "best practices" haven't been institutionalized in most places, you have to continuously fight bad practices, and people like the poster are always inventing non-solutions to the problem venv solves.
(e.g. for instance, what about the person who uses a Docker container to keep python environments apart as opposed to one who uses venv?)
Here's a specific example. In a previous job I had to work with Tensorflow models of various vintages that depended on different versions of CUDA.
Conda (out of the box) did nothing to help with that problem. I think an early version of Conda might have installed one early version of CUDA, but the elephant in the room as that I would need to install as many versions of CUDA as necessary.
NVIDIA uses a lot of intimidation to fool people as to the nature of the drivers. Specifically, CUDA/CuDNN and such don't depend on kernel drivers or Windows installers, but just require that some huge directories full of userspace libraries be on your path. Thus you can have CUDA version A linked into one conda environment, and CUDA version B linked into another conda environment and it all worked.
That is, it worked when I made my own conda packages (conda wasn't allowed to distribute such package by NVIDIA) and it was astonishingly slow because conda stores packages in BZIP2 files that take forever to unpack and because conda wastes more time with it's symlink machine.
In the end it was possible to solve the problem w/ conda but if I hadn't been distracted by conda I probably would have solved it just as quickly by unpacking the userspace drivers once and just setting the path appropriately.
Hold up, what? I have used both for years, how did you corrupt anything?
>The official oletools install instructions talk about using either pip or setup.py. As a general rule, we're very strongly against installing anything system-wide except through Ubuntu's own package management system, and the environment our Python program runs in doesn't really have a home directory to use pip's --user option, so the obvious and simple pip invocations are out.
No! The obvious and simple pip invocation is as follows:
python3 -m venv ${HOME}/myvenv
${HOME}/myvenv/bin/pip install foo bar baz oletools
${HOME}/myvenv/bin/pin install -e /path/to/my/installable/project
I would also recommend
export PIP_REQUIRE_VIRTUALENV=1
Of course never, ever `sudo pip ... `.
And don't touch $PYTHONPATH unless someone has forced you to use Caffe, who should be shamed on a regular basis for being Bad At Packages.
> [complex commands to set up a venv follows]
"All you have to do is... recompile your kernel... check your version dependencies... maybe do that once or twice. It's so simple, I don't know why everyone doesn't do it!"
Ubergeek.tv's classic "Switch to Linux" spoof ad of 2001:
python -m venv myvenv
myvenv/bin/pip install ...
Compared to "just pip" it's one whole extra line, so I don't think it's amazingly godawful at all.
For newbies, writing one line is impossible.
Which the author explicitly said does not exist for the use case he is discussing:
"the environment our Python program runs in doesn't really have a home directory"
You can certainly put a directory wherever you like and put the code there. In fact, that's exactly what the author did, as he describes in the article.
What you can't do is just magically have that directory be the "home" directory of a user and have the $HOME environment variable always point to it. You might be able to set things up to do that in principle, but it might not be worth the trouble as compared to other solutions. In fact, that's pretty much the position the author takes in the article for his use case.
$HOME or not is a red herring, i apologize if it appears more salient than it really was meant to be.
No, it's "I don't need the full power of venv so I'll just put the code in a specific directory now that I've figured out how to tell pip to do that when it's not a user's home directory". Using pip install's "--target" option hardly qualifies as a "terrible hack". The designers put it there because there were valid use cases for it. Setting environment variables before running a program is hardly a "terrible hack" either; it's one of the most common uses for shell scripts.
> $HOME or not is a red herring
If all you meant by it is "put the code in any directory you like", then, as I said, that's exactly what the author did.
Most Python developers have virtualenv creation done for them via their IDE or scripts that vastly simplify the process so that you don't need to bother with the details in the OP.
development != deployment.
The whole point of a container is to isolate dependencies.
The whole point of a virtualenv is also to isolate dependencies.
If using a container, why bother with a virtualenv at all?
> and the environment our Python program runs in doesn't really have a home directory to use pip's --user option
from the perspective of a developer, debian makes absolutely idiotic decisions about how python is packaged and managed by apt, so i opt out completely!
And --system-site-packages is a baaaaad idea.
Then I build python on target machine.
MAKE_OPTS="-j8" CONFIGURE_OPTS="--enable-shared --enable-optimizations --with-computed-gotos" CFLAGS="-march=native -O2 -pipe" pyenv install -f 3.8.5
Pipenv was a valiant effort that moved the field along heaps, but I haven't used it in my daily life for two years.
Won't/can't fix issues:
Which is a very different use case from being a sysadmin for a network used by a large university, which is the use case of the author of the subject post. So it's not surprising that the "obvious and simple" solution that works for you doesn't work for him.
Also, the post discusses the option of using a virtual environment and explains why that option was not taken. So it's not like your "obvious and simple" solution was not considered. It just didn't turn out to be the right solution for this author's use case. (For one thing, the author explicitly notes that there is no "home" directory for this use case in which a venv can be easily set up.)
I know that, and you know that, but I'm not sure the poster I was responding to knows that. In any case, for the use case of the author of the article, there is no "home directory" to use, so installing to $HOME was not an option.
> He says using virtual environments would be a lot more to learn but he just reinvented them.
He invented a sort of poor man's venv that just meets the particular needs of his use case. Sometimes that's a better solution than adopting a standard tool that requires more work and understanding to use.
This is a shame, because this is one thing that Python gets mostly right. You don't have to fuss around with the shell wrapper, etc. For 90% of cases you can probably do:
$ virtualenv subdir-name
$ subdir-name/bin/pip install <package-names>
$ subdir-name/bin/python your-python-script.pyIf he asks nicely on IRC, he might even get someone else to do it for him.
It can pretty much be summed up as "I don't know much of Python and admittedly don't want to spend time on it, so I did a bunch of hacks to download my dependencies and it worked".
> Nor do we want to learn about how to build and maintain Python virtual environments
I think the author would have better spent his time learning the basic usage of virtualenv/venv than writing this blog post.
Chris has a very low threshold for writing blog posts: if he does something he writes it up. He also has it set up in a way that's very low overhead for him.
I like that he writes these sorts of snapshot blog posts; they're a much better view into what people are actually doing than "I'm going to introduce this shiny new thing" posts.
I don't. Virtualenv is a hack to workaround the mess of Python paths that they won't actually fix. I'd put it in a similar bucket as Docker (hacking around the mess of binary software distribution on Linux).
I don't want to spend my time learning workarounds.
So instead of that, you'd rather... make you own custom hack workarounds? That will be idiosyncratic to your installations/projects? That will hit unique errors/tradeoffs the standard convention has already worked around? That will be harder for others to understand?
OK, to each their own!