Comment by hprotagonist | Hacker News Reader

Comment by hprotagonist | Hacker News Reader

I do full time scientific python, and this post has some painfully wrong ideas.

>The official oletools install instructions talk about using either pip or setup.py. As a general rule, we're very strongly against installing anything system-wide except through Ubuntu's own package management system, and the environment our Python program runs in doesn't really have a home directory to use pip's --user option, so the obvious and simple pip invocations are out.

No! The obvious and simple pip invocation is as follows:

  python3 -m venv ${HOME}/myvenv
  ${HOME}/myvenv/bin/pip install foo bar baz oletools
  ${HOME}/myvenv/bin/pin install -e /path/to/my/installable/project

and then edit your code in `/path/to/my/installable/project/project-name/*.py` to your heart's content and tell your IDE or editor that `${HOME}/myvenv/bin/python` is the python to use. Activation is strictly optional, but if you like that, `source ${HOME}/bin/activate` (or use direnv to do it for you).

I would also recommend

  export PIP_REQUIRE_VIRTUALENV=1

I divorce what $USER thinks `python` means from the system python as soon as I can with `pyenv global 3.8.5`: https://github.com/pyenv/pyenv

Of course never, ever `sudo pip ... `.

And don't touch $PYTHONPATH unless someone has forced you to use Caffe, who should be shamed on a regular basis for being Bad At Packages.

> No! The obvious and simple pip invocation is as follows:

> [complex commands to set up a venv follows]

"All you have to do is... recompile your kernel... check your version dependencies... maybe do that once or twice. It's so simple, I don't know why everyone doesn't do it!"

Ubergeek.tv's classic "Switch to Linux" spoof ad of 2001:

https://www.youtube.com/watch?v=Xtah_05BOe8

I generalized a bit for completeness and to emphasize that contrary to the article's point, you can do all of this in $HOME and never install anything outside your home directory, which was a concern of the author as a usage policy for their server.

  python -m venv myvenv
  myvenv/bin/pip install ...

works just as well, i just needed a concrete path to talk about in the rest of the text.

Compared to "just pip" it's one whole extra line, so I don't think it's amazingly godawful at all.

After we come up with that “one line”, we tend to forget how long it took us to get there.

For newbies, writing one line is impossible.

> you can do all of this in $HOME

Which the author explicitly said does not exist for the use case he is discussing:

"the environment our Python program runs in doesn't really have a home directory"

so put it wherever! it's just a directory, put it where you have write access.

> so put it wherever! it's just a directory

You can certainly put a directory wherever you like and put the code there. In fact, that's exactly what the author did, as he describes in the article.

What you can't do is just magically have that directory be the "home" directory of a user and have the $HOME environment variable always point to it. You might be able to set things up to do that in principle, but it might not be worth the trouble as compared to other solutions. In fact, that's pretty much the position the author takes in the article for his use case.

the position the author takes is "i've never heard of `venv` so i'll do these other terrible 5 hacks instead".

$HOME or not is a red herring, i apologize if it appears more salient than it really was meant to be.

> the position the author takes is "i've never heard of `venv` so i'll do these other terrible 5 hacks instead".

No, it's "I don't need the full power of venv so I'll just put the code in a specific directory now that I've figured out how to tell pip to do that when it's not a user's home directory". Using pip install's "--target" option hardly qualifies as a "terrible hack". The designers put it there because there were valid use cases for it. Setting environment variables before running a program is hardly a "terrible hack" either; it's one of the most common uses for shell scripts.

> $HOME or not is a red herring

If all you meant by it is "put the code in any directory you like", then, as I said, that's exactly what the author did.

Using virtualenvs is standard practice when developing with Python. What the OP did will work with every standard Python installation out there.

Most Python developers have virtualenv creation done for them via their IDE or scripts that vastly simplify the process so that you don't need to bother with the details in the OP.

> .. standard practice .. when developing with Python

development != deployment.

They're standard for deployment, too.

You really don’t want to do that... they can break silently when the system updates Python. Use safe system wide installation for deployment, such as docker or traditional system packages. Alternatively, you can look at one of the many bunglers/static linkers like PyInstaller.

Ideally you'd deploy to a virtualenv in a container, recreating the virtualenv with each new image. You wouldn't update the system Python at all.

> Ideally you'd deploy to a virtualenv in a container

The whole point of a container is to isolate dependencies.

The whole point of a virtualenv is also to isolate dependencies.

If using a container, why bother with a virtualenv at all?

That's what cool kids use today; just dump complete dependency tree of every project into random location on filesystem. That's not how Debian OS works though, and that's what author, a sysadmin, talks about very clearly. So I don't get why you propose exactly what he dismissed in the 2nd paragraph.

> and the environment our Python program runs in doesn't really have a home directory to use pip's --user option

I do all of this on debian, literally every day.

from the perspective of a developer, debian makes absolutely idiotic decisions about how python is packaged and managed by apt, so i opt out completely!

Really? I do `apt-get install foo` and later I in a Python script I write `import foo`. This Python script can be put ANYWHERE on the system. I don't need to mess up with virtualenvs and I have one version across all scripts on the system that's regularly updated on system upgrades (after being tested by community in bleeding edge versions of the OS). To each his own I guess.

apt doesn't have my stack or its dependencies at proper versions, and cannot in general meet the version-locking of dependencies i and my dependencies require.

And --system-site-packages is a baaaaad idea.

The first thing I do when deploying to an existing machine/hard metal: I Grab the pyenv installer, set it up.

Then I build python on target machine.

   MAKE_OPTS="-j8" CONFIGURE_OPTS="--enable-shared --enable-optimizations --with-computed-gotos" CFLAGS="-march=native -O2 -pipe" pyenv install -f 3.8.5

This takes a while but end result is a well enough optimized python. I also replace standard asyncio loop with uvloop to speed things up a little for async tasks.

I don't usually go beyond --enable-shared, but yeah, me too.

Would love to hear your thoughts on pipenv as a real-world user of python environments. I leave it on the table that your current solution just works and you have never considered pipenv. ( I am not associated with pipenv. I've seemingly organically gravitated to it lately. )

my hot take: poetry won, and PEP517/518 build systems are going to be the way forward. Poetry might not last forever, but I think`pyproject.toml` has legs.

Pipenv was a valiant effort that moved the field along heaps, but I haven't used it in my daily life for two years.

https://github.com/python-poetry/poetry

Thanks!

Last I checked, pipenv doesn't work without an internet connection. That's a requirement, and design mentality, that doesn't work for me.

Won't/can't fix issues:

https://github.com/pypa/pipenv/issues/1783

https://github.com/pypa/pipenv/issues/408

Valid point, I hadn’t considered it. Thank you!

> I do full time scientific python

Which is a very different use case from being a sysadmin for a network used by a large university, which is the use case of the author of the subject post. So it's not surprising that the "obvious and simple" solution that works for you doesn't work for him.

Also, the post discusses the option of using a virtual environment and explains why that option was not taken. So it's not like your "obvious and simple" solution was not considered. It just didn't turn out to be the right solution for this author's use case. (For one thing, the author explicitly notes that there is no "home" directory for this use case in which a venv can be easily set up.)

Virtual environments and installing to $HOME are different things. He says using virtual environments would be a lot more to learn but he just reinvented them.

> Virtual environments and installing to $HOME are different things.

I know that, and you know that, but I'm not sure the poster I was responding to knows that. In any case, for the use case of the author of the article, there is no "home directory" to use, so installing to $HOME was not an option.

> He says using virtual environments would be a lot more to learn but he just reinvented them.

He invented a sort of poor man's venv that just meets the particular needs of his use case. Sometimes that's a better solution than adopting a standard tool that requires more work and understanding to use.

Reply on news.ycombinator.com