Hacker News Reader

Comment by neckardt | Hacker News Reader

I'm one of those people using 'sudo pip install'. What should I be using instead?

Python -m venv in your project directory. Then source ./bin/activate. Pip install from the project directory as your regular user (no sudo). You can also set up venv (virtual environments) in home or somewhere else instead of for each project. This is called python virtual environments and very easy to adopt.

> Python -m venv in your project directory. Then source ./bin/activate.

And how do I do that from inside an already running Emacs session? (Or Vscode?)

Needing to "prepare" an environment before you can work with it, prevents you from simply "jumping" into a task from an existing workflow.

Working with .NET, Rust, Node, go, Haskell, etc... I have no such issues.

This "venv"-requirement only applies to Python, and it's really the opposite of ergonomic. I really think it's about time the Python-community starts recognizing this.

I generally recommend against `source ./bin/activate` for virtualenvs — it's convenient if you're using a single shell session interactively, but it plays quite poorly with the Unix process model of environment variables being inherited from parent processes to child processes.

Instead, I recommend just running the other executables in the bin directory. `./bin/python` will drop you into a Python REPL with all the packages installed in that virtualenv available. Your IDE should let you specify what Python interpreter to use for running tests and other integrations; give it the path to that Python executable and everything should Just Work. Likewise, if your package uses setuptools entrypoints, they'll be installed into the bin directory too, and you can just run them with no further configuration.

For what it's worth, Rust (for one) works pretty much the same way. The only difference is that Rust doesn't have a concept of dependencies installed system-wide, they're always handled per-project like venvs. As a result, they don't have decades of stale tutorials online telling people to do the wrong thing.

I usually give myself a `./exec` shell script that sources `./bin/activate` then `exec $SHELL $@`. That gives me a subshell with everything set up, so when I want out it's just Ctrl-D.

Now that I'm used to the subshell idea, I use the same concept across ruby, node, and python; it's almost always a better fit than what the usual packaging tools give you. I don't need `bundle exec` in ruby, for instance.

> Working with .NET, Rust, Node, go, Haskell, etc... I have no such issues.

Respective solutions: NuGet, cargo, npm, go get, cabal. They all solve pretty much the same issue as venv and prepare your project in pretty much the same way. Raw venv can be a bit rough, but poetry solves that.

> And how do I do that from inside an already running Emacs session? (Or Vscode?)

From VSCode I just use the integrated terminal to create the venv, vscode's default python extensions recognizes in-process venvs by default and uses them. (The way it installs tooling is broken for them, though, but that's a bug in the extension.)

Though it would be nice if Code integrated poetry so that just having a poetry-compatible pyproject.toml was sufficient.

> Working with .NET, Rust, Node, go, Haskell, etc... I have no such issues.

For most of those (Haskell with Cabal included), they have the same basic issue and solve it with a project file which is pretty similar in purpose and effect to poetry’s use of pyproject.toml.

VSCode will detect virtual environments within the directory structure.

Node definitely has this issue. There's a reason Node NVM[1] exists.

[1] https://github.com/nvm-sh/nvm

In emacs, use pyvenv-activate. I think it's part of elpy.

Emacs:

  M-x pyvenv-activate

Or in your init:

  (pyvenv-activate "~/code/python/venv_dir/emacs_venv")

VSC:

  C-P
  Python: select interpreter

It's pretty simple, and claiming nuget, node, and cabal/stack as superior means you're either lucky or inexperienced.

Ruby has this figured out well by comparison. Install the bundler gem (which you're going to be installing anyway to get dependencies), cd into the project, and prefix your commands with `bundle exec` to use project gems

Remembering to start/stop virtualenv and remembering the names of your virtual envs is one too many steps

Or just use poetry/pipenv shell (or poetry/pipenv run for one-off commands). Both provide a good wrapper, although poetry seems a bit more stable these days.

Perl has copied this as well with Carton. Works really well and as standard keeps the modules in the project dir.

https://metacpan.org/pod/Carton

If you prefer "bundle exec foo", then "bin/python -mfoo" is pretty much the equivalent and doesn't require running activate before.

Debian already has system packages isolated from user-installed with the dist-packages directory. It is less necessary to manage your own environment.

… and itis a non-reproducible setup.

The issue here is, that you are not storing hashsums and therefire are not safe.

To store hashsums you need a frozen requirements file, which needs another puthon lib … at this point it might be easier to use Poetry or Pipenv.

Use your OS to install the stock Python and the stock setuptools, wheel, and virtualenv Python packages. Then make a virtualenv for every single Python application (virtualenv ./some app/). Then make a requirements.txt file with the pip packages you need and their version numbers and install with ./someapp/bin/pip install -r requirements.txt. If you do something bad, rm -r ./someapp/ and start over.

I disagree to use the stock OS for any part of Python you're using as a regular interactive user. The easiest and safest way is to use a Python vendoring tool like pyenv to install a Python version of whatever you need. After that you can use regular pip and virtualenv without any special paths, and the environment is local to the user, so you're unlikely to interfere with another user's environment.

Pyenv is not unlike nvm for Node. Python moves too fast to expect the OS distributions to keep up.

I've been a regular Python user for a decade and every Python app I've ever used has worked on stock Python. And vendor builds often ship with patches that make it work better on the OS in question. But I can see how devs often want bleeding edge releases.

From a stability, reliability, and CI/CD performance standpoint, I would discourage building your own Python. Things break more the more often your base version changes.

To be fair, any language (and database) moves faster than distributions. I use language package managers or docker for Ruby, Python, PHP, Node, Elixir, PostgreSQL and MySQL. I recommend asdf to support multiple languages and databases in the same project.

It's not necessarily easy to install "stock python". e.g. installing python 3.7 or newer on ubuntu requires quite a bit of initial setup.

You'll be required to install a ton of apt system dependencies, then add a custom system apt package index (ppa), then install it from the new repository and hope you managed to get all required dependencies. I failed several times in a row last time before I got it working, after trying 2 different methods.

You should use `pip install --user`. Or even better, use virtual environments instead. Or, if you need to install system-wide tools with pip, install and use `pipx`.

The current version of pip automatically goes to --user mode if it detects it doens't have permissions to access the system pip dirs aka not being run with sudo

That's great - I did not know that.

I asked this a few weeks ago, and got 8 different answers.

After parsing the sibling threads, in this case there were only 4 distinct answers:

- Python -m venv + ./bin/activate + pip

- poetry

- pipenv

- pip (without isolation of projects)

You may add varying recommandations on the usage of OS python or a user installed one, but I think the same concern exists with most languages.

The main problem is that Python's mantra is supposed to be "There is only one way to do it". Less than two years ago, I read the official PEP doc about "Managing Application Dependencies", and it was a mess. pipenv was recommended (and still is), but there was a critical bug that ended in fatal errors on my development system.

"Python -m venv + ./bin/activate + pip", pipenv and "pip (without isolation of projects)" are really all the same thing, just with different options (or a script around it).

Basically, start with "pip (without isolation of projects)" until it doesn't work for you because of library version clashes, then move to either pipenv or "Python -m venv + ./bin/activate + pip" (which are basically the same thing).

Poetry isn't as widely used.

https://docs.python-guide.org/dev/virtualenvs/

Reply on news.ycombinator.com