Hacker News Reader

Comment by masklinn | Hacker News Reader

Most of the tools you mention don't even overlap.

- pyenv is used to install multiple versions of python (and even distributions e.g. you can install conda via pyenv), independent of the distribution, it can have a plugin to more easily manage virtualenvs for those but that's optional

- virtualenv and venv are largely the same thing, the latter is a subset of the former merged directly into python, it's used to create isolated sets of installed packages (site-packages), as installing everything globally is messy and inconvenient and can be problematic when two project have different requirementws

- poetry is a packaging and dependency management tool, that is definitely the messiest and most confusing part of the ecosystem as there is a lot of stuff and movement, frankly I don't really get it, and the entire thing annoys me to no end

View on HN

Personally, I find Poetry really nice. It abstracts away pip and venv, but in a way that you can still understand what's going on and jump in if you need. Once you understand that it doesn't do much else, it is easy.

Can you please explain the pyenv usecase a bit more? Let us say, I needed Py3.6 for something - I can just use virtualenv with appropriate flags to install 3.6 tooling. How does PyEnv help further.

I have read often pyenv helps maintain different version of python on same system. But if that is also possible by virtualenv/venv what am I missing?

Venv helps maintain separate virtual environments for a given python installation, but it won't actually install python. E.g. if you have Python3.6 installed venv can use it, but you can use PyEnv to install 3.7, 3.8, 3.9, ... Which venv can then use.

Basically Python team sat down to consider how they can make their deployment setup worse, and they came up with the idea to split packaging, dependency management, virtual environments/isolation, and Python version management to each need a different too, and then had different random projects make several tries at of each tool, all at more or less equivalent popularity...

I was told the opposite - that the current mess is because the “Python team” did not sit down to consider deployment setups at all.

You're missing that venv and virtualenv can't do that: they can only create environments for your current Python interpreter, not arbitrary Python versions.

There's no flag that I'm aware of that makes venv install a different Python version than the one it was invoked with.

The --python flag for virtualenv will let you specify a different Python executable to use, but requires that executable to already be installed somewhere on the system.

Getting another python executable is normal the challenge though.

Not really. Not under OSX, brew makes this simple. Under Linux, worst case, you just download the source and compile it. (Not sure about Windows though.)

And pyenv is a wrapper around “downloading the source and compiling it”.

Brew makes this easy? Are you sure? I installed python 3.9 with brew and it’s magically become a 3.10 install by upgrading another package

The version pinning on brew certainly isn't great, but you can do:

> brew install python@3.9 python@3.10 python@3.11

> I can just use virtualenv with appropriate flags to install 3.6 tooling.

No. I have no idea where you got that idea.

> But if that is also possible by virtualenv/venv what am I missing?

That it's not possible? venv is literally just a module of the standard distribution. And while virtualenv can do a few more things (hence why it remains as an independent project), installing new pythons is not something it can do.

Virtualenv and venv can only create environments for previously installed python (especially venv as you create environments via the python you want to create an env for, it doesn't support creating an env on behalf of a different python than the one it's invoked through).

Oh, my apologies. I confused that with the conda --python flag. Much appreciated catching that.

Poetry is basically NPM, Cargo, or Composer, but for Python, and my impression is generally that people who have used those tools see it as incredibly valuable and useful, and people who haven't used those tools don't really understand what the fuss is about. (Disclaimer, I am in the former camp.)

It's a way of wrapping all the different facets of maintaining a package and its dependencies into a single tool, which as much of the details transparently taken care of. So for example, if I clone a new project, and run `poetry install`, then:

1. A new venv is just automatically created, I don't need to think about accidentally clobbering my global Python installation, but I also don't need to think about creating my own venv every time. It just happens automatically.

2. All of the packages I need will be resolved and installed as I expect. Additionally, because there should already be a lock file in the repository, I will be installing the same versions as everyone else working on the same project, meaning we're not going to run into weird dependency issues.

3. I can automatically activate and run the environment with ease to run the test or linting tools that I need.

4. I can create wheels and even distribute them to PyPI or other repositories as necessary.

5. I can update the dependencies in the project based on semver requirements, but also individually, on a case-by-case basis, and go back to previous "known good" easily via git.

But most of all:

6. If I switch to a different Poetry-based project, I can do all the same things in the same way, using exactly the same standardised interface. Which means that I can just jump into new projects, support other open source tools and libraries, join other teams, etc, with a significantly reduced onboarding time. No more funky Makefiles, no more figuring out which tool I'm missing: just use Poetry, and everything is consistent, and it Just Works™.

FWIW, it's not perfect. It's not as powerful as package managers for other languages, so there's a handful of use cases like workspaces for multiple projects that are not well supported. It's also not officially blessed by the Python maintainers in the way that Cargo or NPM are, which means it's always slightly more of a risk, and means people need to install it before they use your project. Also even if it were perfect, brilliantly maintained, and used by nearly everyone, there's still a handful of use cases where it would break down - even with Cargo, some projects still have manual calls to rustc to handle their specific use cases.

To me, right now it's an 80% solution (in the sense that 80% of projects will find 100% of their needs satisfied by it). I think it could become a 95% solution if it came popular enough, and if there were more core Python support. (Or alternatively: an alternative that was officially maintained by Python maintainers like venv or pip could be a 95% solution.)

> Poetry is basically NPM, Cargo, or Composer.

First of all, no, not even remotely.

But second, my remark was about that segment of the ecosystem, not poetry specifically or exclusively: poetry is just one of numerous players in a field that's been in flux for years.

What do you mean no? I genuinely don't understand what you mean by that. The whole point of them is to be a consistent CLI for managing projects, dependencies, and packaging. It's even using the same sort of vocabulary and covered m concepts, right down to the .lock file.

How do you see Poetry operating then, if not as an equivalent to those tools?

The comment you are responding to did a great job of explaining the situation. I don't know what you are trying to say.

Reply on news.ycombinator.com