Hacker News Reader

Comment by pjmlp | Hacker News Reader

I just have latest Python installed and the nice support on Visual Studio.

https://www.visualstudio.com/vs/python/

This, me too.

I never understood the need for virtualenv and similar. Do people really encounter trouble with conflicting packages that often? I try to write scripts so they run on different versions of python anyway, unless there is a very specific reason why that is not possible; and even then you can run python versions in parallel on a Debian/Ubuntu box, with different pip installs for each of them.

As for production, I usually ship things in a Docker container anyway, so there is no chance of mismatched libraries.

I guess I just never saw a problem that virtualenv solves.

I experience version conflicts all the time. I work on multiple projects and virtual envs are life savers.

The linked-to essay describes several reasons why the author uses virtualenv. Which of them don't you understand?

My package supports several optional back-ends, selectable at run-time, and it runs under Python 2.7 and 3.6+. I use tox to manage different virtualenvs for the combination of {backend X but not Y or Z, Python 2.7}, {backend X but not Y or Z, Python 3.6}, etc. for Y, and Z, as well as {backend X and Y and Z} for the two versions. Oh, and I support several releases of each of the X, Y, and Z. That's a lot of virtualenvs.

Usually I only develop on the X+Y+Z version because full test suite across all the combinations takes about 15 minutes.

I also do coverage testing, and the Python coverage tool isn't hard to use under this setup to combine tox results across multiple virturalenvs.

I don't know if Docker is better. I've been using this setup for some years.

I use virtualenv so I can just easily `pip freeze > requirements.txt` for individual projects.

pipenv is basically the next level of this process: it records every install and locks only the things you installed rather than their dependencies which often change over time.

I'm a bit confused by this statement, do you mean:

I install foo 1.0, which depends on frob >= 1.0 (which happens to be frob 1.1 when I installed it). Foo 1.1 comes out, as does frob 1.2 and 1.3. If I reinstall from the pipfile, do I get foo 1.0 with frob 1.3?

I ask because that sounds like a bug waiting to happen. IMO, frozen requirements should remain frozen.

Here's what happens:

You type `pipenv install foo`. It will create the virtualenv if necessary and adds `foo = " * "` to the packages section of the Pipfile. The Pipfile.lock file will add a section for the package you installed _and_ all of its dependencies, including the hashes of the downloaded packages.

That avoids accidental breakage if the package you depend on doesn't set their versions correctly but it also means that your main Pipfile documents the things which you intentionally installed so a year from now you're not wondering why all of your servers have frob 1.2 installed, which only works on Python 2.7, even though nothing you're using now depends on it.

As a concrete example, here's what `pipenv install requests` looks like in a clean project:

Pipfile:

    [[source]]
    
    url = "https://pypi.python.org/simple"
    verify_ssl = true
    name = "pypi"
    
    
    [packages]
    
    requests = " * "
    
    
    [dev-packages]

(had I used `--python $(which python2.7)` it'd have recorded that as well)

Pipfile.lock:

    {
        "_meta": {
            "hash": {
                "sha256": "a0e63f8a0d1e3df046dc19b3ffbaaedfa151afc12af5a5b960ae7393952f8679"
            },
            "host-environment-markers": {
                "implementation_name": "cpython",
                "implementation_version": "3.6.4",
                "os_name": "posix",
                "platform_machine": "x86_64",
                "platform_python_implementation": "CPython",
                "platform_release": "17.4.0",
                "platform_system": "Darwin",
                "platform_version": "Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64",
                "python_full_version": "3.6.4",
                "python_version": "3.6",
                "sys_platform": "darwin"
            },
            "pipfile-spec": 6,
            "requires": {},
            "sources": [
                {
                    "name": "pypi",
                    "url": "https://pypi.python.org/simple",
                    "verify_ssl": true
                }
            ]
        },
        "default": {
            "certifi": {
                "hashes": [
                    "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                    "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
                ],
                "version": "==2018.1.18"
            },
            "chardet": {
                "hashes": [
                    "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691",
                    "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae"
                ],
                "version": "==3.0.4"
            },
            "idna": {
                "hashes": [
                    "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4",
                    "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f"
                ],
                "version": "==2.6"
            },
            "requests": {
                "hashes": [
                    "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
                    "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
                ],
                "version": "==2.18.4"
            },
            "urllib3": {
                "hashes": [
                    "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                    "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
                ],
                "version": "==1.22"
            }
        },
        "develop": {}
    }

(EDITED: the HN Markdown parser appears to be a simple regex match and breaks formatting with a * and uses only the ASCII definition of whitespace so I couldn't use a zero-width space. The real output doesn't have spaces around the asterisks).

> Do people really encounter trouble with conflicting packages that often?

Yes it's quite common. For example I work on multiple projects with different versions of Django being used in each one.

A rather simple practical example I'm working on right now: moving a Django app from 1.9 to 2.0. There are a few code breaking changes in 2.0. All I need to do is create a new virtualenv with Django 2.0 and I can just switch between both versions simply by changing the environment. I may even need to change some further packages that do not work with 2.0 yet and swap them for something else. I can do that in the new env. Then, when all is working, I export a requirements.txt from my virtual env and can quickly set up the very same virtual env on my production machine.

Yes, it is very common. I work a lot with ML code released as part of research papers. Everyone uses a different version of TF/PyTorch or whatever framework they are using. I can't imagine working without something like virtualenv (though I use pipenv)

Do you just use one environment for every project?

That gets messy. What if you want to setup a new machine? Or a new employee's machine? Etc..

It isn't only about conflicting packages.

Yes - but then again, I don't have that many projects and I tend to use the same libraries across them anyway.

Installing on another machine is just a matter of pip install -r requirements.txt, if it's for a dev. other wise it's just a docker run ....

I do know that most Python devs use virtualenv, I just never understood what all the fuss is about. Of course I don't mind if the others in team use it, I just never saw the appeal so I'm curious what other peoples' reasons are.

Maybe I don't see the need because I came to Python quite late, a little before Docker came around, and we were early adopters for Docker. So we solved testing and production with Docker (and thus never needed to support multiple Python versions in parallel). It's really one of those small mysteries I don't understand. :)

Yeah I understand what you mean. I have around ~12 projects, for different clients, that are just recent enough that I might need to access the environments.

If that was one environment (forget some of them being stuck on 2.x) the number of damn dependencies would be monstrous!

When I need to bring a partner onto the project, I can't give them a requirements file that's 5x what it should be!

> I just never understood what all the fuss is about.

It's to keep environments clean + easy to maintain. Virtual is probably a bad word for it. It isn't like Docker or a VM.

I don't think too many Python projects that are not libraries support many versions, that isn't the point of virtualenv or conda anyway.

I've found that the library dependency issues appear in direct proportion to the number 3rd party libraries you use. If you're mostly developing against the standard library with a handful other stable libraries, it's not an issue. If you're developing with a requirements.txt (or pipfile) with 10 or more entries, you'll start running into conflicts on a regular basis.

It can also occur as projects age, especially with 3rd party libraries who don't provide API backwards compatibility (which I fully acknowledge is a PITA to develop for, more effort than is frequently justifiable).

Both of these are why I prefer to use the standard library when possible. It's going to remain API stable for a very long time, and the occasional 2-3 lines of boilerplate to do HTTPS requests (and other similar convenience functions) is a cost I'm willing to bear for that stability.

I think pretty much everyone would "prefer to use the standard library where possible". But try building a modern web app with that. No one uses libraries because they want to, they use b/c they need to.

> No one uses libraries because they want to, they use b/c they need to.

I'll have to respectfully disagree with this one. Everyone pulls in requests the moment they have to make any kind of http request for the sole reason that it's more ergonomic, not because it's "needed".

And requests brings in 4 of its own dependencies. Right there you've created a prime chance for everything to go sideways (and I've watched it happen explicitly with the requests library as it bolts on more and more ergonomic features).

For what it's worth, the last web app I built had a DB library from the OS vendor, flask, and gunicorn. All of which, since they were quite stable, never introduced library conflicts.

Docker containers and virtual envs are in some way both capsules that can serve a similar purpose. virtual envs are in general more lightweight, but limited to the Python world, whereas Docker containers are heavier, but enable you to also include other dependencies in that capsule.

I sometimes feel that Docker containers are a bit misused if they are just used as environment capsules. They provide more isolation (process space, etc.). But of course, that would be still fine.

The software I write mostly does not require databases and other supporting services to run on my laptop. So virtualenvs are all I need (most of the time). There is a docker tax that I don't want to pay, unless I heavily benefit from it.

I guess, if I could deploy docker images, the return of invest would be much better and I might use it more.

How does it compare to PyCharm?

Having worked a few years on both, they're both great.

It's on par, as long as you don't do too many refactors on nasty codebases. Now, PyCharm chokes on those nasty refactors too, guessing the right stuff only 70% the stuff in a way that sometimes feels purely random, but at least its find and refactor preview UIs save the day sort of...

One thing that annoys me on VSC is that some operations (jump to defnition etc.) have delays on larger projects, because it does not create an "index db" of your code behind the scenes like PyCharm does, so it can show you usage and search for stuff practically instantly.

Overall VSC is great for developing Python on not-huge and well-behaved projects. For degenerated overgrown messes it's not a good solution :)

Visual Studio Code is not the same as Visual Studio, which is what the op was referring to.

Just an FYI..

The parent comment was about Visual Studio(https://www.visualstudio.com/vs/python/), and you seem to be commenting about Visual Studio Code(http://code.visualstudio.com), which are 2 totally different products.

Microsoft has terrible naming practices.

Type hints have made PyCharm an even more amazing IDE. Start using them if you're on a version of Python that supports it! :)

just for curiosity, how you type an equivalent of a ts union?

Thanks pjmlp!

Just a note of clarification. There are two "visual studio"s:

https://www.visualstudio.com/vs/python/

This is the classic Visual Studio & runs on Windows only, is a full featured IDE and has goodies like mixed mode Python/C++ debugging.

https://code.visualstudio.com/docs/languages/python

This VS Code, a cross-platform Editor++ with Python support.

Both are being actively worked on and will see continuous improvements.

[disclaimer: manage the dev team]

Reply on news.ycombinator.com