Python Environment
xkcd.com
xkcd.com
Pipenv is the most promising solution today but is still very, very new. It's modeled after yarn and has been officially blessed as "The One True Way" of installing stuff by the Python documentation. It has a way to go still to be as good as yarn (especially in terms of speed). The Python ecosystem has never had proper declarative packages like package.json (setup.cfg can be used to have fully declarative package metadata, but I seem to be the only one using it that way), which is a problem for package managers.
To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. It avoids the need for virtualenv, kind of, but so does pipenv and it does so more reliably and reproducibly (pipenv implements lockfiles).
Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.
Oh? I don't see any mention of pipenv in the cpython git repo.
> Application dependency management
> Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Dependencies for more details on using pipenv.
Package managers for languages were just starting to become a thing back then. (At least I think that is true...). Now days you wouldn’t dream of releasing a new language without some kind of standard package manager...
Dunno where I’m going with this comment but it is definitely interesting to see how much has changed since then. Hell even Microsoft has a package manager for it’s languages...
Isn't this one of the major criticisms of Go?
Rust is the exception here in that Cargo is a very nice piece of kit.
In terms of generic runtime version managers, ASDF is very nice cross language version manager for scoping versions to projects and switching between them easily.
Also Rustup, which is pretty key to Cargo remaining flexible going forward. There's a huge advantage to having the build/dependency management piece being highly coupled with the language runtime/compiler. Making cargo work with all versions of Rust would mean having to build in backwards compability and make it much more difficult to do projects like the new modules changes.
It's a key insight that there needs to be two tools surrounding a language, not just one. You need good dependency management and good version management. Node also has this right with nvm and npm/yarn.
Just consider urllib, urllib2, urllib3, and requests. Or getopt, optparse, argparse, and docopt. Or countless others.
I appreciate that the Python community continually strives for ever-more obvious and ergonomic ways of accomplishing tasks. While there is an ecosystem complexity cost to the solutions-that-weren't, it's no greater than the churn than I've seen in other popular languages.
The Zen of Python even pokes fun at the "one obvious way" by writing its three em-dashes three different ways in the same document.
I did not know this! Thanks :)
- It's very slow: re-locking after updating 1 dependency often takes me ~1 minute.
- It has lots of bugs. To name a few in 11.6.9: clobbers comments in the Pipfile, inconsistently includes dependencies for other build environments in Pipfile.lock, stores the wrong index in Pipfile.lock for packages not on PyPI.
- They release multiple times per day, often breaking things in patch releases.
- Kenneth Reitz is quite unpleasant to deal with in GitHub issues, which I often have to because of the previous 2 issues.
From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
https://github.com/pypa/python-packaging-user-guide/pull/443
He doesn't seem to be a pipenv maintainer:
Keep up the good work.
> From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
This is blatantly false.
Racket's exe are so simple and quick to share.
raco exe foo.rkt
(Edited tendon to heel)
Back in the day, we used one of the various "freeze" applications (cxfreeze, bbfreeze, probably others) and some tools developed in-house to package and distribute them for various platforms.
It has been a while since I actually needed to do this, so I don't know what the current state of the art is.
Racket has such a great story for so many things that are painful in other environments. It's an under-appreciated language.
A) Racket is so enjoyable everyone builds their own solutions
B) "The view that full-fledged problem solving almost always calls for language design" Which sadly falls onto deaf ears especially people from Python.
I'm not sure it's because of backwards compatibility. Perl has amazing backwards compatibility, but doesn't seem to suffer from this.
That's not said to pump up Perl or degrade Python, but because if you misidentify the problem, your proposes solution has a much lower likelihood of fixing it.
(Then again, maybe you mean something different by "backwards compatibility" than what I thought you meant)
First of all it's not in Debian unstable. Ok, maybe it's just super new... so I'll install it with pip. A python3 -m pip install pipenv later and let's try it out!
$ cd ~/src/nexsan-exporter/nexsan-exporter $ pipenv install Creating a virtualenv for this project… Using /usr/bin/python3.6m (3.6.5) to create virtualenv… ⠋Running virtualenv with interpreter /usr/bin/python3.6m Using base prefix '/usr' New python executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m Also creating executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python Installing setuptools, pip, wheel...done.
Virtualenv location: /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG Installing dependencies from Pipfile.lock (ca72e7)… 0/0 — 00:00:00 To activate this project's virtualenv, run the following: $ pipenv shell
Cute emoji and nice colours but... ~/.local/share? For a directory that will end up containing arch-specific libraries? Ok, I guess no one gives a shit about this in the modern world, oh well. Wait... "python3.6m"? That isn't the Python interpreter I asked for... but it seems to be a hardlink to the same file as /usr/bin/python3.6 so I guess maybe this is intentional? Anyway let's check out the venv...
$ ls -l ~/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3* lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3 -> python3.6m lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6 -> python3.6m -rwxr-xr-x 1 yrro yrro 4576440 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m
So this is the Python folk's what, fourteenth attempt to get this right, and they are still copying the python executable into the virtual environment instead of symlinking it in? This seems to be a regression from venv, which seemed to get this right!
Right, time to install my dependencies... according to 'pipenv' the right command for this is 'pipenv install -e .' Hm, I wonder exactly what the -e option does?
$ pipenv install --help
* no menition of -e in the output *
:sadface:
Oh well, let's just run it blind!
$ pipenv install -e . Installing -e .… ⠏ Error: An error occurred while installing -e .! Directory '.' is not installable. File 'setup.py' not found.
Maybe I screwed up and ran this from the wrong directory?
$ ls setup.py setup.py
Weird, what's going on here?
$ strace -f pipenv install -e . 2>&1 | grep setup\\.py stat("/home/yrro/src/nexsan-exporter/nexsan-exporter/setup.py", {st_mode=S_IFREG|0644, st_size=1769, ...}) = 0 stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0200) = -1 ENOENT (No such file or directory) [pid 7365] stat("./setup.py", 0x7fff84176e30) = -1 ENOENT (No such file or directory) stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0300) = -1 ENOENT (No such file or directory) write(2, "Directory '.' is not installable"..., 62Directory '.' is not installable. File 'setup.py' not found.
Oh FFS, I give up. I think I'll let pipenv pass me by for now.
I agree it's frustrating. I'm planning to donate some of my time to the project to help the situation. I hope others can too.
Can anyone give me a rundown of the benefits of other systems over virtualenv?
On a side note I hope Python isn't going to turn into a shambles like JavaScript - its certainly starting to look like it as far as package managers go.
But as I said, it's too young. If you use pipenv, you still have to interact with pip, so that's two binaries whose commands you have to remember. And urgh, it's so slow. And it's kinda useless for libraries... pipenv doesn't publish stuff, pipfiles aren't appropriate for packages.
But we'll get there. One day. Hopefully it won't be too late.
Pipenv is promising. That means at some point I'm fairly certain it'll be and deserve to be the standard in Python packaging. In the mean time, it doesn't.
I am teaching adult beginners and it would be best to avoid the messy parts in the beginning.
I thought I could get by with just pip and briefly touching on virtualenv.
I sin myself by avoiding all the mess by cloning a full bare bones Lubuntu virtualization whenever I need a clean project.
Theoretically Docker should suffice but it is yet another layer of complexity.
Dongles for days.
The OS comes with a python runtime, which is different from the runtime you can download from python.org.
You can then also install it from Homebrew or from MacPorts.
I'm probably even missing something.
conda install r-essentials
All of these methods people are listing that requiring me to do more than manage one file (requirements.txt) is sort of missing the point.
I get that a lot of Node developers are coming to Python to do things, and that's great. But the environments don't need the exact same tools. pipenv seems to fix a problem I never encounter with pip/virtualenv during daily development work.
Pipenv is the first (popular) sane tool to make Python accessible to programmers coming from other languages. Sure you can also just use bare virtualenv, but then you have to figure out how it works together with everything displayed in the picture.
No amount of adding things to the Python comic is going to simplify it.
> No amount of adding things to the Python comic is going to simplify it.
It doesn't simplify it, but it puts you in a whole new canvas where you don't have to worry about whats in the messed up one.
How so? I use virtualenv and don't have any issues. Seems pretty straight forward to me. I haven't moved to pipenv because it forces a folder structure on you, which I hate. The great thing about virtualenv is that you can be anywhere on your system, it doesn't matter, as long as you see the project name in paren's next to your prompt you know you are good.
> The great thing about virtualenv is that you can be anywhere on your system
That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
>That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
Yes they should be decoupled, but why would I want in the same folder? If am building a Django app I don't want the entire instance of python inside my project folder for a few reasons. 1) It clutters it up. 2) The web service (user/group) needs to access the Django folder, why would I put python in a folder that the web service has control over? I much rather have my web application in /svr/www/webapp and my python virtualenv in /opt/python-virtual/webappname
yeah but then I can't be in the Django web application folder and just type "python manage.py makemigrations", for example. I would have to be the pipenv folder then "python /srv/www/webapp/manage.py makemigrations".
Unless I am missing something, which could be the case.
Homebrew pulls it directly from the Python.org, but it's also slightly opinionated in what else it installs. I would suggest just reading through the formula to see what it does (it's straighforward).
Likewise, you can quickly see what any Homebrew package is doing with `brew info <package>` and looking at the "From:" link.
I hope it becomes the standard going forward.
https://github.com/pypa/pipenv
It combines many features that people manage separately with virtualenv, pip, and any custom scripts. I think it reduces boilerplate and simplifies env management.
It is still under active development, but I think it is stable enough to use for production. I think it's probably better for people learning Python to use too.
If you're using pyenv, it's easy to do virtualenv with the pyenv-virtualenv plugin, at which point you don't need Pipenv because it doesn't solve anything (except locking, see blow) that you can't do with requirements files. Furthermore, if you're testing under multiple versions of Python, you'd use tox to manage your test environments anyway.
_Further_ furthermore, they authors recommend _against_ use pipenv to manage dependencies for anything but development, so it's on you to have a copy of runtime dependencies isolated from Pipfile. And since you have to put your runtime requirements in setup.py anyway, there was never any way around the issue of duplicating requirements to begin with.
The one thing Pipenv does that pip and requirements files can't is lock dependencies down the entire tree... except it's abominably slow at that because it has to download dependencies and walk the entire tree. There's a debate ongoing about using pip's cache to solve the problem but there's no end in sight.
If you're really sensitive to having the entire dependency tree frozen to exact versions of you requirements, just `pip freeze` them occasionally. It's way less hassle.
Packaging in Python is far better than it was a few years ago. There's still a lot to improve but pipenv shouldn't be part of the solution.
I think the use-case is slightly different than that.
Pipenv is a tool that enables reproducible builds for python app development and deployment. For example, many web apps, aren't going to be packaged up using setuptools, they are just going to be deployed somewhere (possibly containerized first). And with pipenv you can avoid the need to vendor all your dependancies to ensure reproducible builds.
But its not a tool that solves problems for library packaging and distribution (on say pypi). You still have to use setup.py for that, and shouldn't be pinning your dependancies on exact package versions. But you can still use pipenv to manage your dev environment in those cases, and for reproducible dev environments.
https://news.ycombinator.com/item?id=11851871 https://archive.is/MVVU1
Explanation edit:
Image alt text is "The Python environmental protection agency wants to seal it in a cement chamber, with pictoral messages to future civilizations warning them about the danger of using sudo to install random Python packages."
The problem space is very difficult - it'll be something like AD 12000 before the waste is effectively harmless. That's further in the future than all of civilization[1] is behind us. The earliest surviving language specimens are proto-heiroglyphics that are less than 5000 years old.
It's difficult to craft a message that won't be taken like all the myriad curse-promising looter-deterrents guarding e.g. plundered tombs throughout the world. It's especially unconvincing when the hazards we're trying to protect future humans from include things like a hundredfold increased risk of cancer.....after inhabiting the site for twenty years.
It's further difficult to attempt to ensure that the message survives that massive amount of time. The designers came up with a linked system where elements are supposed to reinforce and index each other, with redundancy of information and presentation at multiple levels of complexity, with written and pictographic forms, attempting to avoid overstatement, attempting to ensure that protective structures cannot be usefully scrapped and re-used for new construction, ensure that the communications resist deliberate vandalization, etc.
1: https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_c... claims "Though early 'cities' appeared at Jericho and Catal Huyuk around 6000 BCE,[32] the first civilizations did not emerge until around 3000 BCE in Egypt[33] and Mesopotamia.[34]"
It’s so tempting to just apt install everything until it’s too late.
Sure, it caused a lot of grief for Debian packagers (because the name "site-packages" was hardcoded everywhere) when it was introduced back in the Python 2.6 era; but I don't think I've ever had to worry about it as a user.
It's not just official Debian packages that someone needs to maintain...
When it comes to binaries it's still a bit of a mess (homebrew? pecl? yum?) but the native PHP code story is clearer than any other language ecosystem I've seen.
I encourage people to try it out, interactive usage looks something like `nix-shell -p python36Packages.pyaml -p python36Packages.aiohttp --run "python some-script-with-dependencies.py" `.
But yes, if you're making sure that you're always careful about passing the location of your environments, you don't really need it.
Edit: It looks like pipenv, which hopefully will replace virtualenvwrapper and the like, supports storing the environment in the project folder: https://docs.pipenv.org/advanced/#changing-where-pipenv-stor...
But in development, where you manually switch between environments, a centralised setup is great. You don't have to worry about gitignoring the virtualenv directory, or maintaining paths in general -- a common problem with virtualenv in your code directory is that IDEs and linters and similar tools tend to just cut through and parse everything, unless explicitly prevented. With virtualfish/virtualenvwrapper, the process is simply `workon {envname}` and you have everything in place.
Legacy setups from previous decades, laziness, automated installers written by others (so they play by different rules), semi-broken packages with messed-up dependency graphs that require manual treatment. The comic is, of course, exaggerating - or at least I hope no one have things gone that bad.
> every sane programmer uses ... pip in a virtualenvwrapper environment
Check out Pipenv <https://docs.pipenv.org/> - you may like it. It aims to make things saner that bare pip+virtualenv{,wrapper}, and IMHO it really does.
The packagers and package tool makers of the world inherit ALL of the technical debt from ALL of the upstream software devs. They're either the liver or the colon. An upstream dev decided to not document which compiler they use during dev time? That's now your problem. A compiler maker (GNU, MSFT, etc.) decides to change how they distribute the C runtime? Congrats, now it's your packaging system's job to know how to differentiate between Windows 7 and Windows 10 running particular versions of Visual Studio.
Nobody wants a "single simple unified solution" for packaging more than the packaging tool makers and distro vendors. Believe me. But it's not going to happen until software devs become take more responsibility for what they build upstream, and how they build it.
Python is big and important enough that it needs these kinds of quality assurances. Punting on them is as bad as compromising on language design decisions.
- Install brew w/ the one-liner on https://brew.sh
- `brew install python3` (which includes pip) and any system dependencies you need: C libraries, etc. For example, `brew install openssl libpq libffi libsass libsodium ossp-uuid` and so on
- Create a virtualenv and install all python stuff in there. Doing `virtualenv -p python3 venv` in the root of your project works. But, if you want to get fancy, there's pyenv, pyvenv, and the like. Use one that works for you.
This seems to be the same as the experience with Linux, except you don't need to install the package manager. In Docker, you also don't need a virtualenv.
Or maybe people are using more difficult to install dependencies than I am? Even if you have to install from source though, it's not a huge deal.
Another possibility is people are using IDE's that want you to do things "their way". I don't know anything about this b/c I use a terminal-based text editor (not trying to start a flamewar).
And the final thought is just that people are not experienced with package managers or general terminal use. You obviously have to be fluent in terminal to use the workflow I outlined above.
https://packaging.python.org/guides/tool-recommendations/#ap...
I install all my Python modules into the system's Python using my OS's package manager. I have both Python 2 and Python 3 installed but beyond having the same module installed in each of those locations, I have never found the need for multiple versions of the same module installed at the same time.
On the rare occasion where my OS doesn't provide a package I need, I use pip install --user. If it turns out to be something I'll want for the long term, I just knock together a quick package that I can install/uninstall using standard OS packaging tools.
If that's the case, then it's as you said: you "just never do anything complicated with Python".
Um... not quite that simple. If you are just developing for yourself, and you have full control over the deployment environment, then of course things are quite simple. (Assuming you don't need to track a fast-moving ecosystem, like machine learning, where packages update constantly and there is a huge rat's nest of interdependencies between OS, kernel, hardware drivers, and algorithmic libraries.)
You might program very complicated algorithms in Python, but you don't anything complicated with them (shipping them to customers, supporting 100 servers, different distributions, and so on).
I also like using Docker because it handles non-language resources like databases. Even installing one version of Postgres locally was painful. It's hard to imagine trying to deal with multiple instances and versions on the same machine (without something like Docker or Vagrant) if two projects use 9.x, and another uses 10.x.
It's still difficult to debug stuff inside your container (unless you do a docker exec and then you're limited to the tools you installed) so it's useful to have the same environment outside of it. `pip3 --user install` has been my friend; and you can create new users instead of full virtulenvs if you really need to.
Still, my python environment isn't as screwed up as this comic. System python is always only controlled by the OS package manager. My own stuff is installed with --user and things I need to deploy somewhere have a Dockerfile or I have a Jenkins workflow that create rpms/debs with fpm and push them to my repo and sign them.
[1] https://www.gnuradio.org/blog/pybombs-the-what-the-how-and-t...
* Worse, when different package builders release binaries, there is insufficient information in the binaries metadata in order to know which sets of binaries will actually install AND run properly on any given system.
* Finally, the Python packaging ecosystem suffers from a "Tesla Autopilot" problem: some kinds of things are actually WORSE if they only work 90% of the time. So, many people get by just fine with pip and virtualenv... until they don't. Since package building and dependency-solving are not exactly the sexiest or most fun part of software development, most devs in the modern era don't necessarily take the time to understand the actual roots of the problem, but instead bat around lore and cut-and-paste stackoverflow until things seem to kind of work.
A major part of Python's current success is due to its numerical and data analysis libraries. These, in turn, are successful because they take advantage of deep capabilities in C, C++, and other "native" code. This means that Python packaging inherits the original sins of C (the dynamic linker) and of C++ (no standard binary ABI). Most other languages do not have to solve such a hard problem: Perl, Ruby, Node, etc. all don't go nearly as deep as Python does in terms of leveraging a rich ecosystem of native code extension modules.
Even Java avoids this and lives almost entirely within the JVM runtime - but even then, classpath conflicts show that dependency management in ANY language is a hard problem unless treated holistically and intentionally. Python got its package system bolted-on after the fact. Then several tools came and went as maintainers entered and exited the ecosystem. With Anaconda and conda, we are just now finally at a point where people can reliably install the basic scientific and numerical libraries, across hardware and OSes..... 20 years after I started using the language. :-)
This issue is one of the things that drove me to utilize a dedicated VPS for each project. This keeps environments clean, ensures project/client isolation and allows me to utilize the exact resources needed, as opposed to worrying about the local machine.
Once done or if the prod environment changes, I can blow away the VPS and rebuild as needed.
When I do a new OS install I will try pipenv again...but it dind't work well for me the first time due to conflicting with something I had installed outside of a virtualenv!
Seriously, starting from a clean osx, how do you install python? I always liked using vex for venv management. But I would also like to have anaconda for some tasks..
I was just going through an old server of mine. Python 2 and Python 3 dependencies, installed over years, in a bit of a mess.
We may have virtualenv and the like today, but a lot of old stuff was never setup to use it. :p
docker pull; docker run, you are golden. Even if you are going to customize the official image, it's much less cumbersome than that xkcd cartoon.
https://www.liquidweb.com/kb/how-to-install-docker-on-centos...
I think what a lot of comments here miss is that while new tools come out that can help make things better, they don't suddenly fix people's existing installs. It seems weird that there isn't more documentation out there to help people get out of these kinds of messes.
As someone responsible for probably one of your astronomer friend's tools (Anaconda), I have to say that we can only do so much: the nature of Python itself is that we cannot ignore what the user puts into PYTHONPATH, PATH, etc. On Windows, at least, we do create shortcuts in the Start Menu to ensure that we get a well-defined command shell. But if the user forces things like PYTHONPATH, there's not much we can do to un-fsck the setup.
Maybe it would be useful to have a python-safemode (or python-portable) binary that explicitly ignores all environment variables, configuration files, registry settings, dotfiles, etc. etc. and only looks at files that are next to it in the path....