Python on Wheels
lucumr.pocoo.org
lucumr.pocoo.org
Turns out the answer is:
1) Yes, you can kind of support packages with c-plugins, if you're specific and careful.
2) They make server deployments significantly faster.
That's actually pretty neat.
...but damn, they still look like a massive pain to work with.
Apart from all of this pain, I'm very happy to be able to use them.
I'm really waiting for someone to build a "wheel on demand" website that would be a proxy to pypi. Or to turn pypi into a wheel farm (I don't know how realist this idea is).
I have plans for either turning PyPI into a build farm, or making a secondary service that acts as a build farm for PyPI. We've just been more focused on cleaning up other issues.
Removing 'bundle' was painful for us, and I even contributed a bunch of code to 'pip2pi' [2] to make it easy to set up an S3 bucket as a "local" pip mirror for our EC2 infrastructure, but it was all bandaid after bandaid.
So far, Packer has felt much more elegant (despite the effort it took to put it in to our workflow) than wheel, pip2pi, or any other solution.
[1] http://tech.flyclops.com/replacing-pip-bundle-374
[2] https://github.com/wolever/pip2pi
(Edited for grammar.)
Thanks for these links!
It is just an unfortunate circumstance of the community not efficiently converging on well designed standards quickly enough, and so, the rocky soil has sprouted strange twisty crops. There are articles on how the setuptools/distutils frankenhack pretty much set off the whole trainwreck that you see today, e.g.: http://blog.startifact.com/posts/older/a-history-of-python-p...
Armin (the OP) has also written about this sad history previously: http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
The problem with package managers is the same as npm/gem.
..yum.
Actually it is even true for music instruments: A piano is very complex inside and can be played relatively easily, while a violin is much more simple in the fabric but much harder to play.
This is a solved problem with a really bad case of "Not-Invented-here" in the python community.
What's funny is I'm pretty sure gems/bundler and npm/packages were heavily influenced by pythons plethora of crappy attempts (eggs, distutils...etc) at package management.
Every solution has drawbacks, but python's solution has almost every drawback.
Anyone who's worked with all three of those would tell you that this isn't yet a “rest on your laurels” situation.
There are improvements to be made to all of them but gosh python is a complete mess in this regard.
Another illustration, in reverse: the Chinese chopsticks are among the simplest tool you can imagine, and it requires a somewhat steep learning curve, but once mastered it is really the most powerful food grabbing tool ever invented.
The Chinese brush is of the same nature: extremely simple, very hard to master, and very powerful (expressive) once mastered. Just like the violin (and VIM, and git, etc.)
On the other side of the spectrum you have the synth, IDEs, Google search, etc. Simple UIs, simple to learn, but internals are very complex.
And yes, one usually installs easy_install to install pip, but I don't see how that can create a "very long dependency chain". Even when both tools are installed, you can use either to install packages.
First of all, there's the cases in which reality fails to live up to expectations, like this packaging debacle. If there was an obvious and best way to do everything, we wouldn't need to spend so much money and time engineering software. We wouldn't spend so much time arguing about distributed vs. centralized, OOP vs. declarative vs. functional, type systems, build/test systems, etc. The mantra is arrogant on its face.
Secondly, it flies in the face of innovation. It works against building a creative community of users. So often, when Python users point out the holes in the ecosystem, they are simply told "well the long and outdated PEP 5991834 already establishes the obvious way to do it" or "just shim it on top of this inadequate corner of the standard lib" or the use case itself is questioned, which patronizes the criticizer. The mantra becomes an excuse for reinforcing a hive mind philosophy.
In this case, I wonder if it has subtly undermined the development of a diverse and widely supported package index. The whole point of such a package system is to support the belief that there are actually many ways to do the same thing, and some may be better for certain styles/teams/circumstances. Having the opposite outlook as a design principle for a language will attract users less inclined to contribute to or build such a diverse packaging system. I wonder this particularly because the more "laissez-faire" languages like Perl, Ruby, and JavaScript are precisely the ones that have such great packaging systems and ecosystems.
[1]: Somehow, all my lines aligned perfectly in this comment box at its default size! https://www.dropbox.com/s/slq8x0rqkj7ydzn/aligned.png
Put like that the reliable solution would be a locally replicated SQL database. Perhaps the PostgreSQL or SQLite would be a better target for development than the underlying OS?
Apologies for acting the cynic, but this is the kind of uncertainty that comes when there is constant reinvention, instead of improvement of existing tools.
Unfortunately I didn't have more time to work on it so it is basically unmaintained, but every now and then I still use it, mainly when I need to install software on machines where I don't have administrative privileges, and it works great. I wish I implemented some kind of dependency handling mechanism, it would have made it much more useful.
In particular look at the `conda package` command for an easy way to bundle up any non-conda packages across a homogeneous environment.
virtualenvs are just overrides to environment variables that change where things should look to find the python interpreter, libraries, modules, et al.
docker is much more involved, using LXC to force process isolation, an effective chroot, networking restrictions, etc.
many (most?) folks' current usage of virtualenv would not be portable to docker without substantive work to accomodate the restrictions that docker imposes.
(note: this isn't a critique of docker; those things are features. they're just orthogonal to this)
However, it's not there yet. Armin could bring himself to say this in this post: "It's there, it kinda works, and it's probably worth your time."
Conda is not perfect (it needs more people building conda binaries as well as a few pull requests to add https improvements) --- but I will confidently say "it is there, it does work now, and it is worth your time"
This is especially true with a package repository like exists at http://repo.continuum.io and the many personal repositories at http://binstar.org.
People that have used conda for deployment (especially of complex C-dependencies in the NumPy and PyData stack) are saving time and effort today.
Will love to see wheels getting more attention, really feels like a great direction for python packaging.
I personally use a mixture of setuptools and distutils2, and it's a confusing mix.
How does it convert the non-wheel packages into wheels?
What am I missing?
Can docker easily access the host filesystem or do you have to copy all your data into each docker env?
Can I access my GPU for doing CUDA/OpenGL/OpenCL things from withing docker?
Does docker play nicely with GUI apps?
Calling into a running docker env from an external program (when you are using an app that has python as scripting or plug-in language) doesn't work as far as I know.
The overhead of setting up, tearing down an switching between several Docker envs seems to higher than with tools like virtualenvwrapper or conda.
Now there may be workarounds for these (and all the other) problems but on the surface they don't seem easier than virtualenv.
I was mainly thinking in the context of developing webapps.
I'm interested in this enough that I intend to investigate and answer all your questions, perhaps in a blog post. I don't know enough right now to answer with certainty. However:
- it will currently only work well if you are developing on Linux. MacOS isn't bad but requires vagrant/virtualbox.
- I suspect switching between Docker envs may be faster, easier and cleaner than virtualenv which is why I suggested it.
- There is definitely support for sharing data.
Yes, docker can easily access host's filesystem with volumes.
Yes you can access you GPU. I am using docker for mining.
Yes Docker plays nicely with GUI apps, there are plenty of GUI usecases with docker (docker desktop, firefox, etc..)
You can do anything you want within a python app and docker. There is a docker-py library that gives access to all docker features.
You can take a look at this: https://github.com/jpetazzo/stevedore in order to easily switch envs.
Thankfully I've never ran into this issue.