Musings about Debian and Python
notes.pault.ag
notes.pault.ag
It seems like this is where everyone settled and is a good solution:
Virtualenv + pip for development
system wide (apt, dpkg) python etc for system tools and packages.
This makes sense to me. Keep your personal code sandboxed from your system tools.
As an aside, I know there are quite a few tools and programs in both Debian and Ubuntu written in Python. I wonder if we'll look back in 5-10 years and realize/think this was a mistake. Particularly for UI code. I love me some Python, but I cringe every time I have to interact with a Python GUI application.
In the end you find yourself installing as much as possible with pip, and resort to apt for the really heavy stuff like numpy.
Here's some documentation on Django (as an example): https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/...
I think a more interesting question is not around the python side of things, but the infrastructure. Do you compile all your infra tools from source or do you use the system packages? Like, if you want to use RabbitMQ, if debian/Ubuntu provides it, would you use that or grab the source?
I personally prefer to use the debian/ubuntu ones in these cases unless they are horribly outdated, which isn't generally the case. I can understand, however, if someone thinks this is a terrible idea.
How are Python GUI applications worse than C GUI applications?
If you mean a scenario, where you click a button and the app freezes for a few seconds then this is an architecture problem. Whateven happens should happen asynchronously to the GUI. This can be right and wrong in Python and in C.
As for the actual work being done behind the GUI, that's a different matter.
My main issue with them is the common symptom where both "hang" or "freeze" for a few seconds. Visually this can look like the window gets grayed out for a bit (common in Jockey) or it it is just stuck for a few seconds with no clear indication anything is happening (more common in Software Center).
EDIT: I should note that I've previously written a few PyGTK apps and have recently started writing Ubuntu Touch apps via their Ubuntu SDK (Qt/QML). I've found that Qt/QML/Ubuntu SDK is a much better experience in both writing the app and the responsiveness of the resulting application. And I'm by no means proficient in QML, I'm just starting out.
mostly I think we shall cringe that we thought complex operations could benefit from a GUI - it's either a matter of education or curation. either we teach you to use the complicated .conf file or we make sure that the setup and default are so intelligent it's like a mac
Does PIL need to be compiled when you use it via Pip (I'm not familiar with PIL, tbh)? Can you just install those via pip and be done with it? So far using virtualenv + pip, I've not encountered a system library vs project dependency clash...but maybe I've just been lucky.
It seems like a better solution would be to declare a common interface, both to users and to tools, for each language's package system. At least that way there's some guarantee (or at least a strong hint to the author!) that any given tool would support some nice-to-have extra functionality.
A use case for me is that I see a really neat thing Foo written in an unfamiliar language. Install instructions go something like: "Just run 'simple_make foo' to try it out!" or perhaps "Just run 'curl http://get-foo-bar.baz/rootme | sh -'". Okay, now perhaps it's installed. It might even be installed where I want but it probably won't be. But what happens if I need to update it? How can I keep all of these pet projects up to date? I have to go and learn easy_install, gem, leiningen, sbt, maven, npm, go, quicklisp, cabal, etc.
I mean, I'm not too lazy to read up on each tool but it adds up quick and not every tool supports an Autoconf-like --prefix or apt-get {update,upgrade}.
This way I can have my sysadmin hat on while setting up the server and depend on debian/ubuntu to handle security upgrades and generally create a consistent system. Then I can put my devops hat on and use capistrano and bundler to manage the security/dependencies of my own code.
But I see where this breaks down. If the base stack is moving at a much faster pace than the distributions (e.g., right now the version of passenger in Ubuntu LTS is incredibly old) it's attractive to just ignore the system packages and install everything from original sources (e.g., install ruby from source with rbenv). But doing that is just throwing away the integration work the distribution has done. I'd much rather include some extra repositories to get updated versions that integrate through apt for the few things that I care to upgrade faster than Ubuntu LTS allows me. Right now that's puppet, passenger and a few more.
This made Vagrant unusable for anyone not deploying a ruby-based application stack, unless sysadmins accepted that they were going to have to deal with the extra work associated with multiple vendors and update mechanisms for each language on the system. So those of us using other stacks could use Vagrant, but only if we adopted Ruby's app-level packaging system. See https://news.ycombinator.com/item?id=2059964 as an example of life being made difficult for distro packagers by the Ruby community. Since Rubyists tend to insist that distro packages are bad, so this is how it tends to stay.
Thankfully the packaging of Vagrant and dependencies has now happened (finally; after years), and I can "apt-get install vagrant" now. But I hope this illustrates the problem.
In the common use case, there is a dichotomy between distro and app-level dependencies, packaging and development. But in the grand scheme of things we all want to share our work across communities, and this dichotomy is a barrier to this ideal. Dependencies don't just go in one direction with language module repositories like pypi and rubygems at the top. Sometimes we want parts of the system to depend on them, too.
1. When in fast development get it from gems (my case with passenger now)
2. When it starts getting mature (or as soon as you have resources to do enough QA) build a deb repository (puppetlabs does this for puppet)
3. When it becomes stable and boring just use what's in the system (apache and thousands of other things)
Moving from step 1 to 2 is probably not natural for a lot of ruby (and python to a lesser extent) developers because they see their software in the context of apps that are "things you deploy to a host" not "things that are part of the base system".
Things are trending towards user programs that focus exclusively on high-level problem solving and delegate all building blocks to 3rd party modules, and these modules may in turn do the same.
I'm a casual python user and I find virtualenv to be very helpful.
Copyright (c) Paul R. Tagliamonte, 2013 under the terms and conditions
of the MIT/Expat license.
This blog is powered by http://getpelican.com/
-->D'aw, thanks so much. I really appreciate that. I identify as a OS-level hacker, and very focused on code, not design, so it means a lot to hear such kind words when I put a bit of effort into it.
Thank you!
(And when it doesn't, it's because I did something deep within dpkg/apt I should not have.)
I can't say I've always had the same experience with other distributions which is the reason why I moved to Debian in the first place. Truthfully, I can't ramble any specific scenarios off from the top of my head.
Any time I've had an issue with a Debian software package, the bug threads have always been constructive with proponents for both sides explaining why it should be one way over the other. Eventually, the best decision is made (even if I personally disagree).
However, the sacrifice for this stability is the fact packages can become a bit 'stale' when it comes to new versions. I don't mind sacrifices like this. And if you need newer stuff, that's why backports exist.
If you want the latest and the greatest use pip. But for the love of all things, couple your pip usage with a virtual environment. Hell, even if you aren't using pip get in the habit of using a venv.
My only bad experience with virtualenvs was a recent Python security update (related to /dev/random). The system libraries changed, the Python executable in the venv did not, sadness ensued. Even then, once I figured out the issue, it was a quick fix. Just re-init the virtualenv, and it fixed it for me. No need to move code around.
In short, if I want a stable system I don't need to babysit, I go with Debian. I trust the people maintaining the packages. I use virtualenvs for 90% of my Python development, and use pip inside of those venvs.
Hasn't failed me, yet. Probably shouldn't jinx myself...
Installing applications as containerized packages (e.g., Gobolinux, OS X .app) should be the norm. Both apt and pip get it wrong because they don't solve any of the above problems by themselves (hopefully you can use pip+virtualenv to achieve containerization, and more recently, LXC).
You can have both, where it makes sense, instead of everything being shared by default (e.g., you can provide all your software a base OpenSSL version). See how FreeBSD solves that with jails and sharedfs.
> or centralized collections of trusted software?
All you need for that is app signing (e.g., App Store). You can still update apps independently, there's no dependency management.
I have been in this Python mess far too long. Now I just make a .war file or tell Go to compile for my target platform. Both result in single files that can be deployed without post-processing, installing/compiling dependencies or whatever ... it simplifies things for everyone. Developers, CI, packagers, ops, security folks.
JFTR: the absolutely same is possible in Python.
In Arch you would wrap this in a PKGBUILD script to generate a Pacman package. You can find PKGBUILD's for most things in the AUR.
Single artifact does not mean 'your code'. It means 'the whole damn project including all dependencies recursively'.
Think myapp-1.0.war that contains all dependent JAR files. Drop it in a container and it runs. Or your Go compiled tool that is one single static executable that has no reference to external code or libraries. Run it and it works.
I know how I can package my own Python code. I am not too worried about that. I am however really tired of packaging the whole thing every time. And then dealing with stupid things like python packages that need a C compiler to build. Or specific patches to dependent C libraries. Or specific versions of things that some distro might not have. Or dependencies for which I need a newer version than the OS provides. Or .. sigh .. there are so many stupid gotchas that are complete time wasters to deal with.
Source code is very compatible across many architectures and operating systems but needs compilation, configuration, and dependencies. Binaries are highly compatible with a single platform and work out of the box. However they are not very compatible with any other platform.
The downside is that if you want any feature not provided by the JVM's abstraction layer, at the very least you are in the exact same boat as a python developer.
It was a wrapper that calls your program, figures out the actual runtime dependencies and essentially creates an image of the all necessary components and bundles it up.
I really wish I could remember its name, but it was designed for science researchers to help them publish reproducible code.
I'd also add that native packages tend to be far more repeatable when you have to deploy and update a lot of systems in any environment.
Note that with tools like fpm or other similar quick and dirty package generation tools, in general don't have to stick to the upstream system's development guidelines, but you get the benefits of using the native package tools.
anyway, side note: switched from nm-applet to wicd-client and haven't looked back.
Thanks! Pairing the fonts took a little while, and fiddling with it took a bit of time - so thank you! :)
Yeah. The goal here isn't to stir the pot in the middle of a massive flame-war, but it's the sort of thing where I see a part of one of these viewpoints now and again.
The hope here is that this can be something that will be given out when someone expresses a viewpoint that starts to stir up this same-old flame again, as well as make people think a bit more seriously about issues regarding integration between language (and native) package managers (on both sides).