Comment by IgorPartola | Hacker News Reader

Comment by IgorPartola | Hacker News Reader

Parent Full thread

My two cents: I am a developer + ops person and deploy Python apps all the time. Typically they are Django and Tornado services. On top we also have a lot of daemons and a ton of library code.

I agree with the OP on most points but do not on a few. First DO use packages that come with the OS. The OP says that you should not have the distro maintainers dictating what you use. I say, use what is widely available. It takes the headache out of a lot of your deployments. If you are looking for a library that converts foo to bar look in your distro's repos before going on GitHub. Your sysadmin will thank you.

Second, DO NOT use virtualenv. It fixes the symptoms (Python's packaging system has many shortcomings such as inability to uninstall recursively, poor dependency management, lack of pre and post install scripts, etc.), but not the problem. Instead, use distro-appropriate packages. Integrate your app into the system. This way you will never end up running a daemon inside a screen session, etc. You also get the ability to very nicely manage dependencies and a clean separation between code and configuration.

Lastly, DO use apache + mod_wsgi. It is fast, stable, widely supported and well tested. If apache feels like a ball of mud, take the time to understand how to cut it down to a minimum and configure it properly.

When it comes to infrastructure, making boring choices leads to predictable performance and less headaches more often than not (at least in my experience).

"First DO use packages that come with the OS."

I'd go middle ground, and start here, but consider a self-built package where necessary. It depends in part on the focus of your distro.

virtualenv. What problem does it solve? Different python version/environments? Wouldn't that be better solved with another (virtual) server? I understand if an extra $20/month is an issue, but otherwise ...

I have just been educated by reading more of this thread. I can see an obvious need for one virtualenv, so that you can separate your service and its needs from the system python and its needs. Beyond that my inclination would be to go more servers rather than more virtualenvs, but circumstances vary and my experience is narrow.

Not every Python application is a big web app. We have systems that run several smaller Python apps. Python is everywhere.

> First DO use packages that come with the OS. The OP says that you should not have the distro maintainers dictating what you use. I say, use what is widely available. It takes the headache out of a lot of your deployments. If you are looking for a library that converts foo to bar look in your distro's repos before going on GitHub. Your sysadmin will thank you.

The sysadmin will have no part in the game if you use packaged virtualenvs. OTOH developer time is expensive. Do you really want to pay your developers to implement functionality that a more recent version of a package has already implemented? A good example is IPv6 support in Twisted. It’s getting implemented right now but I guess (and hope) that I’ll need it sooner than it lands in major distros (please no “lol ipv6” here, it’s just an example and the support is growing).

> Second, DO NOT use virtualenv. It fixes the symptoms (Python's packaging system has many shortcomings such as inability to uninstall recursively, poor dependency management, lack of pre and post install scripts, etc.), but not the problem. Instead, use distro-appropriate packages.

I’m not sure what your problem is, but mine is that I don’t want to develop against a moving target and need to run apps with contradicting dependencies on the same host.

That’s how I started using virtualenv years ago BTW, I’m not talking ivory tower here.

> Integrate your app into the system.

Yes. And I prefer supervisor for that. If your prefer rc.d scripts, be my guest.

> You also get the ability to very nicely manage dependencies and a clean separation between code and configuration.

I don’t get this one TBH.

> Lastly, DO use apache + mod_wsgi. It is fast, stable, widely supported and well tested.

And nginx + uwsgi/gunicorn aren’t?

> If apache feels like a ball of mud, take the time to understand how to cut it down to a minimum and configure it properly.

I know Apache pretty well, because we’re running thousands of customers on them. I’ve already written modules for it and been more than once in it‘s guts. And my impression is not a good one.

My point was to look around before you settle. If you think Apache is da best, knock yourself out. However stuff I see daily on IRC lets me think that it isn’t very unproblematic.

I’m not going to start a “vi vs. emacs”-style holy war here. That’s why I wrote “shop around before you settle” and not ”don’t ever use Apache”.

> When it comes to infrastructure, making boring choices leads to predictable performance and less headaches more often than not (at least in my experience).

Absolutely. nginx is way past the “new and hacky” state though.

Sysadmin/Ops should have a part in what gets deployed. It is their job. My point is that if you have a choice between Python 2.6 that comes with your distro and Python 2.7 that doesn't, go with 2.6. The cost is minimal. Same for various libraries (PIL, NumPy, etc.) I currently have to deploy Django-Piston directly from a master branch on GitHub due to some terrible decisions made by developers who never gave deployment a second thought. Avoid this. If you would save more hours by using a newer library, minus time it takes to set it up and maintain it in production, go for it, but don't discount the cost on the Ops side.

I don't have a problem with virtualenv: it's a fine development tool, but it is not what I would use in production. If you want separate clean environments for each app, use KVM or Xen and give it a whole server.

supervisord is a fine solution. I just prefer that my processes look exactly like system processes. Upstart and rc.d are fantastic and do everything I need well.

As for separation of code and configuration, I simply mean that in my case I use Debian packages to deploy all of our software. This means that each package must be generic enough that it is deployable so long as its dependencies are satisfied. Thus your config files are mostly external to your packages. Then you can easily use Puppet or some such to deploy code. One other reason to use native distro packages: Puppet does not play well with pip/easy_install, etc.

>> Lastly, DO use apache + mod_wsgi. It is fast, stable, widely supported and well tested.

> And nginx + uwsgi/gunicorn aren’t?

I didn't say that. I am simply stating that saying "don't use apache" is wrong. Do use it. You can also use nginx + uwsgi/gunicorn if you want to, but apache is by no means a bad choice. It's got 25 years of use and nobody that uses it seriously is complaining.

> Absolutely. nginx is way past the “new and hacky” state though.

It is not, and I never said so. I use nginx + apache, where nginx is a reverse proxy. In fact nginx is my top choice for front-end server setups. I am saying, don't deploy things directly out of GitHub. Go with slightly older, more tested stuff. It'll be a bigger payoff in the end.

Overall, I think we are saying the same thing, with slightly different tools we normally reach for. I am just trying to throw a different perspective out there and a different way to do things. Thanks for the detailed reply.

> I currently have to deploy Django-Piston directly from a master branch on GitHub due to some terrible decisions made by developers who never gave deployment a second thought. Avoid this.

JFTR, there is a pretty good solution for this (because in the real world, you can’t always avoid that): a custom pypi server.

You take a git version you know that works do a `python setup.py sdist` und push it to a private repo. We have to do stuff like this for Sybase drivers for example which are open source but not in PyPI (or any distribution). It saves so much pain.

> As for separation of code and configuration, I simply mean that in my case I use Debian packages to deploy all of our software.

Well we do the same but the Debian packages contain code _and_ the virtualenv.

> Overall, I think we are saying the same thing, with slightly different tools we normally reach for.

Mostly yes. Just wanted to add the pypi tip as it hasn’t been mentioned yet in this thread.

Nive tip about the private PyPI server. I might use it in the future if we end up in the same situation again.

Also, one more nice thing about using distro packages for your own deploys: if you ever need to mix Python with anything else it is a godsend. For example, at one point I had a network sniffer/userland packet forwarder that was written in C deployed alongside our Python daemons. Debian packages do not care what you put in them: C, Python, JavaScript or pure data.

Reply on news.ycombinator.com