Django Advice | Hacker News Reader

Django Advice

stevelosh.com

152 pointsstevelosh46 comments

I'm a big fabric fan and use if for server bootstrapping and deployment. I've tried to get into puppet a few times but it always seems overly complex to me. Is puppet useful if I'm only running a handful of servers or is it designed for 10s (or 100s) of servers?

I noticed he used puppet and fabric in this article so it made me interested what puppet it doing better for him than fabric.

Is there a repository of examples? Like just bootstrapping a debian box with nginx/apache/postgres (or mysql)?

Thanks!

I'd say Puppet is useful even if you're running one server, especially when you use it with Vagrant. If you use it for EVERY bit of setup on your server (no manually installing packages!) you can be sure that your development VM and production server are both the same environment.

Puppet is declarative, Fabric is imperative. With Puppet I can say:

    Make sure the directory /var/www/foo exists with permission 755 and owner foo.
    Make sure the nginx vhost "foo.conf" exists in sites-enabled with content X, and this depends on /var/www/foo existing.

With Fabric you have to handle all the edge cases. What if /var/www/foo already exists? What if it's owned by someone else? You have to manually code all the 'mkdir' and 'chmod' commands instead of just saying "this directory should exist like this".

Right now there's no public example repo, but I'm planning on cleaning up and open-sourcing our Puppet/Django/Vagrant skeleton this week so we can use it during the Django Dash.

I'd be interested in seeing this once you're done. Is there some github profile I can watch or somewhere that you plan to put it?

I don't know for puppet but Eric Holscher has a nice tutorial/walkthrough[1] with Chef with a repo of examples.

[1]http://ericholscher.com/blog/2010/nov/8/building-django-app-...

A general tip that has served me well: use tmux ( http://tmux.sourceforge.net/ ). My typical environment consists of a a few vertically stacked panes. Up top is my vim session, below that is a shell. Below that is a horizontal split with runserver on the right and cake watch on the left.

I used tmux for a while and it's great. I've since switched to DVTM + dtach because they're more lightweight and simple than tmux.

I never used 80% of tmux's features, so the simpler tools are better for me. If you're a tmux power user DVTM+dtach might not work for you.

dvtm: http://www.brain-dump.org/projects/dvtm/ dtach: http://dtach.sourceforge.net/

Great post.

But I still don't think Vagrant is that big of a help. I agree that compiling stuff like PIL on OSX can be a bit of a bother, but on production machines it's no bother at all.

And developers usually have to do it once, as, at least in our company, they do not change the projects they work on that often, but rather do them from start to the finish (we find it more productive this way). PIP requirements file with proper package versions is a must of course (Fabric deployment). As is staging server, but speaking from the experience - it's just a precaution, it works well.

Cheers for a part about Django Admin. I think it's the single most annoying, hack-needing piece of Django (as Django is great itself, we should expect perfection).

We are starting our own series of Django tips, mainly related to deployment, performance and scaling - at least one tip weekly. But thanks to you I decided that first we'll write about things we use to make Django admin work with you not against you - http://www.askthepony.com/blog/.

Great article, but I'm still not sold on using Vagrant. Is there a case where it can help if I'm not deploying one app per vm? I often host multiple apps on the same VM slice.

Wow, this is pretty comprehensive. Definitely gotta try django-annoying and automate things more with fabric/puppet/vagrant.

vagrant looks interesting. here are some answers to questions i was going to post here but ended up finding myself via the docs:

- what this adds to just using VB itself is that they provide pre-configured, minimal environments, kept as small as possible (less than 500MB).

- pre-configured means that it includes ssh, VB extensions, sudo, assumes NAT addressing.

- it has some support for multiple vms (DB and web separate, for example).

BUT i still can't work out what environments are available for download. there must be an obvious list somewhere... :o(

BUT i still can't work out what environments are available for download. there must be an obvious list somewhere... :o(

http://www.vagrantbox.es/ has quite an extensive list of user-created Vagrant boxes.

What have you found to be the best way to implement a REST interface? I noticed the @ajax_request shortcut but nothing response code specific. I've been using Django-Piston which has been... challenging at best.

You might like tastypie: https://github.com/toastdriven/django-tastypie

If your rest interface is deeply tied to the Django DB, I'd suggest TastyPie.

Otherwise Django-Piston sadly is the way to go.

I find it to be exactly the opposite. Django Tastypie is in no way tied to the Django Orm.

It's not intrinsically tied to the ORM. Sorry if I gave that impression. However it has many helpful shims and subclasses that interface with the ORM.

I'm looking at ModelResource as my example.

If you aren't working with a NoSQL database or a search engine, or grabbing data from an tertiary resource then the sheer complexity found in TastyPie might not be for you.

Django-Piston doesn't offer many features, but its dead simple in its usage.

Yep, I agree. Tastypie has a steeper learning curve, due to beeing more flexible. When I used Piston, jespern was not actively maintaining it, but using a private branch for bitbucket.

I have used both Piston and Tastypie, and by far prefer Tastypie. Tastypie is actively maintained and super flexible. It gets you working out of the box with the django ORM, but also has the hooks to plug in everything you want.

Very informative article. I always find it interesting to see what experienced Django developers are doing. I will definitely be trying out some these tool, especially Vagrant and Puppet.

Yeah I knew a lot of these tricks. But I've been considering learning chef and I didn't know about Vagrant. Now I know what I'm doing this weekend!

Geez, I've been using Django since 0.92 and there's quite a few things here I never knew about. Good stuff.

I would love to see more details of the fabric+vagrant integration. For instance, you don't show how the prod/dev prefixes in your fabric commands are used to set the vagrant destination host, or the site urls. The vagrant way of things seems to be using "vagrant ssh". I know it sets up port forwarding to accomplish that, but how are you establishing the map of port# -> environment type? Just by convention, within the Vagrantfile?

Good point. Here's the 'dev' task:

    def dev():
        '''Run on the development (i.e. Vagrant) server.'''
        env.env_name = 'development'
        env.site_name = 'MYSITE'
        env.process_name = 'gunicorn-MYSITE'
        env.process_owner = 'gunicorn'
        env.puppet_manifest = 'dev.pp'

        ssh_info = local('vagrant ssh_config', capture=True).splitlines()[1:]
        ssh_info = dict([l.strip().split(' ', 1) for l in ssh_info if l.strip()])

        env.key_filename = ssh_info['IdentityFile']
        env.hosts = ['%(User)s@%(HostName)s:%(Port)s' % ssh_info]

        env.site_path = '/var/www/MYSITE/MYSITE'
        env.venv_path = '/var/www/.virtualenvs/MYSITE'

        env.site_url = 'http://localhost:4565'

In a nutshell: it uses `vagrant ssh_config` to get the SSH info at runtime.

Vagrant looks very interesting. For writing code I prefer Vim but I also use PyCharm on OSX mainly for its excellent debugger and code browser. Does anyone know if there's a way to get PyCharm or perhaps other IDEs (running on OSX) to run the Django dev server through Vagrant?

you can set PyCharm to upload files on save. it'll scp all changed files for you and can run post upload scripts, for example to restart httpd.

Settings -> Deployment

Point to your Vagrant instance IP

Settings -> Deployment -> Options

and select 'upload changed files automatically'

EDIT: sorry I lied re opst upload script, must have confused with something else

Thanks for that tip. What I'm really curious about though is if there's a way to set PyCharm's Python interpreter to something not on the local host (e.g. the Python interpreter on the Vagrant VM). I'd be surprised if this turned out to be possible but thought it wouldn't hurt to ask.

Edit: I just found out about the remote debugging feature in PyCharm which may actually do what I want:

http://blogs.jetbrains.com/pycharm/2010/12/python-remote-deb...

I'm not really a Django user, but I was interested in the Vim section of the post. I work with Rails and my personal approach to setting framework-specific commands, mappings, etc. is by using the proj plugin: http://www.vim.org/scripts/script.php?script_id=2719. It's fairly simple, it just sources a project-specific vimfile when you tell it to. I keep a ton of settings in "~/.vim/projects/rails.vim" and I place a "runtime projects/rails.vim" in the project file. This kind of a workflow might give you more freedom in customizing your vim for Django projects, but it really depends on whether you can get used to that. Just a suggestion :).

Great Great article. Just started using Django and this basically covers every single pain point that I've had thus far.

Thank you so much.

It's a little bit odd, that the first tip is using Vagrant, and almost all the other tips are hacks to make up for that decision.

Uh, what?

    Topic                                       Vagrant-Related?
    ------------------------------------------------------------
    Vagrant                                     Y
        Why Vagrant?                            Y
        Using Fabric to Stay Fast and...        Y
    Wrangling Databases with South              N
        Useful Fabric Tasks                     Y
    Watching for Changes                        N
        Using the Werkzeug Debugger with...     N
        Pulling Uploads                         N
        Preventing Accidents                    N
    Working with Third-Party Apps               N
        Installing Apps from Repositories       N
        Mirroring Repositories                  N
        Using BCVI to Edit Files                Y
    Improving the Admin Interface               N
        Enter Grappelli                         N
        An Ugly Hack to Show Usable Foreign...  N
    Using Django-Annoying                       N
        The render_to Decorator                 N
        The ajax_request Decorator              N
    Templating Tricks                           N
        Null Checks and Fallbacks               N
        Manipulating Query Strings              N
        Satisfying Your Designer with Typogrify N
    The Flat Page Trainwreck                    N
    Editing with Vim                            N
        Vim for Django                          N
        Filetype Mappings                       N
        Python Sanity Checking                  N
        Javascript Sanity Checking and Folding  N
        Django Autocommands                     N
    Conclusion                                  N

5/32 sections are Vagrant-related. 5/32 != almost all.

EDIT: My bad, there are only 31 sections total. Still, 5/31 is not almost all.

You are right. My bad. My point though, was that I think the cost in terms of complexity of using Vagrant for development is too high. The goal of having your dev environment match your prod. env. is not the right approach in my opinion. This the role of the staging environment.

The goal should be to keep environments as similar as possible, which vagrant helps to accomplish. Just because you can develop locally on your Mac before pushing to a QA/Dev/staging environment doesn't mean you should.

Obviously your Dev environment isn't going to be able to match every aspect of your production environment, but keeping the same platform builds across environments is something that all teams should be aiming for.

That might be your goals. My dev environment should be as painless and fast as possible. This is what I spend all day using.

It's what I spend all day using as well. The dev environments I've either implemented or been a part of (web centric envs) have all used local folders mounted to VMs that provide that painless and fast requirement you speak of while removing any pain points associated with varying platforms. You dont need to sacrifice the former to achieve the latter.

And wat hasn't been touched upon here, but what's maybe even more important - at least for my team - : The same platform builds at everyone in the team.

The goal of having your dev environment match your prod. env. is not the right approach in my opinion.

Indeed. It's not just the right approach, it's the only approach for non-trivial applications.

Sorry for being a bit snarky, but I've seen the alternatives crumble under stress way too often.

Doesn't this just move the pain of environment mismatch from dev->prod to dev->staging?

If you use something OS-specific on your dev machine you'll need to fix it at SOME point, right?

Yes, it does. I just think imposing prod. setup testing on your dev env., is the wrong place to do it. For many django apps, I've been able to use testserver, staticserve, dummycaching, synchrounous Celery tasks, etc for development. Just configs that works on checkout. If I need a special database, or something with a non standard interface I add only that to my setup.

Excellent article!

Would you detail more on puppet please? "Run Puppet to initialize the VPS."

I find that the warnings and hints from http://jshint.com/ are better than what jslint offers.

I'm sure jshint could be integrated into Syntastic pretty easily.

Steve, when you say that you mirror 3rd party apps, does that mean you don't install anything from PyPI (like South)?

I love your articles, but this one hits the sweet spot. Thanks for another great article.

Unrelated: His blog looks very nice, what is powering it?

It's a static site rendered with Hyde.

http://stevelosh.com/blog/2010/01/moving-from-django-to-hyde...

Great gems in this article. Thanks for the great info.

Reply on news.ycombinator.com