Django plugins you shouldn't start without
blog.hndigest.com
blog.hndigest.com
I read this post, and I see bandaids. I see bloat, and I see warts. I don't want to build on a codebase that needs 5 extensions out of the box to be sane.
Granted, I know this is not the real story with Django. It is a rather lovely, if rather rigid and stubborn framework (at least it was when I was writing with in circa 1.3). And it is powerful.
But, and this is my point, it seems to me that there is almost no reason I would use Django over something like flask for anything but the most basic of apps. Any time you need to do "serious" work — beyond cookie-cutter stuff — you're going to need to seriously extend Django (or any other web framework for that matter, in my experience/humble opinion). And then it gets ugly, and you're better off building from the ground up on a minimal codebase than trying to contort an existing one to do what you need it to.
Actually, most of my thoughts have been summed up 5 years ago by one of Flikr's lead architects (iirc) at DjangoCon '08 in a brilliant (and hilarious) talk: http://www.youtube.com/watch?v=i6Fr65PFqfk
On the list, the only one I use with any regularity is South, which I do consider must-have. That said, it seems that the Django core team agrees, as it's being (or maybe is even finished) pulled into Django core.
After that, I personally add allauth, storages, and s3-folder-storages as well, but that's a personal preference thing and there are very good reasons to use alternatives to each of those components.
I've been working with django full time for 3 years and the only one I use is South.
Implementations that stand out, like south, may get incorporated. Especially when they're valuable to the core of Django; all may not agree, but migrations (and class based views) should have been in there from the start.
Others depend on your needs. Don't force me into using TinyMCE as a rich text editor, I'll regretfully make that decision myself. I want to choose the best available package for my CDN host. I'm switching thumbnail processing engines because mine hasn't been updated in over two years.
Importing all of my models into the shell is best left as an explicit modification for those that want it. If it was a good idea, why wouldn't Django do it by default?
Reasons to use Django over other, lighter frameworks:
* The ORM
* Forms – The ORM
* contrib.admin
* Authentication
* Extendable command line interface
* A standardized, battle-tested system for organizing your code.
* Lightweight view definitions. Views are functions from Request to Response in Django, which makes them very easy to define. Django, Flask, and Pyramid all get this right; Python, web.py, and webapp2 get it wrong. Also, Request/Response objects are very sanely designed.
* Very full-featured templates & helper functions. If you're going to run into a problem, chances are that Django has already solved it. Lighter-weight microframeworks like Flask don't have this property.
* The ability to swap out components you don't need, and to bring in other components as necessary. For example, none of the Django apps I've used since joining Google have used an RDBMS - they all use RPCs as the backend. Oftentimes they need to run proprietary Google code in a controller. This is all not a problem - it's just code, passed between functions as necessary. (Granted, many of them use the Django-nonrel fork.)
The thing that irritates me about django is request is passed as an argument, and if you need if further down the chain, you have to pass it around. I want the request object to be available throughout the request cycle without it being explicitly passed. I like the flask model of importing request when you need it. In Django, I can have a middleware to handle it for me:
from threading import local
_locals = local()
class ThreadLocalMiddleware(object):
def process_request(self, request):
_locals.request = request
@classmethod
def request(cls):
return _locals.request
@classmethod
def locals(cls):
return _locals
> Very full-featured templates & helper functions.
Django templates are nice; Jinja2 is nicer:-)
> Lighter-weight microframeworks like Flask don't have this property.
I think it's not just about Flask being light weight. It's more a function of Django existing for longer and being more popular. Granted that Django itself includes lot of batteries which Flask doesn't, but the "somebody has already solved it" is mostly either an extension, or a forum/stackoverflow solution.
I'm writing a retrospective now about things I've learned in almost 5 years of working on Google Search, and the entry I just finished was "In almost every case where we allowed non-local effects, it has bitten us. Badly." The cost is in understandability, and is usually not apparent when the choice to make it a global is made. But 2 years later, things inevitably grind to a halt because bugs are being shipped and nobody has a handle on the code any more.
I've cursed out having to explicitly pass arguments around many, many times before (not just with Google work; when I was doing Haskell stuff I used to curse how inconvenient it was that I needed to explicitly pass around stuff, and often wrote a state monad to hide it from myself). But I've found that over time, those apps never hit the brick wall where I'm like "I cannot deal with this anymore"; they remain maintainable, if ugly, indefinitely.
But the request object in flask is not global. If the guarantees break(request locals leak to other requests), it will be a problem. I never have had a request local leak in flask. On the other hand, I have ran into cases where I need the request object in django signals. A simple task of getting the full uri requires either a request object(I would say that's bad api but that's how it is in django) or configuring sites. Bad api or not, if I am in a request, I should have access to it, and Django makes it hard. Dress it up if you want to comply with "no globals" - RequestManager.get_current_request(). Though I don't see how that's any better than `from flask import request` when you know it does the equivalent RequestManager.get_current_request() internally.
So you're saying that if I don't use multithreading my global variables are not, in fact, global?
The request object is context local. If there are multiple concurrent requests, all request have different request objects. I don't know what definition of global variable you have, but this is not it.
The request object is a threadlocal (or, if greenlet is installed, a greenlet-local).
Which is exactly the same as a global variable in a single-threaded program.
request objects aren't stored in python's thread local storage. If they were, a thread re-use will leak context local. Werkzeug uses the thread/greenlet id as key in its storage and cleans up when the request completes.
> Which is exactly the same as a global variable in a single-threaded program.
In a single threaded server, the context locals won't leak from one request to another. If they were global, they would. I don't see how request object has anything to do with global variables.
No more so than werkzeug since it works the same way.
> Werkzeug uses the thread/greenlet id as key in its storage
Which is exactly how Python's threadlocals work.
> and cleans up when the request completes.
You do realize that nothing stops you from "cleaning up" your threadlocal "when the request completes" right? Yes werkzeug's locals and localmanagers conveniently provide hooks for that cleanup, but that's what it is: convenient. The primary purpose and the one explicitly mentioned in werkzeug's own documentation is having a single system for both greenlets and threads (the stdlib's threading.local obviously doesn't work with greenlets, just as werkzeug's locals don't work with non-greenlet coroutines)
> In a single threaded server, the context locals won't leak from one request to another. If they were global, they would.
Not if you cleaned them up after the request has executed. And apparently that's all you need to make a global variable not-global-after-all.
> I don't see how request object has anything to do with global variables.
It's an object available from all scopes. That's pretty much the definition of a global variable.
I don't know if you are being intentionally dense. Werkzeug doesn't work the same way. Thread locals restrict themselves to thread lifecycle; werkzeug locals restrict themselves to request lifecycle. They are 2 very different things.
> Which is exactly how Python's threadlocals work.
Except for different lifecycle.
> You do realize that nothing stops you from "cleaning up" your threadlocal "when the request completes" right?
That "cleaning up" makes it request local, not thread local. They are global for the duration of the request and one request scope is separate from another request which makes them request local(how it is done is immaterial). Spare me the definition of global. There is no point in continuing this discussion any further since I don't seem to have any epiphany from your insights; and doesn't seem like you are going to have one either.
Singletons are no better - if you dress it up as RequestManager.get_current_request(), it's just as bad. If you don't understand why something is a bad idea, complying with the letter of the law without complying with the spirit doesn't really get you much.
I think this is the way in which my programming style has changed the most over the last 10 years. I used to want my programs to do everything and have access to everything. So I'd write PHP scripts that would do "SELECT * FROM table INNER JOIN table2 WHERE ..." and pass around the whole result set to every function and every template in the app in case they wanted to access some data in it. That way, I only had to change the DB schema and the template when requirements changed. (Anyone who's thinking of doing this, you're introducing massive data leakage and security issues into your app.)
Now I want my programs to do as little as possible and have access to as little as possible. Usually I end up pulling out the values I want inside the view function and calling any helpers using only those values, so I don't pass the request around anyway. The actual work of the program happens on strings, or ints, or structs, or other values specific to the domain.
I think what changed, for me, were a few things. One is that I got better at visualizing the operation of a large program as a whole, so instead of thinking "Hmm, I need a request here, how can I get access to one", I started thinking in terms of "This is how data enters the system, this is what we need for each computation it performs, and this is how we combine those results to render a page - how can I change it so the system still hangs together but does what we want now?" Another was that I kept running into instances where I could almost re-use a function, but was blocked because of hidden dependencies that it didn't really need. A third was testability - I learned how crucially important it is to be able to test parts of the system in isolation, and that's really hard when you have to setup the whole system context to run any function. A fourth was that I worked a bunch with both systems and found that adding a parameter to each function along a call path is easy (but tedious - if only there were a refactoring tool to do this for us), while disentangling a tightly-coupled function that depends upon a lot of implicit state is quite challenging.
I don't know. Most of them seem like a non issue to me. Shouldn't the api cover that testability(of request)? At least, it does in Flask. Re-using a function which uses request won't be any different from re-using a function which uses datetime. I do get your point about difficulty of analyzing Python owing to dynamic typing and monkey patching, but I couldn't connect it to context local. As for cross referencing, you just treat request as you treat datetime.
> Singletons are no better - if you dress it up as RequestManager.get_current_request(), it's just as bad. If you don't understand why something is a bad idea, complying with the letter of the law without complying with the spirit doesn't really get you much.
I won't do that. That is just a tongue in cheek comment to appease "death to globals" police.
> Now I want my programs to do as little as possible and have access to as little as possible.
I don't disagree with that but assuming that the request object is available for the request lifecycle is pretty reasonable.
> Usually I end up pulling out the values I want inside the view function and calling any helpers using only those values, so I don't pass the request around anyway.
You don't pass around request in a general workflow(why would you?). But consider this https://github.com/tschellenbach/Django-facebook. This does a post save signal when a new user registers. I need to send a welcome mail to the user asking him to activate his account. I need request.build_absolute_uri to construct the uri, and guess what, I don't have request in the callback and I don't have a way to obtain it. Now there can be other ways I can solve it - configure sites and generate url using sites; simply have the base url in settings and construct the url by doing a urljoin; may be in some alternate universe, django's reverse generates external url. But the thing is, I have needed access to request object on more than a few occasions. Granted if the callback passed me the request object, it won't have been an issue but it doesn't and I had to resort to thread local middleware to save the request object.
> A fourth was that I worked a bunch with both systems and found that adding a parameter to each function along a call path is easy
The issue is apis which you haven't wrote. I can always change the packages myself, but I don't want to sync up with upstream for something I see as a small change.
I want to build on a platform where far more than 5 extensions are needed to be sane -- I want the vast majority of my "core" functionality to be provided by an extension.
Far too often "Batteries included" means "Batteries soldered in with a thick layer of epoxy on top".
Django is not perfect, and will never be. It's now a fairly large codebase, and though there are a bunch of committers, people often feel it's easier to write their own library than push things into core.
Separate from that, some things really don't belong in core - they aren't generally applicable, even if they are valid use cases. We prefer to address minority use cases with extensibility. Rather than saying "your use case is not valid", or saying "give us your kitchen sink", we say: "OK, that's a nice sink - why can't you currently use it? What if we add this valve here?"
In the best (and, I think, fairly common) case, people write libraries, the community finds it generally useful, and the best of that library gets incorporated. We've gained authentication, cache, and migration improvements that way.
There are warts, but the proliferation of libraries are not bloat - they are often a resistance to bloat. Search github for django + jsonfield, and you will see quite a lot of opinions about what it should be. If we'd added the first one that seemed decent, we would have been adding a wart for many people.
I have used all of the recommended libraries before, and while they are nice, I wouldn't call them essential. It works fine without them. Some of the -extensions are very nice, but they rely on 3rd-party libraries, and django has had a pretty conservative dependency policy - python, your database driver, and you're generally ready to go. I think that's a win -- you can add additional dependencies if you want them. Of the libraries, only South is a thing I have used on most projects, and as noted elsewhere in the thread, that's being incorporated into core. As for grappelli - I think the admin needs an update, and we've worked off and on on that quite a bit. Grappelli imagines one direction for improvement, but I don't think it's what core should be -- and I suspect the Grappelli maintainers would agree.
With all that said -- I consider flask roughly comparable to sinatra -- it is smaller, but what parts there are make additional, not fewer assumptions about what you're building. SQLA is a way more powerful ORM than Django's, but I don't often feel limited by Django's. And when I do, there's .raw or .execute. I find SQLA to be overkill for the sorts of queries you'd do in a web request.
If you're saying: you'd rather learn best-of-breed libraries and compose them yourself -- that's a valid point of view, but it's an opinion, not a correct answer for all sites or situations. Since reuse-in-the-large is still an unsolved problem, a thing that falls out of using different libraries and composing yourself is that you lose the community's bounty of components that work with your particular system composition. As for me: I recognize that django is larger and harder to pick up than flask, but I suspect that if you instead (and, IMHO, more properly) compared picking up django with picking up flask, SQLA, jinja, flask-admin, etc -- then they compare favorably in terms of uptake -- and django still gains from the community's many django-* libraries.
If you are doing anything more than a simple CRUD, and heck, even if that's all, you should at least peek at how many queries are happening in views and how page render timing is broken down.
https://github.com/django-debug-toolbar/django-debug-toolbar
https://github.com/skorokithakis/django-annoying
That, and shortuuid:
https://pypi.python.org/pypi/shortuuid/
Disclosure: I maintain one and wrote the other, but they and South are the three things I use for every project.
Something like str(uuid.uuid4()).split("-")[4]
I'd like to say that I then do a duplicate check, but I don't think I've ever had a collision, so I'm not actually 100% true that I do.
Best case, you've saved me a database query, worst case, you've saved me a potentially painful Exception.
Danke.
It'd be nice if you were clear about what versions of django and python are supported.
From django-annoying's feature list:
* render_to - seems to be roughly equivalent to django.shortcuts.render
* signals decorator - now in core[1] (since 1.3)
* ajax_response - seems useful, though might make sense to check request.is_ajax[2]
* autostrip - hmm, seems oddly edge case-y, I definitely wouldn't apply it to all my forms
* get_object_or_None - nifty
* AutoOneToOneField - assumes the related model is valid with only the FK filled in, which seems fairly unlikely in many OneToOne cases. I'm not sure if this was proposed to core, but I'd think it would need some sort of default= fillin feature to be more generally useful.
* JSONField - soo many competing and variously featureful implementations. This has been discussed a few times; it seems fairly easy to implement and hard to gain consensus. I wish we could get it in core, but the juice doesn't seem worth the squeeze.
* get_config - I don't think I understand the purpose of this, unless you're just trying to be more intention-revealing than getattr(settings, ...)
* HttpResponseReload - nifty
* StaticServer - I don't think I understand why you wouldn't want to config urls.py instead.
[2] https://docs.djangoproject.com/en/dev/ref/request-response/#...
As JshWright said, the appeal of render_to is that it's a decorator, it does clean up views by a lot.
The signals decorator is now obsolete, I also find ajax_response very handy (when I'm not using CBVs), autostrip is a bit edge-casey, I agree.
get_object_or_None is pretty much 1.6's .first(), but there was no alternative until 1.6. AutOneToOne/JSON fields are there because of convenience, get_config I have no idea about (I didn't write these), StaticServer, also no idea.
The thing is that this library has been out for years, and most of these things didn't exist in core when it came out, so it was really handy in its time. I've been using it for quite a few years, and that's also why I took up maintenance of it.
In the long run, I'd like to see the things that make sense in it integrated into core, especially the @render decorator.
Lean is beautiful, people. Carefully assess every dependency you add, as each one has the potential to become a burden.
Not to mention that, if the library broke, then the thing you would have written would have probably broken as well, and you wouldn't have the benefit of a thousand other users who might have had (and fixed) the problem before you. This just smells of FUD to me, as it rarely is a problem to me in practice.
Depends what you mean by "need". Often when you need a piece of functionality, what you actually "need" is a small subset of what a third party app is wanting to provide, and I've often found simply using out-of-the-box apps for this adds a fairly huge amount of complexity that is just not needed.
My favourite case in point would be adding very simple blog functionality to something - which is fairly quick & trivial in django, but out-of-the-box blog solutions promote themselves with tempting-features-that-you-dont-really-need that inadvertently add a lot of complexity and before you know it you're a few releases down the line and you realize that the author doesn't really know how to use South properly and your db ends up in a slight migration-state-limbo forevermore.
Though note that I am _not_ saying don't use external dependencies - I use several of those listed in the article (I am a big fan of django-reversion) - but remember to look at everything with a critical eye before inclusion.
> This just smells of FUD to me
It's all very well and good saying that but a stack of previous bad experiences doesn't smell like FUD when you're on this end of them.
Something like shortuuid, on the other hand, is pretty much a slot-in.
3rd party libraries upgraded painlessly in 90% of cases aside from those few that had been abandoned. And usually they were abandoned because the community had coalesced around a much better library.
So 3rd party code? Change a line in requirements.txt and bam. My own code? Had to upgrade it manually each time.
90% of the time as you say so yourself.
Migration implementations for any platform have always been somewhat vexing for me, but South has been all but automatic for me. I've never had to deviate from `schemamigration app --auto && migrate app`.
On the other hand, I ensure that every change that needs to be made to the schema can happen at any time, and both new and old versions of the code will happily work. I believe this helps with upgrading in that there is no delay during deployment and migration. Having not used south on a real project though, I'm unsure if this is a false benefit.
[1]: https://bitbucket.org/offline/django-annoying [2]: https://github.com/niwibe/djorm-ext-expressions
I have to admit that part of me is wondering if I am making the right decision in investing time, effort and money getting up to speed on Python/Django for new projects rather than devoting that time to something like Go. Can't do both. Not enough time and we need to get things done.
Now, I do appreciate that Python has a huge ecosystem and lots of very useful libraries out there. That alone might make the price of admission worth it. I also look at these benchmarks and it seems that PHP (something I am really trying to leave behind) frequently scored above the various Python ecosystems.
Can someone put this into perspective for me? In the end performance can and is hugely important. What am I missing when looking at all of this?
1. Raw language speed is an important factor to consider but for some projects language syntax, community, ecosystem and libraries are much more significant. Those are python strong aspects. Due to its simple syntax, Python is very popular at mooc platforms, such as Coursera, edX and Udacity. Really lots of very good educational content and I mean MIT level good.
2. Usually web projects do not need to scale that much. Website traffic distribution follows power law so if you do not expect your project to be in top 1%, Django will usually handle all standard CRUD workloads.
3. There are Python bindings for many C libraries, for example image manipulation with C speeds is available using ImageMagick bindings.
Here is a great comparison of Go and Django: http://lincolnloop.com/blog/djangonaut-building-webapp-go-go... .
I personally have used Django for quite a few projects, and haven't had any "I wish I used a different framework/language" moments.
Go or Java are wonderful languages if you have serious number-crunching needs or if cutting your server budget in half would save you a billion dollars. My server budget is $500 a month and if you doubled the performance of all of my servers it would be $500 a month. I'll elide the detailed explanation of the whole "User clicks a button" to "User sees something happen" HTTP request waterfall chart but my Rails app is generally less than 5% of it.
YMMV if you're Google, but I'm not Google.
That said, I'm not Google either, and I'm happily using django and love using it.
[0] http://www.techempower.com/blog/2013/03/28/frameworks-round-...
http://www.techempower.com/benchmarks/#section=data-r7
Python didn't beat NodeJS in Database queries though.
Yes Go is fast, but Go is also cluttered with its static type system.
Personally I feel pain to deal with flexible JSON marshal/unmarshal, in Python dict() is nearly 100% identical with JS objects in value.
I can't find what you're referring to. Node was still infront of Django in the JSON benchmark.
Django (until 1.6) hasn't had connection pooling for the database either. Updating those benchmarks with pooling should make significant differences.
But my point was that the framework itself can have a huge impact. Database and other I/O does have overhead, and it may even be significant. But it's not necessarily the biggest factor.
The Nodejs web framework is Express, behind Bottle and Falcon.
But the original point was that database and I/O make the largest difference. After restricting the set of languages and frameworks to python and javascript, I can only (now) agree.
109,000-5700 rps for JSON requests, and 4800-117 rps for multiple database queries. Quite significant.
My advice is not to sweat it and know that if/when the time comes that there are plenty of options for making things fast such as load balancers, serving static files from another machine, database on a different machine, various kinds of caching (whole page, database level or somewhere in between), tweaking the number of python threads/processes, etc and that you can choose these based on the results of benchmarks. A lot of the time there are big wins to be found in optimising frontend stuff first anyway.
So, if you're doing heavy calculations in your user interface layer (that's what a web site is) - you're a) supposedly doing something wrong b) probably better with something else than plain CPython.
In many other cases, Python is fine. Not the very best option, performance-wise, but there are many other aspects than performance, too.
That's not to say runtime performance issues have no effect, for example if you had some kind of algorithmic server at the center of a high frequency trading system or something like that, then benchmarks might be more critical (in which case write that one service in Go or C, then have your Python infrastructure talk to it via REST or as a C extension). It kind of depends on what you're doing but in the vast majority of business cases, it's not.
Pypy is also a great alternative to cPython which I hope starts to become viable for widespread production usage very soon - it features performance a lot more like the JVM.
Your performance concerns (and I believe the other comments have touched on this) likely don't matter for most real world sites, and then, even if you get big, you can probably work it out (Pintrest is done with Django I believe, Github is rails, etc).
The question you really want to ask yourself is what does your compiler/runtime do for you. I've been pretty much solely a Python developer for the past year or so, but I like Java (... yea, I know), and find Go interesting, too. A compiler is no excuse to not write tests, but you know what you've written is going to run after it compiles. Added datetime's wrong in Python by adding one that has a timezone marked and one that doesn't? Runtime error. In fact, everything is a runtime error. Tests should not be optional regardless of language, but with Python and Ruby, they really take the place of the compiler in statically typed languages.
I don't want to discourage you from Python by any means, the Django community is fantastic, and Rails 4 with it's Russian doll caching seems pretty sweet. And nothing can beat the libraries for these two projects. The other options that I've seen (ASP.NET, Java Play, Go Revel, all for which, admittedly, I've only done preliminary reading about) just don't seem to have the libraries to get you off the ground as fast (though, if you've never written all the tedious stuff like user creation, email password resets, it is a good exercise in web development basics).
If you stick with Python I encourage you to: 1. Tackle learning the best practices for testing as you go. 2. Check out Two Scoops of Django [1] after having used Django for a bit. 3. Learn good server management skills (maybe look into Puppet [2])
The third seemingly is an offshoot, and would be necessary for really any framework, but once you're ready to deploy your app, you'll likely spend a great deal of time comparing your various options for stack choices and will get your sever set up just how you want it. Make sure that process is easily repeatable. If you don't like how Puppet does it, look into Chef or Salt Stack or whatever; these will save you tons of time down the line.
[1] https://django.2scoops.org/ [2] http://docs.puppetlabs.com/mcollective/reference/basic/getti...
django-admin.py startproject --template=https://github.com/twoscoops/django-twoscoops-project/zipball/master --extension=py,rst,html PROJECT_NAMEhttp://www.python.org/dev/peps/pep-0020/
Choosing a minimalistic web framework like Flask or Sinatra makes sense if you're experienced enough to know what choices Django and Rails make for you and you're comfortable making those decisions on your own. Otherwise, pick Django or Rails and start building things.
That being said, I'd love to see a microframework like Flask for Lua/openresty gain in popularity.
Doing this right saves you from having to go all out on caching, and it can help keep your infrastructure simple, the benefits of which shouldn't be underestimated if you have a small team.
Also, if you're building interactive B2C sites and have some success, server costs can quickly become nontrivial, to the point where a 2-3x speedup can make a large monetary difference, and it's easier to not have to rewrite things for speed. If you're building B2B SAAS sites, server costs are probably a complete nonissue.
I would say that django_extensions is handy enough to have it included (mainly for the sake of shell_plus and runserver_plus), South is invaluable (although migrations are going into the core in near future[0]), but others—it very much depends on the project.
sekizai is one I've never heard about before. Judging from the first sight, I think it can make templating more flexible, but at the cost of increased complexity.
Earlier I thought that reversion is great to have by default, and I was adding it to every project. Then, as I recall, mysterious errors started popping up one day in one project after Django update. They turned out to have something to do with reversion compatibility. I looked into it and found that we had no use cases for reversion—I could simply throw it away without any consequences, which I promptly did.
I've used grappelli for a while, but in the end I found its UI less usable than native Django admin UI. (Recently I stumbled across http://djangosuit.com/ — this one does seem more functional, although I've never used it in my projects yet.)
[0] https://docs.djangoproject.com/en/dev/topics/migrations/
Or have we?
* django-debug-toolbar
* django-secure
* django-celery
* django_compressorFurther reading on why sekizai and what it does in this blog post: http://ojii.ch/post/why-sekizai/
> of django projects as fastcgi backends.
Or just use uwsgi[1] and supervisord[2] if possible.