The Practical Guide to Scaling Django
slimsaas.com
slimsaas.com
First, this guide should emphasize the need to measure before doing anything : django silk, django debug toolbarsm, etc. Of course, measure after the optimizations too, and measure in production with an apm.
Second, some only work sometimes : select_related / prefetch_related / iterator will lead to giga SQL queries with nested joins all over the place, and ends by exploding ram usage. It will help at first, but soon enough one will pay any missing sql knowledge or naive relationships.
Third, caching without taking the context into account will probably lead to data corruption one way or another. Debugging stale cache issues is not fun, since you cannot reproduce them easily.
Fourth, celery is a whole new world, which requires workers, retry and idempotent logic, etc.
Finally, scaling is also about code: architecture, good practices, basic algorithm, etc
I'll end by linking to more complete resources : - https://docs.djangoproject.com/en/5.1/topics/performance/ - https://loadforge.com/guides/the-ultimate-guide-to-django-pe... - https://medium.com/django-unleashed/django-application-perfo...
Which is darn hard if you are a beginner in a framework, loops in loops still bites me after reality does the integration test for me. This is especially true when you try to do a simple thing as a beginner. By scaling I am just talking about normal production, going from 2 developers to a couple of thousand customers.
If you need a fast solution then add an integration test so that the system stays fast.
One can only hope it's data corruption and not a sensitive data leak.
The nice thing about that is that spotting those, and the basic approach to fixing them, if not the exact implementation details, are cross-platform skills that apply basically anywhere.
I actually can’t recall any other notable performance problems in those sorts of systems, over the years. Those are so common and the fixes so effective I guess the rest has just never rated attention. I’ve seen different problems in long-lived worker processes though (“make it streaming—everything becomes streaming when scale gets big enough” is the usual platform-agnostic magic bullet in those cases)
A bunch of TFA is basically about those things, so I’m not correcting it, more like nodding along.
Oh wait I just thought of another I’ve seen: serving large files through a scripting language, as in, reading it in and writing it back out with a scripting language. You run into trouble at even modest scale. There’s a magic response header for that, make Nginx or Apache or whatever serve it for you, it’s a fix that’s typically deleting a bunch of code and replacing it with one or two lines. Or else just use s3 and maybe signed URLs like the rest of the world. Problem solved.
ActiveRecord pattern saves you a few lines of code now, and explodes your foot off later.
ps - I didn’t know about template “cache” directive
The only place it's possible worth it is if you do a lot of database queries from your template rendering, and you're therefore caching database results (as rendered text). In that case, it's an easy patch. However a much better solution is to fetch all database results up front.
In my previous company we had a very significant Django codebase with plenty of templating, and found that using the templating system for (lazy loaded) database queries or caching was more hassle than it was worth and avoided it as much as possible. Treating template rendering as a pure CPU bound function was always better.
I wrote a short blog post on recent optimizations we did on our Django codebase: https://tmarice.dev/blog/better-living-through-optimized-dja...
It would be nice to include the generated sql queries along with the code samples though. I've been on a similar path recently and being able to see the queries was really helpful (even the ones that failed!).
Using a library like keyring [1] is a significant step up from a .env file sitting in your dev environment.
In other words:
- Store secrets in settings.py (bad)
- Store secrets in .env file (better)
- Store secrets in OS-level key vault (even better)
When the secrets are in a plaintext .env file, that file can get leaked in many non-obvious ways. Your antivirus uploads a copy, your IT department runs backups, someone on the team clones your git repo to a OneDrive/Dropbox folder and puts the .env file there. Then any of those services that has a leak, or any of the services those services use has a leak (improperly configured S3 bucket, etc), your secrets are leaked.