Notes from building a blog in Django
til.simonwillison.net
til.simonwillison.net
One catch: I like to deploy static sites to GitHub Pages. It turns out the Django ecosystem has an answer for that in the form of Django Distill [1]. Distill adds a new management command that generates static content by repeatedly invoking your views.
I've used Distill to build quite a few static sites. It's great to be able to deploy to Pages but use Django's admin UI locally. Typically, for these sites, I check in my 'production' SQLite database directly to my git repo.
(Combine this pattern with GitHub actions and you can do a lot with a little. For instance, here's a toy I built to summarize upcoming Seattle city council meetings [2].)
[1] https://github.com/meeb/django-distill
[2] https://scc.frontseat.org/ -- source at https://github.com/front-seat/engage
I had the opposite opinion. If you were trying to convert Rails, Laravel or CakePHP users, this would just convince them that Django was more manual and involved than those frameworks. It would be better to utilize extensions like `scaffolding` to give a more equivalent workflow. It does demonstrate general MVC benefits and architecture, however.
I was recently looking for a static site generator for a local relay race I help organize and django-distill was the top of my list. I ended up writing my own SSG because I had very little HTML that actually had to be generated on the site. But might check out django-distill in the future if you found it worth learning.
https://instagram-engineering.com/types-for-python-http-apis...
https://engineering.fb.com/2023/09/07/culture/threads-inside...
https://engineering.fb.com/2022/05/02/open-source/cinder-jit...
https://engineering.fb.com/?s=django
Django is also excellent for e-commerce with projects like https://github.com/saleor/saleor - or you can roll your own e-commerce system quite easily, which we've done at my travel startup.
To the larger point about Django and e-commerce, though, I'll share that we do a large volume of complex transactions where multiple funds flows for different parties need to be coordinated, so it's been vital to be able to have a dynamic data model to track the payables/receivables lifecycle of a given booking and add overrides/configurability to handle unique situations as they come along. Django's been a perfect match for that, the framework source code is very approachable (IMO much less "magical" than Rails in this regard while having the same level of expressivity and shorthand), and since it's Python, it works extremely well with data science tooling as well.
Here in Seattle, Rover (a $1B+ market cap company) also uses Django extensively. As do lots of interesting local startups.
One of the many more complex sites we've used it to build is https://www.business-humanrights.org/en/ which tracks over 10k companies, 100k+ articles in 15 or so languages. We've also built internal dashboards, live updating forums, donations platforms and so on. Another interesting one might be https://www.achurchnearyou.com/ which is used by the church of England to provide "mini sites" for all their congregations around England, so several thousand church administrators use that every week.
Simon Willison calls it the “baked data pattern” — where a read-only copy of your data is checked into your repo alongside your code. [1]
(And, just to tie the threads together, that’s the same Simon who wrote the OP we’re all commenting on, and who is also the co-creator of Django and more recently of the very handy Datasette!)
There is an excellent tool - https://djangobuilder.io/ that can also really speed up the boilerplate even more.
Django project itself has kept pace with recent developments in web development. I still remember migrations being an external project, getting merged in and the transition that followed. Ecosystem is pretty powerful too with projects like drf, channels, social-auth etc., covering most things we need to run in production.
https://github.com/trypromptly/LLMStack is a recent project I built entirely with Django. It uses django channels for websockets, drf for API and reactjs for the frontend.
Initially though Python got a lot of flak for this mandatory identation feature but later it's really its main advantage against other interpreted programming language of its time and programmers consider it a language with pseudo-like codes, but with functionality. Then come the era of data science and analytics popularity where Fortran and Matlab were kings, but the former is too primitive and the latter is a proprietary language. Interestingly, Guido himself was involved with new Python matrix operation syntax in order to make it more intuitive. Due to the rise of AI and machine learning where matrix operations are pervasive, Python becoming the best open source alternative and the rest is history. This is a very interesting read on HN for the reasons why Python win over its competitor and now the most popular language on the planet:
[1] Ask HN: Why did Python win?
Django and Wordpress are still valid use case for a blog.
Then again, with Django-Distill (linked above), you can get the best of both worlds.
Powerful text editing tools I'm familiar with, source control, fully offline work, and very lightweight tooling that starts quickly (just a python script).
I wrote a bit about the experience here: https://nicolasbouliane.com/projects/ursus
but emoji laden docs and 500/month patreon pay better than contributing to established things i guess
Also I think I remember with DRF I had trouble generating a good openapi client. With ninja it works really well.
Yes. They are currently migrating to Pydantic 2 which Ninja 1.0 will be based on.
I'd rather have DRF (a subset maybe) inside django. No external dependencies and integrated auth could enable interesting patterns like complete modularization of the admin panel, making it json api driven and making deploys very easy such as `python manage.py generateadmin` and you can load that on S3 or whatever.
I also know the existence of whitenoise + many workarounds, but python is already a quirky language, django has its own and its starting to have a huge chunk of incidental complexity (my last django project, very mature codebase, had like 3 libs to work with JWTs...) to work proficiently with. Not to mention the footguns of it!
People like options, but with opinionated frameworks I'd rather have lack of them.
``` commit 53eddd4a0f8786e23f511a653d8d7ffa947ad8db Author: Simon Willison <simon@simonwillison.net> Date: Mon Apr 23 21:24:41 2007 +0000
Added HTTP_HOST example header
git-svn-id: http://code.djangoproject.com/svn/django/trunk@5063 bcc190cf-cafb-0310-a4f2-bffc1f526a37There are earlier references as well, but I think the author was making pretty deliberate choices.
Ask HN: I plan on developing and deploying a bunch of Django apps in the coming months as I’m doing a 3 month stint at https://recurse.com
GitHub Pages is great for purely static content - but what is the preferred/""best""/cheapest option for django/postgres hosting?
I know there’s tons of options - I used to be a Heroku fan until recently. Now am on Render (where things get expensive)
Is Pulumi/Terraform/IaC on AWS the way? I want git-deploys, not have to write .sh scripts to restart nginx and pip dependencies that auto un/install if possible.
I also thought of deploying one mega django project and having everything be a django app under that - it doesn’t feel sustainable/clean but maybe it’s ok for a bunch of small projects?
Thanks!
I prefer digital ocean for VPS and have found you can get pretty far on the smaller droplets.
I’ll typically have two environments, one getting auto deploys from anything on a feature branch, and production which gets deployed when there are changes to main.
For my setup, I initially learned using blog entries from the folks who blog at:
I blended multiple entries to achieve just the setup I wanted.
I haven’t gone up to terraform though, finding it sufficient to use a pre built image of ubuntu / docker at the initialization.
My oldest projects still use .sh to handle deploys but I’d never go back to that stuff.
Alternatively GCP and AWS both have elastic compute options that should work as well. (App Engine and Beanstalk? respectively.)
Running a VM is going to be annoying (way more ops toil) and cost you more than you need, if you're making on the order of requests-per-hour instead of requests-per-minute. But if a VM is cheaper than your best elastic option you can make it work. I would not recommend sharing apps within a project as migrations can be annoying and you'll get downtime on every app whenever you deploy or break any of them. But you could have one VM serving up lots of projects, and give one port per project, then wire up a load balancer & DNS name for each project/port.
It's possible to do git-deploys with a VM but it's a little work to wire up. You'd set it up with GitLab, and you can wire up a simple deploy script that effectively does a `docker stop && docker run` to launch your new app container.
You can run it on top of a single cheap VM, for example Digital Ocean or Hetzner, for a few dollars/euros a month. It is compatible with a lot of Heroku buildpacks e.g. PostgreSQL and Redis, has a similar CLI, and includes plugins for things like LetsEncrypt that can be installed with a couple commands. As with Heroku and other PAAS you can deploy with a simple git push.
It's absolutely fine for the MVP stage of a more serious project, but it's not really meant for scaling out beyond a single server. I mean sure, it's possible, but there are better tools for the job, and at that point you should have a bigger budget/team.
Django really is beautiful.
The blog described in this article renders Markdown directly. My https://simonwillison.net/ blog doesn't do that yet, instead I paste my Markdown into https://til.simonwillison.net/tools/render-markdown and then paste the HTML into the admin interface.
Has anyone heard if Django (or FastAPI, etc) will be ported to Mojo programming language?
I've seen all these JAMStack websites marketed as 'fast' when the only reason they are fast is that these web frameworks (11ty, Astro, Hugo, Gatsby, etc) are essentially static generators to HTML which is pretty much the same as creating a plain html file if you want to.
I'm mainly interested in the SEO and speed side of things but does using a web framework like Django or an SSG make a difference?
Or... run a Django app behind a caching proxy like Varnish or Fastly or Cloudflare. I do that for my https://simonwillison.net/ site (Cloudflare) and helped implement Varnish at Eventbrite - it's a really solid pattern.
Otherwise static is 100% the way to go, if only to reduce how much code to deploy.
I’m the sole maintainer of a Flask app that uses SqlAlchemy, and the experience is similar except that I really miss the admin and auth. However every single flask app ends up using slightly different libraries and architecture, which means that onboarding always takes longer than an equivalent Django app.
Also, I have over a decade of Python experience; I wouldn’t count on people with little Python experience to produce well architected apps in Flask, whereas Django does a lot of guiding the user towards established patterns.
Some parts of Django are antiquated and I do not recommend them (specifically the Forms API really has been superseded by pretty much any newer forms library), but that’s still way less baggage than having to pick up a new stack for every new Flask app.
Are you recommending people use something like WTForms directly?
I think the rendering aspect in the Forms API has always been a PITA and the validation aspects always left something to be desired. Rendering and validation/serialization should be decoupled IMO, and in that sense the Forms API fails.
Sometimes I wish for the Django admin to be reimplemented in FastAPI/SqlAlchemy as a more modern approach to the same problems.
https://adamj.eu/tech/2019/04/03/django-versus-flask-with-si...
Flask works for simple projects but as your needs grow, the built-in and extensive Django ecosystem will have you covered. And as seen above, Django in its simplest form is no more complex.
Some people really don't like the Django ORM, so in this case you might avoid Django altogether. It's possible to use Django without its data layer but it's a big part of its value.
Also if you are building a pure API project, FastAPI is worth checking out for a streamlined process. Django Rest Framework works too, but is an extension.
Could Django do some things better? Sure, but it gets so much right, and has so much community support, it should be the default answer. If you ever get stuck with a problem, thousands of others have faced the same thing. With Flask, owing to your special snowflake configuration of extensions and wiring, you may not find any guidance anywhere.
If you mean postprocessing the feed server-side with an XSLT sheet, to then serve the resulting page: sure, that would work, and a lot of people used to do things like that around 2003. It fell out of fashion because XSLT is just hard to safely combine with advanced JS-based features.
xsl-stylesheet with css works for me as well.
My blog is a pretty straight-forward Django setup without many other dependencies, so it's a lot less of an attack surface: https://github.com/simonw/simonwillisonblog
The core Django app manages SSO and team creation, then calls Fly APIs to create volumes and machines for each team. I'll be writing this up in a lot more detail soon.
I'm finding stability on Fly is excellent for my deployed containers.
The problems I've seen are more around deployment - occasionally there will be incidents where fresh deploys can't go out for a few hours, all of which are reflected on https://status.flyio.net/
Many frameworks try to be good at this but finish development at a level of abstraction below these frameworks, leaving the last level to a plugin ecosystem that requires so much more work to wrangle into a fully working web app (see: Flask and its modern successors in Python).
Other systems come at this from the opposite end, trying to be low-code or less-code, like headless CMSs or static site generators. The problem at that end always seems to be a lack of flexibility and over-reliance on proprietary systems or SaaS products.
Django is just Python. Rails is just Ruby. There's nothing special to them and extending when necessary is often trivial, particularly because they have such mature extension points.
There's a lot of criticism that can be levelled at these ecosystems for not keeping up with whatever the hype of the year is, or failing to "scale" in various ways, but damn do they get you a very long way with very little work.
[^1]: Phoenix may be just about in this category, I don't have personal experience with it though, just going on the fact it seems to be very Rails inspired.
Eh, I wouldn't agree with that. Both Django and Rails extend the languages quite a bit with deep class overloading, reflection and fluidic approaches.
Especially Rails, which pushes Ruby's DSL-like abilities to an extreme. If you go from learning RoR to then having to make a standard Ruby script, you quickly realize how much of what you built relied on Rails-specific functionality. Django, more or less, is just Python with a bunch of logic built for you, relative to that.
Sometimes it feel like a minor superpower in just how much more productive it is over pretty much everything, especially in the early stages of development.
Even later on things like working auth and permission systems and API frameworks are huge timesavers.
Proof that we are a fad-driven industry.
- inability to use static type checker without writing most of the annotations yourself
- half jinja2 functionality
- ORM that makes it impossible to get your query count down to a manageable level
- 3rd party modules that break the types even more
I am lobbying for moving to FastAPI with Jinja2 and SQLAlchemy. We write our admins in Vue anyway.
Jinja2 is a supported template backend for Django these days: https://docs.djangoproject.com/en/4.2/topics/templates/#djan...
- There is nothing wrong with the ORM, you may be resorting to using deferred attributes too much. There are ways to eliminate them (select_related, prefetch_related) if you are having performance issues.
- Jinja2 has been a supported templating engine for years.
- I don't understand the question about static type checking. Can you elaborate?
And with Django you often don't need to write anything.
You're right to some extent with all of your points (jinja2 is available, not default, ORM is equivalent to SQLA most but not all of the time), but they're one side of the trade-off. The other is that you're writing admin views that you likely wouldn't need to with Django.
It's not perfect for everything, but it is much faster to write with than anything else for reasons like this.
I think of the all the ORM's, I've ever used. Django was the greatest.
However django is living in the age of past glories. The admin is extremely unacceptable for anything beyond trivial borderline trivial use cases, and the modifications you'd have to make are just awful especially the more interactive something needs to be. The extremely tight integration between models and the modeladmin is a blessing and curse.
The people who like Django,also tend to overload it to do everything. This makes sense at small companies. The only place I really see Django at large companies is as an api using DRF or something.
For internal admins, I've been lobbying to use https://directus.io/ at my company.
This is not a bad thing. Using Django as an API backend is amazingly fast in terms of development time, especially with modern frameworks such as django-ninja [1].
Just use the built-in ORM to create models, write your endpoints, and use the built-in admin interface to play with the database if you don't have endpoints for everything.
There is also a less known feature of Django called admindocs [2], which automatically generates a human readable, hyperlinked documentation for your models and relations between them.
[1] https://django-ninja.rest-framework.com/
[2] https://docs.djangoproject.com/en/4.2/ref/contrib/admin/admi...
- It's incredibly more productive.
- Seamlessly integrates with any django website, or hold your hand to create one from scratch.
- Play nice with the whole django ecosystem.
- Is easy to pick up because under the hood "it's just django". It uses django models, routing, auth, etc. It's uses standards, best practices, and is a good citizen.
- The blog admin and the provided block editor are good out of the box.
- It still gives you all the flexibility you want: it doesn't force any template on you, any page structure, workflow, nothing. You get to decide how simple or complex you blog is.
- The team behind it is super nice.
The benefits of static site generations...
If you don't mind answering a couple of questions:
- I have a requirement for basically a CMS with the capability of serious extensibility, built in either python or javascript (I considered ghost but it does meet my requirments). Main job is for a membership based cms but to potentially add functionality outside of standard cms stuff. Would you say wagtail / django combo would meet that requirement? (Sounds like "yes")
- How does wagtail compare to django cms? (Less interested in feature comparisons, more interested in your developer experience with both)
Thx in advanced!
On one hand that means you have to do more work: define exactly what you want, then declare it in code, then build the templates.
On the other hand it means you can do exactly what you want, including the requirements you just mentioned.
Edit: Since I got extremely downvoted for this comment. Here is why:
You should use static site generators for new blogs in 2023.
Performance: Static sites typically load faster than database-driven ones.
Security: Without a database, the risk of SQL injection attacks is eliminated.
Scalability: Static sites can handle high traffic without complex hosting solutions.
Maintenance: No database means fewer maintenance tasks and potential failures.
Hosting and Costs: Static sites often have cheaper and more flexible hosting options.
Decoupled Architecture: Modern trends prefer separating frontend and backend, reducing the need for databases.
Development Simplicity: Static site generators offer a straightforward content writing environment.
Backup and Portability: Static sites are easier to backup and migrate.
Edge Hosting: Static sites benefit from faster load times with distributed hosting solutions.
IMHO, it's true that a static site generator should be the way to go in 2023, instead of using a web framework with a db.
I am personally running a hugo blog, on netlify, with netlify CMS. I have 0 costs, great performances, everything needed out of the box. What else to ask for ?
Also, for a company blog, an SSG likely means a requirement to know Git, which could be inconvenient.
If you really need comments in your blog when using Hugo, you can still integrate them with Disqus https://gohugo.io/content-management/comments/
In regards to the requirement to know Git: It is just necessary in the setup phase, the blog posts creation and edition are controlled through Netlify CMS for me.
Plus tag pages, archive-by-date-pages, series pages and more.
I don't particularly want to have to wait for all of that to build!
It also offers faceted search against all of that, powered by PostgreSQL: https://simonwillison.net/search/?q=static%20site%20generato...
That said, personally, I love PG and enjoy using a DB. But I can see pro's to both sides.
I use SQLite FTS for full-text search (it’s the only non-static endpoint).
A full site re-render on a Raspberry Pi takes 5 minutes, and a full reindexing - FTS plus linkmap - plus publishing around 10, but I only do that yearly or when I update things like CSS, layout, etc.
It all runs off a Git hook, uses SQLite to hold all the FTS, base HTML and metadata, and is as asynchronous as can be (including my own asyncio blob upload library). Costs me effectively zero.
Code: https://github.com/hparadiz/technexus/blob/release/public/js... https://github.com/hparadiz/technexus/blob/release/public/js...
Source: the fastcomments docs, blog, and several of my sites are SSG. I don't like it anymore.
I'd call the let's-insert-an-unnecessary-database-here outlook the crazy thing. Not to mention, basically a violation of the Principle of Least Power <https://www.w3.org/DesignIssues/Principles.html#PLP>.
K. Ignoring that...
None of those are prerequisites. They're not listed by the author of the post linked here as among the features that he considers "essential for a blog in 2023" (nor are they even present on the Datasette Cloud blog that is the subject of the post), and it's not a "crazy optimization" to not go out of your way to introduce them.
(Is he, in your view, not actually running a blog?)
Static sites sound nice and clean to a junior dev, I get it. But after using that approach for years I'll never do it again. Too much of a pain as things grow. Build step can also get slow. Django is so easy to use and extend. You can use it with sqlite, no DB to run.
Wikipedia is a nice example. Could that be SSG. Yes. Is it, and should it be? No. If your site has more traffic than Wikipedia let me know.
Do you know how much support calls I have to field for fastcomments with people trying to integrate into SSG systems with SSO or whatever that would be simple with SSR etc? A lot.
No, Wikipedia is a terrible example. Absurd, even. We're talking about blogs. We can stick to that example. We don't need to reach for bad ones.
I'm being a bit dramatic, but it's to stop people from using SSG. I don't like working on said applications because adding every little feature is a pain and adds to the build time. Running a simple service and DB is easy - I've been doing it for a decade. I ran blogs and sites with databases when I was in HS - surely big companies can do it and put cloudflare in front of it.
There's a reason static site generators exist. This should be made clear that this is just a toy/example app to demonstrate Django, not something someone should actually use.
Generating the pages on each request is madness, and is why Wordpress in the default configuration (without WP Total Cache, which allows the httpd to bypass Wordpress entirely for most requests) falls over as soon as it's linked from any media site. Let's stop repeating these engineering mistakes in language after language.
Compile your blog to static pages and deploy those. Hugo, Jekyll, and a million others await you, as well as CF Pages, S3, GitHub Pages, Netlify, and others.
My main blog https://simonwillison.net/ runs as a Django+PostgreSQL app on Heroku behind Cloudflare, with a 15m cache TTL for every page.
This works perfectly. I survived a surprise Elon Musk tweet a few months ago which the server didn't even notice, because Cloudflare absorbed all the traffic: https://simonwillison.net/2023/Feb/17/analytics/
They are usually easier to administer for less professional users, as well as being able to quickly modify from standard web interfaces.
If it’s backed by a cache like redis it’ll easily handle Hackernews level traffic, even at very short cache times.
> Generating the pages on each request is madness, and is why Wordpress in the default configuration (without WP Total Cache, which allows the httpd to bypass Wordpress entirely for most requests) falls over as soon as it's linked from any media site.
No, it is not. WordPress is slow because it has an exceptionally poorly designed database schema, which requires SELECT DISTINCT and joining on the same table multiple times per query in order to do anything meaningful.
> Compile your blog to static pages and deploy those. Hugo, Jekyll, and a million others await you, as well as CF Pages, S3, GitHub Pages, Netlify, and others.
One downside of this is that you have wait for a compilation process every time you make content changes. Another is that any forms you might have still need a backend of some kind. Static sites are really only appropriate for sites that rarely ever change. A blog may or may not change frequently enough to warrant a backend.
Any modern SSG can build and deploy in under a minute, usually in single digit seconds. This isn't really an issue in practice at all.
Static sites work fine even if you are updating many times an hour. This is a far cry from "rarely ever change".
It's Phoenix in my case, but I've had a dynamic blog with Django for a long time and it's not massively different in terms of performance. I also do like static site generators, but more for being able to dump stuff into S3 and avoid some deployment complexity (but they introduce other complexity).
The main point is really that Wordpress, with the default configuration is just off the charts terrible. Like, it had to have taken massive amounts of engineering to make something that performs so poorly. If you don't pile a million features into a product and overengineer the living daylights out of it, it's really not very hard to handle quite a bit of web traffic.