Django 4.0 will include a built-in Redis cache back end | Hacker News Reader

Django 4.0 will include a built-in Redis cache back end

github.com

130 pointsstanulilic58 comments

Sensible decision, although hardly a surprise or a killer feature for something that is a major version bump.

I assume the async story is going to be more fun.

By the way, if you are looking for a good key/value store with expiration without the need to setup redis, have a look at the excellent diskcache:

https://pypi.org/project/diskcache/

It is based on sharded sqlite files, and it's surprisingly capable, even in multiprocessing based env like a django site.

Sqlite is really showing up in all sort of use cases these days, like huey, a task queue that doesn't need any broker to be installed to be useful: https://pypi.org/project/diskcache/

Now things can get fun...

With supervisord (supervisord.org) you can manage multiple processes, and with fastapi + starlette, you can serve your WSGI site without a proxy pass, including static sites: https://fastapi.tiangolo.com/advanced/wsgi/

Then with shiv you can bunddle all that in a pyz file: shiv.readthedocs.io/

Meaning you can now just upload a pyz file to your server, run python on it, and that's it, you have a full stack deployed, with a web framework, db, caching, task queues, static file serving, which can handle around 100k visitor each day without a sweat.

No more infra needed. Just Python code, all bundle in one file.

That's really neat.

Totally agree on hoping there is at least some movement forward on async--particularly async ORM. I love Django and DRF. It is my go-to for a lot of things, but the lack of async support is increasingly a problem for more organizations. Honestly, it would not surprise me that if a solid, kitchen sink web framework emerges in golang, Django will be abandoned rather quickly.

2 cents.

I love Django but it seems like such a limitation for certain workloads. You can often work around some of these issues with Celery but still…

.Net already offers a kitchensink with async, it’s growing a lot and is much more attractive than a decade ago.

Yes, maybe in 10 years when this Go framework catches up with all the features and becomes as battle proven, well documented, maintained and funded as Django and somehow we know is not yet another thing that will be abandoned next year.

I've been writing web applications using python and django for close to 15 years now. Smooth, concise and stable. Doing the same using a verbose and boilerplate heavy language like golang doesn't sound attractive to me, at all.

Not sure I understand the fuss about upgrades either, at my current job I've gone from python 2 to 3 and from Django 1.8 all the way to 3.2 LTS without too many headaches. The release notes provide detailed instructions on what is deprecated and what you should do to fix it.

The Go hype is as bad as the frontend hype in my opinion.

If you're replacing Django with Go then either you never needed Django, or you have no idea what you're getting into.

At a previous job backend team decided to replace rails "because monolith slow ruby not fun anymore" with Go. Three years later they were still reimplementing things rails gives you since day 1. I've seen abominations as doing migrations as SQL scripts ran by cobbled together bash scripts because "as we don't have orms then orms are bad".

No need to say now they're about 4 years in and the one still running the business and making all the money is the monolith, not all the Go microservices around it.

Go is a replacement for c/c++, maybe java and for building infrastructure probably.

If you're using it for doing CRUD forms, you're using it wrong.

Out of curiosity: what features from Ruby are they reimplenting? I have no experience in Ruby but in a few other languages and framworks. And having encountered something missing yet when writing Go web services.

I've not even mentioned Ruby.

Rails. Features of rails.This is exactly the problem...people focus so much on the language they forget the ecosystem they're in, and the ecosystems matters a lot more than the syntax or the performance.

> And having encountered something missing yet when writing Go web services

How do you do form validations, translations, database access, migrations, email sending, background jobs that don't die on a deployment, user authentication, file uploads, caching, etc?

Of course all of this can be done, you just need to take 100 decisions and tie together 256 libraries many of them probably abandoned in a few years from now.

If you don't need any of that, then just what I said before: ok, you never needed rails from the get go.

GP (BiteCode_dev) made a valuable comment for how we can build a easily deployable python application with

>web framework, db, caching, task queues, static file serving, which can handle around 100k visitor each day without a sweat.

i.e. with 4-5 large dependencies of frameworks, libraries, software etc.

In Go you can off the bat reduce those dependencies to 1 or 2, even completely eliminate them, you get deployable package and async for free with faster development cycle and lesser headaches of managing dependencies in the long run.

Interesting. I'll look into some of those projects.

I think you put the wrong link for Huey, btw.

Yep: https://huey.readthedocs.io/en/latest/

Last year I tried Huey for a project where usage was very lightweight. Performance was more than enough with SQLite. Unfortunately Huey with SQLite was not great at handling unexpected crashes like server and docker restarts.

Switching to Redis solved all of these issues.

Maybe it's better now.

But then how do I do microservices in a monorepo based SPA server rendered running on kubernetes?

Also, pair it with Unpoly, tailwind and slippers and you get a pretty amazing frontend setup as well.

https://unpoly.com/

https://mitchel.me/slippers/

>you can serve your WSGI site without a proxy pass, including static sites

What about HTTPS?

Django is, of course, an open source project and the core team and contributors are largely free to work on whatever they prefer.

However, as a user of Django I sincerely wish they would focus more on stability:

Changing jsonb deserialization in a patch release, 3.1.1: https://code.djangoproject.com/ticket/31956#comment:16

Queryset .get() can cause OOM, ticket has gone back and forth, problem existed as recently as Django 2.X: https://code.djangoproject.com/ticket/6785

Django's cascade options differ from the database's, are manually implemented: https://code.djangoproject.com/ticket/21961

Can't update keys via ORM, related to the above: https://code.djangoproject.com/ticket/21265

Inconsistent behavior around constraints checking in unit tests, can lead to inexplicable behavior and difficulty testing when a constraint is violated: https://code.djangoproject.com/ticket/11665

In general, the ORM functionality and the inconsistency, bugs being introduced in patch releases has me looking elsewhere for all new development and recommending to other engineers the same.

We have a running joke in our engineering team that every bug we hit in Django will have an open or "closed wontfix" issue 5-10 years old.

I love Django, but version upgrades are a gigantic hassle.

What would you suggest as an alternative for a new project?

Django is still my go-to. Specifically [Django-REST-Framework](https://www.django-rest-framework.org/) with a front-end written with [react-static](https://github.com/react-static/react-static).

Django's ORM is so nice and the ecosystem around it rocks.

Its biggest downside is painful upgrades. They don't really follow [semantic versioning](https://semver.org/)

To me however the ORM is one of the most annoying things. Not a fan of ORMs in general, but Django's implementation promotes a usage pattern where django-models are passed around everywhere, leading to hard to untangle dependencies between services once they do literally anything with the objects.

I'm relatively new to Django (longtime Flask user), and what struck me about the Django ORM so far is that it seems much easier to write extremely low performance queries. I didn't really feel this way when using SQLAlchemy or peewee, but perhaps that's just my inexperience.

I really enjoy the Django ORM, and it is perhaps the main reason I still use it today. Right mix of simplicity and power.

we are using FastAPI + SQLAlchemy here.

Are you married to Python? Ruby on Rails is mature, with a large set of useful libraries.

We're not married, but we're "living in sin"... Once python shows you that blocks are always encoded with indentation for the human readers, then braces or beg/end are just static/visual interference on top of your code that makes it harder to read. Going from python back to braces or beg/end is like going from DVD back to VHS. You didn't notice how bad the static on VHS was until DVD showed you a better way. :-D

I guess I've tended to be more focused on how much I can produce and the quality of the ecosystem vs that level of syntax. My limited experience is that other frameworks, languages and ecosystems allow for more productivity overall. The few places I know using Django are not as productive as the orgs I know using either Laravel or Rails. There may be some 'chicken/egg' stuff going on, but... if lack-of-braces means that much to you, that you'll cut off using other tech stacks altogether, so be it.

Heheheh. I'm a code craftsman and certain tools I enjoy more than others. It's OK to love the heft and feel of a model-77 wormdrive skilsaw and also to enjoy the cleanliness and readability of python code. Right tool for the job, but ceteris paribus, I prefer python. The tool feels good in your hands. I regularly code in C++, javascript, Rust, and python so I haven't joined a cult, but yes, I do enjoy python.

funny, I feel the same way about Ruby going to Python - i don't like list comprehensions, for example - feels so crusty and limited compared to yielding blocks in ruby

Is there a framework that does them better? The 1.6 transition was the only one that took more than a day for me.

I'll second this - I maintain a few different Django backends and it's been smooth sailing upgrading most of them.

Yeah. Tons of other frameworks do it better. React for instance. Any framework that actually follows [semver](https://semver.org/) and really cares deeply about backwards compatibility. I've hit backwards incompatibility on almost every upgrade. Not just major 2->3, but each minor release.

I still love love love Django, but I do wish upgrades were easier.

What kind of things have you hit? I think the last time I had an upgrade take more than a few minutes was around the 1.6-era.

(As a reminder that projects vary, I've had more times where I've had to spend time fixing React projects — usually not core but related plugins – so I think there's a heavy element of what you do and how the team works coloring impressions)

Maybe try the new django-upgrade package. https://github.com/adamchainz/django-upgrade

It claims you can either just run it or hook it into your CI. It will automatically take care of some of the upgrade chores for you.

I really wish they'd prioritize multi-column primary keys. I don't have a ticket handy, but it's been out there for many, many years as well. I feel like it would be huge for projects built on legacy database, where you have to rely on existing schema.

Even if it isn’t a legacy project, sometimes you just so happen to have a natural composite key on a table and don’t want to assign a separate single column primary key just to satisfy your ORM.

My understanding is Django 4.0 would include SUPPORT for a redis cache backend using a subclass of django.core.cache.backends.BaseCache

Now they are shipping a built in Redis cache back-end?

That would be so totally crazy that I am having trouble believing this headline.

How about "Django 4.0 will include built-in Redis cache support"

> Now they are shipping a built in Redis cache back-end?

no. actual link as you surmised states:

    This PR aims at adding support for Redis to be used as a caching 
    backend with Django. As redis is the most popular caching backend, 
    adding it to django.core.cache module would be a great addition for 
    developers who previously had to rely on the use of third party 
    packages.

so just a bad link title

I was somewhat surprised to see that the django.core.cache module didn't support an mmap based cache. The default cache is per-process, so it's somewhat limited in effectiveness. Something like Redis solves that, but adds another running process to manage.

In the PHP world, for example, acpu is a popular user-facing cache that uses mmap so that it works cross-process, then you can move to redis if you want to scale across more than one host.

This is a welcome addition but isn't such a big deal... There are great third-party packages (i.e https://github.com/jazzband/django-redis) for whoever needs a redis cache backend in a project.

Myself, I just use good ol' memcached for all my caching needs (https://docs.djangoproject.com/en/3.2/topics/cache/#memcache...). It is rock solid and has never failed me till now!

I think Django needs to "fix" serving static assets in production without the need to muck about with witenoise or nginx settings.

What is being cached?

I don't understand what there is to cache in a web framework.

Is it to cache rendered pages without dynamic content?

If so, I think it is better to let a CDN handle that. Instead of the server that runs the web application.

What am I missing?

Generally the approach is fragment caching... so you're caching some block of HTML that would otherwise hit the DB (Think something like a block of configurable sidebar links, or a "latest posts" block.

HTML, not assets.

Some such setups (like one I wrote for a busy Drupal newspaper site way back when) can support clearing relevant cache entries on DB write, so you're essentially never serving stale data, but can save _substantial_ DB load - and possibly webserver load too, depending on how computationally intensive the HTML generation is.

I guess this is easier than read replicas? I can't imagine there's too much work to be saved by skipping the component building.

Yes, if the queries involve lots of fiddly calculation (for instance, the newspaper system did a lot of fancy stuff around calculating good line breaks and wrapping, or the queries themselves are complicated, involve subqueries, many joins, etc.

DB lookup + rendering vs cache lookup + concatenating. Yes it adds up.

First, there are people that don't want to use a CDN (don't trust them with censorship), or can't (local network).

But you can cache other things that are not the entire page itself:

- database queries

- user sessions

- api calls

- page fragment

- function memoization

What's more, cache backends have the ability to expire things, which can be helpful for all sort of stuff.

The data being cached does not need to be related to the web framework as much as the application logic.

E.g. results from external services, calculations, anything that an application might want to cache.

Am working on a somewhat large Django system, and there is a ton of different things we cache in Redis.

Also HTML template results as another comment mentioned, but mostly something else entirely, related to the application logic.

Database queries are a big one in some cases.

Anything.

What's annoying with Python though is how hard it is to just cache stuff in some dict or other data structure, since the GIL basically forces each request to have it's separate process (instead of separate thread). So non-trivial to share some state application wide without having to store it outside the application.

Of course a shared cache is sensible when there are multiple nodes running the application. But not everything needs a global cache either, or maybe the point of the cache was exactly to avoid a round trip to something else.

At surface level, Redis isn't currently on this page, but will be there when the 4.0 version comes out: https://docs.djangoproject.com/en/3.2/topics/cache/

However, it seems this will enable the community to establish using redis as a best practice. So it will be a bigger deal than adding just another cache backend.

This is good because the default is local-memory caching, and redis can easily beat that. https://docs.djangoproject.com/en/3.2/topics/cache/#local-me...

Django also needs to pin a default Postgres container image, and default NGINX image. The default stack should be extensively end-to end tested and optimized, including an optional patched kernel, cgroup2 settings, and untrusted nsjail worker process for handling media files.

Interesting decision. The mailing list is usually full of "things are better served by third party packages" and divesting core of stuff like this.

Ive not been following the mailing list, however, so I'll have to go back and see what the justification was...

Nuts, I was hoping that this was an evolution of https://github.com/Suor/django-cacheops which uses Redis to cache and invalidate ORM result sets by an abstracted representation of filter values. It works, but has a lot of magic (as is the Django way) and takes some tending-to in production.

Interesting...I recall that Django always set itself apart from Rails by not having a lot of magic. Has this changed?

Django has always had some magic, although much less than ruby/rails. That kind of magic is the reason why it is a pain to package a Django project with py2exe, for instance.

This is cool. Be nice if Wordpress did the same.

Redis everywhere!

Reply on news.ycombinator.com