Django 3.2 – News on compressed fixtures and fixtures compression | Hacker News Reader

Django 3.2 – News on compressed fixtures and fixtures compression

paulox.net

86 pointspauloxnet47 comments

If it wasn't for Haskell's IHP framework then Django, Ruby on Rails and CakePHP would be fighting to come dead last in the techempower composite benchmark https://www.techempower.com/benchmarks/#section=data-r20&hw=...

Django, RoR - they're almost 2 orders of magnitude behind the fastest frameworks on the composite test and they're an order of magnitude behind the majority of Java / Go - i was going to say C# here but i see asp.net is much faster than the surprisingly homogenous performing "usual suspects" of enterprise web frameworks.

And yet, if i had to launch a site tomorrow, it'd be Django i'd reach for - and with zero hesitation. Instagram, Youtube, Github, Facebook, Dropbox etc. etc. all launched on these "slowest of the slow" stacks and then optimised. Did anyone start out on a lithium (c++) or actix (rust) or other incredibly high performance stack?

This is all a big diversion anyway, I think these benchmarks measure the wrong thing entirely. Instead - show me some measure of how easy & cheap it will be for me to change or add a behaviour to my site 3 years down the road if i've stuck to the prescribed idiomatic approach.

Some of it may be B2B vs B2C, and volume of usage.

I am B2B. My server costs are like 1% of my revenue using Django. Okay, I could shrink this by 95% by using raw C.. but who cares?

Now if I were an image hosting CDN and Django bill was 50% of my revenue.. it might be a different story.

I think it's the classic right tool for the right job story..

You are a veteran performance software engineer, besides your entrepreneurial bona fides. What is the number you start migrating off django? What is the number you wished you’d picked something else? Is there a value to Django you could have got with a different Lang without the downside?

I’m tempted to go FastAPI over Django on my next project, even though I’ve been a big Django user. A little more performance and a little less coupling, and maybe you can survive longer as a Python monolith. Then convert to Rust/Go/something in the unlikely event of wild success.

Why do you think it reduces coupling? You can use Django apps to have decoupled Django services that all run in the same gunicorn webserver

I mean less logical coupling inside the app. It's more in the family of a collection of libraries versus a batteries included framework. I'm considering that may be the best of both worlds. I never really liked the recreating the universe feeling of Flask.

Why? The FastAPI library ecosystem is anemic.

Django is fast enough. You'll be spending at most a fraction of an FTE salary in hosting even with the best of success, and if it comes to more than that, you'll be likely paying for a DBA or for an improved caching strategy for performance.

> The FastAPI library ecosystem is anemic.

That's kinda the point.

If your business is making webapps for clients, the fact that you can pip install django-half-the-feature is awesome. What you trade for that is a big dependency tree whose packages sometimes snake their way through your whole codebase, since it's all Django anyway.

FastAPI is its own thing, but it's also heavily based on some other big libraries that also have their own support.

And anyway, what I was saying about performance and decoupling are probably just rationalizations. I focus on writing APIs, and that looks nicer to do in FastAPI than how I've done it in Django. I also really like type hinting, and that's a big part of things. Right now, I'd just like to write more Python and less Django.

The biggest challenge I encountered when doing this exact thing was once you get beyond bootstrapping the project, and start implementing your first feature.

Great, you need an ORM, what do you choose? Now you need a serializer, what do you choose? What about validation, what do you choose? Authentication, what do you choose?

A lot of these questions are boring and are not worth spending time thinking about. Almost all choices will leave you with some amount of headaches though (as will choosing Django).

The question to me boils down to... what questions am I interested in solving?

By "doing this exact thing" do you mean making an API with FastAPI? Because you seem unfamiliar with its features.

> Great, you need an ORM, what do you choose?

SQLAlchemy. It's very popular.

> Now you need a serializer, what do you choose?

Pydantic, which is included. (Django doesn't include serializers).

> What about validation, what do you choose?

Pydantic.

> Authentication, what do you choose?

The included auth library.

In a couple of attempts, I have worked with Flask and Falcon.

My point isn't that there are solutions, my point is that you have to make decisions, all of which are likely to have rough edges.

And my point is that for your specific examples, you don't have to make decisions. Django and FastAPI both have 3/4 of the features you listed included.

FastAPI is a tool made for making APIs. Django is not. So in that specific case, Django actually requires more decisions overall.

Django Rest Framework is the one-stop shop for APIs. Probably the most deployed API framework ever.

DRF to FastAPI is a more direct comparison. DRF is what I spent most of my Django time in.

And if you like DRF, the author (Tom Christie) wrote Starlette. FastAPI is based on Starlette.

You might want to check out https://github.com/vitalik/django-ninja. I much prefer it over restframework.

Wow. vitalik was one of the first Django freelancers I hired when I was getting started. Some of his code is still in production for one of my clients.

(Vitaly - if you're watching. Hi!)

interesting, the benchmarks include Flask which is a non-async framework, but not FastAPI

Thanks! That seems very well suited for my stated goals. (Which after conversation here I realized weren’t exactly my real goals :P)

What makes the FastAPI to be fast that other frameworks? Is it cos of async? There is Tornado too, I am curious how do these fair against each other in terms of performance and productivity

We serve one of the largest sites on the internet (top 2,000 last time that alexa's top sites worked, 25k requests per minute) with Django and a tiny web server budget (~$1k/mo). We serve mostly static files, but all requests hit Django at least 2-3 times. If it works for us, it will work for you :)

Django (and pretty much any other framework) is usually "fast enough"; typically you'll hit the limits of your datastore first, and even if you reach the point where the framework becomes an issue it's trivial to scale horizontally due to its stateless nature.

Yup, same story on Ruby/Rails. Yes, its true that our median response times could be faster if we spent less time on Ruby CPU burn. But its also true that this improvement would still be a rounding error for many of our slowest requests, which are dominated by calls to databases, external services over HTTP, or both.

I hear this all the time but in most of our apps average query time is less than 5ms in postgres. And it scales with threads, so it's more like 30k queries per second.

We have a single Postgres instance and serve thousands of requests per second. You're going to be scaling Django way before that.

I've never used Django, but is it really that much easier to manage than a modern Java framework like SpringBoot or Quarkus? It's pretty fast to throw up a REST endpoint, and annotations have replaced much of the old boilerplate.

Yes, it really is. SpringBoot especially has so much unnecessary complexity built in. I'd dread having to debug any non-trivial application written using any of the most popular Java frameworks.

I've recently had to patch a site built on Spring Boot. Coming from Django, it's a disaster.

Spring Boot basically attempts to turn Java from its statically compiled origins to an annotation-ladled mess full of runtime reflections and compiler pluginitis.

If you're doing that, why not just go the whole hog and use an actually dynamic language and a framework that plays up the language's strengths instead of working _around_ it?

Debugging Spring is easy if you know the framework well. That's true for any framework

I don't find following stack traces through a bramble patch of annotation laden garbage calls and pointless abstractions to be a good fit for the word "easy."

SpringBoot epitomizes the worst of the marriage between an OO fetishist and an enterprise "architect."

I wanted to put together an API to open our front door from our phones. I spent way more time putting the phone app together than popping open a VPS, installing Django and the Django Rest Framework, and coding to handle the requests.

Django is stupidly quick to do "standard" things. And Django's "standard" extends a long way. And Django understands a lot more about "standard" than you do until you've been coding your app for a while.

Yes, Django and Python are a bit "boring" today. For those of us with a job to do rather than resumes to build, that's what we need.

Looking techempower's benchmark code, specifically the "dbs" and "update" functions (which are weighted highly), I think there's some optimization that could be done, like using .values() and .update() in more places.

https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

Django ends up getting benchmarked against "uvicorn" which just uses raw sql, so yeah, of course raw sql is faster. Django has a raw sql option too, but they're not using it for the benchmark.

https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

In Java at least, the massive Hibernate ORM is only about twice as slow as raw SQL. Which is still orders of magnitude faster than Django. The lack of performance is far deeper than ORM usage.

Is anyone making extensive use of static fixtures? If so what do you like about them vs. using setUpTestData on dynamically generated models (optionally using something like FactoryBoy to make your fixture code more flexible)? I found static fixtures to be brittle and hard-to-maintain, because you need to manually edit them, or have code to regenerate them, every time your schema changes.

Recently I've been tempted to revisit them and build out a manage.py command to run the domain scenarios that generate these fixtures (to make it easier to keep them up to date), since when testing DB migrations you often actually do want to have stale/old-schema DB state, instead of the default of using the post-migration code to generate your test fixtures. But I'm still not sure I'd prefer them in many tests outside of the migration-testing usecase.

We use them for tests where we a) need representative data that has some sort of interesting format/dependencies that could be tricky (or slow?) to capture in factories, or that we need enough of that writing out ~100s of factories would be a bit nasty to maintain.

I don't think we'd miss them a lot if we didn't have them for this, but they are handy for this. We use setUpTestData and factories or models directly for most of our testing.

That said, I've just written a new tool for exporting a subset of production data for local use (we have a 2TB database but want a <5GB realistic looking dev database, we export a subset, anonymise, maintain FK integrity, install a bunch of Factories for special test cases). I initially started with loaddada/dumpdata but found them inflexible so I'm using the same de/serialisers with custom code around them. That has been somewhat useful, and if that's the level that this compression code is written at I'll probably use it.

> writing out ~100s of factories would be a bit nasty to maintain

I've found that I really like Model Bakery (https://model-bakery.readthedocs.io/en/latest/) a lot more than Factory Boy for this reason - no factories to maintain. It's very intuitive and easy to use.

In a small project I had, I was using fixtures to populate the local dev environment.

``` make dev/init ## runs compose in dev mode make dev/populate ## initializes data ```

would give you an "initialized" data, ready to start working on. When the project has some data in it, I consider it a closer reflection of prod. If you can (data is not sensible), you can update the fixtures with data from prod.

> If you can (data is not sensible), you can update the fixtures with data from prod.

Just make sure all devs use the same days from prod, then. I once was consulting for a dysfunctional team and everyone having different test data drawn from prod made reproducing even easy bugs hard and impossible for more complicated bugs.

Not extensive, it's kind of a last ditch then when I can't generate random data. For example, I have some "integration" tests that I spin through and generate a huge pile of objects, and validate them against XSDs, to make sure the XML I am making is valid. I don't really want to factory boy this, I just want a smoke screen to make sure things are sane.. and as XSD changes, I can re-run the test and catch weird stuff quick.

But this is like... < 1% of my total tests.

I've used them in production before inside of AWS beanstalk (.ebextensions). Was pretty much a static django site on top of a solr index. There was a hidden django where the content team would work on the content. When it was ready, we would dumpdata on the backend server and upload to S3. New beanstalks would grab the file from S3 and loaddata.

I use Django test fixtures in one way!

I have a series of E2E tests with Splinter [0] that go through setting up a variety of things from scratch and testing core features. Instead of having really long, complicated test that goes through the steps, I instead wrote a little helper function that "freezes" the DB after each step as a JSON fixture (which is then loaded by the next test), to allow me to break it down into many smaller tests that "start" from various reasonable database states. Works pretty well!

I don't use them however for unit tests or any non-E2E tests. I just use helper factory functions with hardcoded test data.

[0] https://splinter.readthedocs.io/en/latest/

Not including Zstandard is a lost opportunity -- it is far more appropriate than XZ or LZMA for real-time applications like serving web pages. Should be easy to add, though, and likely would be a very nice compromise between space, memory, and speed (including tuning various levels of Zstandard compression).

gzip is basically never worth using anymore except for backwards compatibility; zstd is both faster and more space efficient.

lz4 similarly is worth adding for utmost speed but I tend to think zstd is better in practice (particularly when the control plane is Python where perf is somewhat limited anyway).

> Not including Zstandard is a lost opportunity -- it is far more appropriate than XZ or LZMA for real-time applications like serving web pages.

The article is about offline creation of compressed data dumps and not about real-time application.

The Django code here use only compression algorithms already present in the Python standard library to avoid unnecessary dependencies.

Looks like .gz is basically a free lunch in terms of CPU/memory impact with a decent size savings.

I was surprised to see xz and bzip2, but not LZ4 or Snappy.

the algorithms chosen are a mix of "most common" and what comes out of the box with python's standard library to avoid extra dependencies.

Exactly. I replied also before reading you answer.

The Django code use only compression algorithms already included in the Python standard library to avoid unnecessary dependencies.

You're totally right. But you can choose also to use xz or bz2 for smaller files if you have spare cpu/memory.

Reply on news.ycombinator.com