How to serve Django Statics (and not go insane) | Hacker News Reader

How to serve Django Statics (and not go insane)

blog.sendhub.com

53 pointsashrust31 comments

Nice post, tons of great advice. However, I have to say that the title is wrong. There's nothing django specific about any of those approaches.

I've tried django-compress and it was a nightmare, the old style synccompress was actually easier to setup and get working for some reason.. Was hoping for a better rolled solution.

Also, the part about cloudfront isn't written very clearly. I had to stop for a moment and think about what it meant. Great idea nonetheless. S3's gzip support sucks. For some odd reason (I don't support IE6..) the gzip from S3 was breaking on IE9. Worked fine on 8 and 10. Broke on 9. =/

Also, I can't tell you how much I hate that "cloudflare" sounds so much like "cloudfront". I've been in at least 6 convos where people were mixing one for the other. Esp, since they solve some of the same problems.

Actually that's exactly what happened to me! :P I thought oh so cloudflare can fix this, eh. I should just reverse proxy my stuff through cloudflare.

Then I realized, oh they said cloudfront. So you could still technically keep your stuff in your S3 bucket, though you'd end up paying more for S3 originating transfer as well. In this case, cloudFRONT (the non amazon one!!) may be the better choice.

Hate that they sound the same. I even get my OWN THOUGHTS mixed up sometimes :(

Hilarious, I wonder how much they gain from devs getting mixed up on recommendations. Probably better for Cloudflare if you ask me. Personally, I don't think it makes sense to use CloudFront if you are already using cloudflare. Its easier to use if your just talking web statics.

Just don't get either confused with "stormfront".

You're right, most of this stuff is not django specific. But since we're using it, we thought we'd let everyone know since its likely the most popular python web framework.

Anyone know what proxies still don't like foo.js?v=1 style URLs? I thought we were past this.

Also, a static build process (a la require.js) solves a lot of these for you in a way that's not specific to Django or any other framework. You also don't end up bloating your web app code with the concern for how static files are to be processed, which I personally like a lot.

Another way to go is to use Google's mod_pagespeed [1], which once again is not framework-specific.

Lastly, you can try another trick where you pre-generate .gz versions of all the files too, to really speed things up. It's nice not to have to do things on the fly and web servers like nginx can take quite a lot of traffic serving static files, so you can hold off on going the CDN route, unless of course geography matters more to you than offloading server resources.

[1] https://developers.google.com/speed/pagespeed/mod

"Most proxies, most notably Squid up through version 3.0, do not cache resources with a "?" in their URL even if a Cache-control: public header is present in the response. To enable proxy caching for these resources, remove query strings from references to static resources, and instead encode the parameters into the file names themselves."

https://developers.google.com/speed/docs/best-practices/cach...

I've gotten https://github.com/cyberdelia/django-pipeline working recently. It's a young project but it's supporting a heavy frontend asset pipeline for us now and working great.

I've tried django-pipeline and it seemed great when I started, but ultimately I had to abandon it because I discovered it would've taken an inordinate amount of work to get it compressing the CSS/JS assets efficiently. Out of the box (just following the documentations suggestions) the django-pipeline compressed JS was 5-10X the size that I get after django-compressor is done with it (and with compressor I didn't have to do any tuning or optimization beyond basic configuration).

At any rate, I'm happy that you were able to get it working, Mike! That is really neat. Do you think you'll write a blog post about how you did it?

You know, I've always found django-compressor to be adequate for my django statics needs. Was there a real need to reinvent the wheel in this case?

Were you simply unable to configure django-compressor (or any other pre-existing django statics library) properly?

A little off topic, but a fault with django-compressor is that if you ever want to use jinja2, it only supports generating the compressed versions on the fly rather than via manage.py - I made a nasty hack that lets us do that but it wasn't easy.

An even worse problem is that you won't be able to transition to using async dependency loading for your JavaScript since there's no way to get the filenames of each compressed bundle that you'll need to pass to whatever lib you're using.

These problems don't apply much when you're just prototyping something, but it turned out to be an awful hole we dug ourselves into when we ran into severe performance bottlenecks with Django's template rendering and serving all our JS on page load.

preach! Haha. Turns out there's more than one way to hang yourself with this stuff. The problems we ran into were similar.

Check out django-require

You make a fair point. I didn't really spend a lot of time trying to get django compressor to work. At some point you have to make a decision about whether or not its worth your time to continue trying to make a library work, or just getting rid of it.

In this case the 2 hours I spent implementing my own solution means it works exactly how I want it to and going forward it will be way easier to maintain.

We got it working but there we're a TON of drawbacks that made it not worth the effort. Here are the top three.

1. The config was overly complex and not flexible enough for our needs.

2. It added extra, and unnecessary deploy steps which slowed down our deployments.

3. Configuring GZIP on S3 and Cloudfront is prohibitively complex.

CloudFlare on the other hand, solves many of these problems for us.

In the article you referenced CloudFront, but now you're saying CloudFlare.. so which is it?

The article has been modified to clarify which service we are using as a reverse proxy and why.

Hang on! So your article says "CDN to cache them (which CloudFront supports)" but you're actually using CloudFLARE ? :O We're all so confused by this! Haha :D

Both CloudFlare and CloudFront can do this.

http://aws.typepad.com/aws/2010/11/amazon-cloudfront-support...

Actually, on looking deeper it appears that cloudfront does NOT support dynamic gzipping.

So unfortunately, cloudflare would be your only option.

Also, google's pagespeed service (in beta) is also an option. Also free, so that's good :D

EDIT: Okay, wrong again. Appears that cloudfront will serve dynamically gzipped content if your origin server responds with gzipped content to the Accept-Encoding header. So origin server = S3 is not a good idea, but origin through your own web server should work fine.

I've been using CloudFlare on a project and have been mostly very pleased with it.

The only persistant problem that I haven't been able to iron out is that some non-US users of my application can't get any of our javascript assets to load in Google Chrome correctly. VPNing into a US IP seems to fix the problem. Has anyone else had any issues like this?

That is very interesting. I am very keen to know if this is a widespread problem.

Have you filed a ticket with CloudFlare or contacted them about this problem?

Yes, I opened a ticket with them a few weeks ago. Still working with them on a resolution.

I've done something similar to the hash in the static url. Used the git commit hash to break cache. Nginx would see the hash in the url and just ignore it. Cloudfront would see the hash in the user and thinks it's a new file.

https://gist.github.com/4129151

I might be completely wrong, but due to HTTP/1.1 and Pipelining, surely it's quicker to serve [small] static files from the same server as your page, because you don't have the overhead of re-establishing a connection?

I've been using django-require in place of compressor and it's fantastic. Also the AMD pattern of loading js seems to be the future.

> breaking anyone who doesn’t have support for GZIP

What client doesn't have support for gzip?

Please label your chart at the top of the post, not sure what the lines are.

How often do you hash the static files? Each time you generate the URL sounds like too much overhead.

In production we only compute the hash on deploy.

Reply on news.ycombinator.com