Nginx: a caching, thumbnailing, reverse proxying image server | Hacker News Reader

Nginx: a caching, thumbnailing, reverse proxying image server

charlesleifer.com

209 points_pgmf57 comments

His setup is vulnerable to hash length extension attacks:

  secure_link_md5 "my awesome secret $uri";

This means anyone can extend the URI (eg. adding "/../../../some/other/file" at the end of the filename) and compute a valid key, and force the server to access an arbitrary image file on the backend. If the backend is a private server not publicly accessible, then this may be a security issue.

This is why in the ngx_http_secure_link_module's documentation the secret is appended (instead of prepended), see http://nginx.org/en/docs/http/ngx_http_secure_link_module.ht...:

  secure_link_md5 "$secure_link_expires$uri$remote_addr secret";

The Nginx doc is also at fault because it fails to explain why it is important to append the secret and not prepend it. I just emailed security-alert@nginx.org to let them know.

Length extension attacks work for both append and prepend. I don't know if you're right, or whether or not either of them are vulnerable, but the correct solution to this problem is to use the HMAC[0] construction.

[0] https://en.wikipedia.org/wiki/Hash-based_message_authenticat...

Yes some other attacks can be performed if the secret is appended (we don't call them "length extensions attacks"), however they do not apply in this guy's case because the attacker does not control the URI that is hashed.

But yes, when in doubt, and to future-proof your code, it is good practice to use an HMAC anyway. Or use SHA-3 which is not vulnerable to any attack when prepending/appending a secret.

Thanks, I've updated my post to append the secret key.

This is a cool little project showcasing some of NginX's lesser known modules in action.

However, I have one nitpick.

The caching server's main location is a bit of a cargo-cult unoptimization. They have specified:

    location ^/(.+)$ {

When the equivalent:

    location / {

Is shorter, clearer, and does not incur a regular expression capture on every request that is never used.

The difference is that the last one also matches requests to "/", while the regex ensures that there is at least a single character after the "/".

That allows "/" to be used for other purposes, e.g. redirecting to the main page of the site, or displaying a nice error page. (However, in this case it simply shows the default Debian Nginx installation page: http://m.charlesleifer.com/ )

For that use case, I would recommend two locations like so:

    location = / {
      # Just whatever happens for "/" requests goes here
    }
    
    location / {
      # Every other requests ends up here
    }

Actually, Nginx are way ahead of you. Here's the way to handle that.

    location = / { [ configuration A ] }

    location / { [ configuration B ] }

To learn more about this, I recommend http://nginx.org/en/docs/http/ngx_http_core_module.html

The Nginx docs are not always great but they are far more correct than most blog posts about Nginx so that is why I have turned to use only the official docs when I have a question about how to do something with Nginx.

I (I'm not the author) shamefully do some cargo cult Nginx because I don't understand how the location matching really works.

My ideal web server would have a configuration that resembles a decision procedure in a programming language, so I can have a mental model for what makes it choose one route over the others, and how settings are combined, etc.

Somehow Nginx repeatedly makes me and my coworkers very confused—are we crazy or just mildly incompetent? Should I read an Nginx book?

(One time I wanted to do something fairly straightforward, I think using a conditional inside some block, and when I looked in the manual it warned very strongly that this might cause segmentation faults. I was like WTF?)

NginX has come a long way and so has the documentation: http://nginx.org/en/docs/

I highly recommend this document to start, which describes how NginX processes a request: http://nginx.org/en/docs/http/request_processing.html

For getting the mental model right, you have to understand that NginX configuration is declarative, like SQL or PROLOG. That means as opposed to an imperative or procedural language you might be more familiar with, in NginX you describe "what" you want to have handled, not "how" and NginX worries about the "how" under the hood, for the most part.

Of course like any abstraction it sometimes breaks and leaks, but this is practical software we're talking about. One specific big headache for people is when using "if" in NginX, which is a decidedly imperative construct! As such you do have to tread lightly with it; observe If Is Evil: https://www.nginx.com/resources/wiki/start/topics/depth/ifis...

Thank you, I've updated the post.

One of the great benefits of Google Appengine and Google Cloud Storage is that you get this for free. Merely changing the size param at the end of the url resizes the pic. And since this all happens on the edge cache, it never touches your app, and you don't get charged for bandwidth.

http://lh4.ggpht.com/sTP-3BqnirkHm40qfb496w85A1bf7BpeXthFJ92...

I found the nginx native image library completely inadequate a few years ago, so I had to use the perl module to connect it to imagemagick libraries instead. Much higher quality results.

Not sure if they have improved the native library but I doubt it has the flexibility.

The important part is caching the results because of the expensive cpu time.

ps. googling about the current state of this came up with an interesting nginx module which talks to imagemagick (or gd) directly https://github.com/cubicdaiya/ngx_small_light

Agreed

I second this, for a recent e-commerce project we started using nginx's image_filter but the quality of the resized images was unacceptable.

We ended up using Thumbor, which was able to give us much better looking images (but at the cost of being extremely difficult to deploy on the CentOS 6.x servers we had at that time).

We use a similar set-up at work, but we don't do the image resizing in nginx, rather we have a python backend that it proxies up to, that runs a tiny wsgi app that uses requests & pil to do the resizing and transforming. It uses proxy_cache_lock to dedupe requests to the backend.

If all you need is crop or resize, might as well use nginx in my opinion.

There was one annoyance we have to fix that I don't think the nginx module supports & that's rotating images based on their exif orientation (mobile devices love to upload landscape images w/ an exif-orientation in portrait).

Python is definitely an OK solution for this as it turns out. We resize a few million of images a day on-demand on two n1-highcpu-8 GCE instances. Although, they could easily be a fourth the size, as CPU generally peaks out at 20-25% during peak hours.

We've actually tried more specialized services for this, like sharp + a threadpool using node.js and it actually performed terribly. As it turns out, blocking the gunicorn event loop by processing image resizes created a good amount of back-pressure to load balance incoming requests to each worker process.

That's a good point. The nginx module can rotate, but I don't know of any built-in support for rotation with respect to exif. You could also use lua and the lua imagemagick bindings and probably see much better performance though.

I'd love to see some performance benchmarks for Nginx's caching vs Varnish (memory, reqs). Even if Nginx can do the same thing, if I need more RAM to do it, or it handles less requests - I'd rather keep Varnish.

A few years ago everyone was jumping on nginx saying it's the 2nd coming of Jesus and that being a simple web server without dynamic config or modules to do processing made it better than Apache because it's "fast".

How many of those same people are now going on about how nginx has finally got some support for loadable modules and built in processing like this?

I like nginx because its config is about fourteen times easier to follow than apache's.

> How many of those same people are now going on about how nginx has finally got some support for loadable modules and built in processing like this?

I would doubt many of them are because early adopters tend to also be the kind of people who don't mind compiling their own webserver for whatever reason: time, patience, availability of dedicated hardware to do it on, etc.

The sort of people I saw jumping to nginx were largely not the sort of people who would compile it themselves.

These are often people who would compare Apache + mod_php to nginx + PHP-fpm, and not understand why it's not a realistic comparison of the web servers.

We do something similar : when nginx doesn't find the file for a given format (width and height are in the url), it's a 404 and we use the 404 handler to proxy to a php app that does the resize/crop (and we have a list of the formats we use more often in the app so we store the generated thumb on the disk), the resize/crop operation is centered around coordinates that we determine with opencv (it detects faces and/or focus points) when the user uploads the picture.

We use varnish in front of nginx

If you use a CDN such as Cloudflare, you can do a similar trick but forget about the proxy cache (and associated storage needs) - as long as every image at every size gets a unique URL and you control http cache headers correctly, Cloudflare will store copies of every image size ever requested across their CDN network.

In terms of this blog post, you'll only need the "resizing server". This is a very simple and stateless way to get image thumbnailing to work. I'd also personally not bother with all the API key stuff - how 'malicious' can it be to generate a thumbnail of a public image?

This will even work with 3rd party images (the case of the app I used this for, it was product icons from Apple's App Store). If you somehow want to use images hosted elsewhere inside your app, but worry about speed or about hitting their servers too much, proxy them with Nginx and let Cloudflare handle the rest.

The blog post implements the "resizing server" on nginx to prevent the hits on application server. One alternative is to use AWS Lambda which has a sample code solution precisely on S3 image thumnailing.

That's not quite true. Cloudflare (like virtually every CDN) only caches files at the edge server that handled the request. And they purge the files after a short period of time--even if your cache-control headers ask for a longer time. Most won't tell you how short (including Cloudflare), but some CDNs will delete files after as little as two hours.. 24 hours is very common. Most CDNs also give you no visibility about what is cached and where.

In Cloudflare's case, they operate 76 data centers. So every "short period of time" x up to 76, your server will need to regenerate each image.

So while a CDN is useful, a proxy cache can still be helpful.

(None of this applies to NuevoCloud though.. NuevoCloud has a global cache, with dedicated caches for each customer. So it's entirely possible to keep an image at the edge indefinitely)

Might want to throw a disclaimer on the end there?

I'm curious. $99/TB seems like an incredibly high price to charge for a CDN that only has 10 points of presence globally.

Maybe this is typical with 'website accelerators' as opposed to CDNs that focus on caching truly static content, but even so features like dynamic acceleration and global caching are par for the course with any traditional CDN.

Cloudflare, which I use mind you, is non-traditional and their business model seems more suited to DDoS prevention and edge SSL than anything else.

So what sets NeuvoCloud apart?

Dynamic acceleration at Cloudflare (Railgun) is $200/website. Origin shield is $200/website at MaxCDN (could be wrong, but I don't think Cloudflare has an origin shield equivalent). Both are included with NuevoCloud, and you can use them on multiple websites.

Global Cache does not exist at any CDN that I'm aware of. We spent a lot of time developing this. It's true other CDNs will cache files at each POP, when that POP sees a request for it... but that isn't what we mean here. (I understand the confusion.. I get a lot of questions about this, and we should probably rename it.)

Global Cache means we have a single cache that is used/managed globally. Let's say a visitor in France gets a cache miss for a new file on your website. Every PoP at NuevoCloud now knows of that file and has it cached. So if the next visitor to your website is in Tokyo, they'll get a cache hit and the file will be served from the Tokyo PoP, even though it's the first time the Tokyo PoP has seen a request for that file. A cache hit at a traditional CDN means that PoP has the file; a cache hit at NuevoCloud means every PoP in the world has that file.

The cache is managed for each customer individually with guaranteed space at each PoP. Which means, even if you have a small website that receives no traffic.. you can still keep your files at the edge. The only other way to get this is to pay for a dedicated CDN.

Finally traditional CDNs send requests from the edge directly to your server (client > edge node > your server) which is a long distance connection. Connections through NuevoCloud are routed through our network: client > edge node > edge node > server. So both the client and server are talking to an edge node near them. This speeds up SSL and connection negotiation between the edge and your server. We also use this setup to dynamically route requests around high latency, network partitions, etc.

I put our 10 points against other CDNs daily... and it's faster. You would be surprised at how dumb the average CDN is when handling requests. This is the reason they scatter edge nodes all over the place.

Hey, thanks for the detailed comment spam. I might become a customer in the foreseeable future!

I was wondering if an additional cookie based security token is possible with secure_link module. (show image only if session cookie is present)

Apparently, it is possible, great! One of the google results: https://gist.github.com/hilbix/5921589

Secure_link module shouldn't be used for anything security related. I think it uses plain md5 and without even an hmac.

I wrote a similar article a few years back that employs Lua and imagemagick bindings: http://leafo.net/posts/creating_an_image_server.html

Yeah, I'm pretty sure I linked to your post. I was going to use your script before I realized I could do everything with plain old nginx.

Could someone here explain to me how it's possible that a bug like [1] concerning the main functionality of nginx (static file routing) is open since many years, and yet nginx keeps its popularity? Is that really the best we've got?

[1] https://trac.nginx.org/nginx/ticket/97

Probably because every single popular project that has existed for a long time has bugs that have been open for years, so the fact that a bug has been open for years says nothing about the quality of those projects?

I disagree. There are bugs and then there are bugs. If a web server messes up static paths and there is no reaction from the maintainers in 4 years other than putting the prio on minor, I have serious trouble trusting the code quality of a project. But so far I haven't really found something that would replace nginx, especially when it comes to serving wsgi apps. Hence my question "is that really the best we got?".

https://trac.nginx.org/nginx/changeset/2797b4347a2af4e8fd46d...

So, do I get this right - the bug has been fixed since mid 2015 but the ticket wasn't updated?

Does the image_filter module use the GPU?

Image filtering is an expensive operation for the CPU, with large latencies as well.

Actually GPU latencies and transfer costs dominate most simple image filtering operations. A GPU might be fast once the data is in GPU local RAM, but if just transferring the data back and forth takes 3x the time running the filter locally, what's the point?

You're better off performing it on a CPU, of course SIMD optimized.

Besides, web servers probably won't have GPUs anyways.

Intel is building entire Xeon lines with embedded GPUs for serving web imagery.

And these Intel Xeon's with embedded GPUs don't need to transfer data to the coprocessor, since they operate off of system memory, so transfer latencies are a non-issue.

Similar idea but more flexible image manipulation: https://github.com/hcarvalhoalves/django-rest-thumbnails

I wish the author had gone into more detail about why they chose to stop using Varnish.

I was already using Nginx to proxy to my Python sites, so getting rid of Varnish meant one less thing to fiddle with.

I stopped using varnish because, after all, my site is just a blog and Nginx is just fine for my needs. It has a single backend server, so there's no load-balancing. Basically, I was already using Nginx to serve static files, and my caching logic was so simple it was actually easier to move it into Nginx.

Perhaps better a better question is why you were using Varnish in the first place?

I'm genuinely curious because I've never stood up Varnish before so maybe there's an anti-pattern / gotcha to learn from.

Well without some sort of caching, his site very well could fall over when being on the front page of something like HN. Putting Varnish in front a site, even with something simple like a 1 minute cache on everything, makes you pretty much immune from having "large amounts of traffic" being a real problem.

Exactly, thanks frank! :P

By removing varnish aren't you now exposed to that risk once again?

I feel like I am missing something.

He would have been if it hadn't replaced the Varnish functionality with nginx.

He's basically just moving functionality from Varnish to nginx and thus simplifying his stack a bit.

Nginx can cache static content of any type, so it can act as varnish for both his blog content and images and resize them on the fly instead of relying on multiple image sizes on disk or a separate app to serve images. Nginx is a highly efficient webserver when it is already holding the items in cache (and otherwise, it passes off dynamic content just fine but that is then up to the underlying application to perform as well).

Fewer moving parts is generally preferable, unless you can demonstrate a significant advantage in adding it.

Very cool! I've been playing around with nginx-lua to handle our shortened URLs without having to invoke the upstream application.

Clever and thanks for sharing. It was a good read.

Reply on news.ycombinator.com