How Imgix Built a Stack to Serve 100k Images per Second

> Before, we'd have to run processing tasks on a fleet of EC2 instances and wait a few days.

Few days? Why?

> They sure are making it complicated, though.

Seems like a large degree of it is marketing-related.

"Any intelligent fool can make things bigger, more complex, and more violent..."

[Talking purely CPU] You can resize 100k images a second on a 2013 laptop. [Source: the Google guy who serves your images.] A couple ARM cellphones could meet this entire company's [CPU] image processing needs.

Think of what YouTube is doing for videos in the time you read this sentence if you want to put their silly claims in perspective.

Anybody remember when Steve Jobs said the 400Mhz PowerPC G4 Altivec was a supercomputer? But the only code that showed up was a Reverb that took a weekend to run and a couple FFT libraries that had accuracy problems? Good times.

K20s are old GPUs, get a Tesla K80 :)

The apple hardware is either Nvidia or ATI gpus, but the software apis are what are really nice about dealing with some of these things on OS X. I think the Author of the post is confounding new hardware from Nvidia / ATI with new hardware from Apple.

This is just speculation but probably nothing. Imgix should be doing their image processing with GPGPU technology (CUDA or OpenCL) which should be pretty much available anywhere.

Technically Apple has had their Metal API available for a while now but I kind of doubt they're using that for image processing.

These run on the same hardware, so of course, someone could implement Core Graphics on Linux. However, nobody has as far as I know, and I think it would be rather expensive and time consuming to implement that.

It seems it was cheaper for Imgix to use Core Graphics rather than implement it themselves on Linux.

Commodity CDN's will always have some amount of cache misses. For most services that's fine; there isn't too much additional overhead in serving a static Javascript file a few times. If you're doing computational work for each request, you want to try to avoid unnecessary work as much as possible.

It's helpful for us to hold onto master images after fetching -- these images can be large and may not be served to us from a particularly fast source. The speed of light does come into play, so anything we can do to shorten distances is beneficial. This cache layer respects the Cache-Control headers of the original objects, so the behavior there is under our user's control.

There's a similar need to cache rendered output ("derivative images"), because while most operations are very fast they do still add a little bit of latency. This helps with our rendering capacity, but it's really about improving time-to-serve latency wherever we can.

We evaluated a few open source object stores and decided on MogileFS. It largely came down to the simplicity of what Mogile does and how it does it. We figured that we could rip pieces out and adapt it to our needs with a lot less work than more elaborate systems (like ceph). With that said, we didn't actually do an implementation test with ceph, so it may in fact be great for our use case.

Not sure off-hand about the number of servers deployed for use with Mogile, but it isn't a gigantic deployment. We aren't storing anything that can't be re-acquired, so we can adjust our storage to hit particular cache hit ratios rather than storing the enormous long-tail forever.

The first generation of storage servers are 16 core / 32gb / 6x4T systems -- more like what I would use for computationally related storage like HDFS than a large object store. We could stick with this hardware spec, or we might move to SSDs if latency becomes an issue (I personally doubt it will). We might also move to more dense systems, with more like 20-30 drives. That's all based on our metrics and how things perform in practice.

They are more expensive, if you look at mac mini hosting services and compare then to linux you'll see that. But it says that they used them for the graphics processing only. And used "traditional servers" for the other services.

Cheaper than re-writing and re-testing everything to work on Windows/Linux + whitebox servers.

My quick and dirty math from a previous HN discussion: https://news.ycombinator.com/item?id=9510610

Short version is: not too bad, and we get a lot of utility from OS X frameworks, so the tradeoffs are worth it for us at this stage.

This is pretty typical in my experience for StackShare posts. For example: http://stackshare.io/500px/how-500px-serves-up-over-500tb-of...

imgix is very interested in open sourcing things we create, and there are a few smaller projects at https://github.com/imgix now. Hopefully Levee will join them in the future, once we have reached a certain point in development.