Batch Processing Millions and Millions of Images | Hacker News Reader

Batch Processing Millions and Millions of Images

codeascraft.com

29 pointsGhotiFish19 comments

"But at this rate it would take 170 years to resize all of those images.

But here’s the spoiler… We did it in nine days. Every single image."

TL;DR: Computers are faster than humans.

This is an old post and it suffers from some shiny rhetoric, but it was very technically relevant for work I did recently.

I had a similar problem as the OP: millions of images stored on S3 that each needed to be rescaled to a set of four new dimensions. Etsy's work encouraged me to dive in with the hope that I could accomplish my task with relatively little pain.

Indeed, I was able to write some multi-threaded Python that ran on a single XXL EC2 instance and was able to rip through every image in less than 48 hours.

Thanks to Mike, and the folks at Etsy, for posting about your experiences!

Hopefully this comment will find it's way to the top instead of the snarky one that's sitting there now.

Why use per-image multithreading at all? Process one image per core.

This article is from 2010.

I would of loved to be more precise with the title, but HN has really restrictive rules about it.

It's also been posted to HN previously: https://news.ycombinator.com/item?id=1502179

ok. I don't understand HN's submission system at all.

I was looking to submit an article [1] that was an intensely fun read. Before I did I looked for previous submissions, and found this: [2] and this [3]

Given the seriously low scores these stories had vs how good the article was, I assumed it just got unlucky, so I posted it again.

In response to my POST, Hacker News sent me a 302 redirect to [3].

Ok, that must be because the url refers to the same story. Which I figured was some automatic thing HN does to avoid reposts. Given that the OP url hadn't obviously mutated in any way, I knew I didn't need to check, so I submitted again... and nothing, that was fine.

So I have no idea how hacker news works. I plead innocent by way of insanity.

1: http://www.datagenetics.com/blog/december32011/index.html 2: http://news.ycombinator.com/item?id=3971657 3: http://news.ycombinator.com/item?id=3474763

Both of the HN links refer to Battleship: the Movie? I don't really understand - isn't this story about some guy at etsy figuring out how to do his job?

In my experience, Etsy has a tendency to do average through questionable work and then spend an above average effort glorifying themselves about it.

NFS, seriously? Wow, weak sauce.

you know. I wish you had expanded on your complaints, because I am not familar with the problems associated with it. It did prompt me to go looking, though.

I turned up this: http://www.time-travellers.org/shane/papers/NFS_considered_h...

good read.

Well then, tell us about the system YOU built to batch process millions of images that doesn't use something "weak sauce" like NFS.

In the real world, you use what works, not the flavor of the month.

Given that for images one does not need to do random I/O, any kind of object storage system will do much better than NFS or any other kind of network filesystem with POSIX semantics guarantees. And no, in real world, you don't use what works, you come up with a design other people won't have to painfully maintain for the next 5 years.

Sure, I built a system for a company specializing in image processing and uploads/sharing called Eye-Fi.

We used MogileFS for image storage.

Resizing millions of images was a daily task.

NFS is not the right tool but is frequently mistaken for such.

NFS can be the right solution, depending on the problem. Storage backend for serving live images to users? Probably not. Taking uploaded images and storing a copy (after processing), maybe. Depends on the amount of IOPS you expect to need to throw at those copies.

I haven't seen any posts from Etsy on their normal image pipeline, so it's hard to say. It sounds like this was a one-time conversion. Their live image serving is on Akamai, but nothing says they're back-ending the origin servers on NFS.

I have to imagine with the amount of ops talent at Etsy, they're not using NFS because they're too dumb to know about a better option. I'm guessing John Allspaw knows a thing or two about how to serve images at high volume.

The thought that Etsy had good ops people was what lead to my surprise in relying on NFS for anything of the sort. Two mitigating facts: one, this article is from 2010, two, this may have been a one-time thing. Still, surprised and dismayed that they would let NFS into production.

Etsy grew quite quickly, after having been founded by people with little experience running a website at scale. Once an experienced and competent team came on board, I believe they spent a lot of effort recovering from poor decisions made by inexperienced people under duress.

>Well then, tell us about the system YOU built to batch process millions of images that doesn't use something "weak sauce" like NFS.

dmourati was being a jerk but boy, this particular response is hilarious. I suggest Googling his name with some image processing terms if you feel like eating crow.

Reply on news.ycombinator.com