Batch Processing Millions and Millions of Images
codeascraft.etsy.com
codeascraft.etsy.com
The author spends time tuning the number of workers and the number of OpenMP processes per worker for GraphicsMagick on his 16-core machines. Isn't this type of tuning a waste of time? Even using just two cores instead of one seems to introduce inefficiency. Wouldn't he have been better off just using 16 workers, each compiled without OpenMP so that it would run serially (and more efficiently)?
This is the sort of stupid batch task that can be done with a shell-script using all the machines in the office after hours. Cut out the middle ware, cut down the threads and fancy IPC, and hunker down with good ole minimal number of processes on a bunch of machines.
Heck, they might even see a big win if they use a different "server" memory management algorithm that's more tailored to batch, not-so-responsive applications.
$ cat filelist.txt|xargs -n 1000 -P 16 ./myconvertprog
find images | parallel -j+0 –trc {.}.out -S server1,server2,: “do_stuff {} > {.}.out”
Watch the intro video for GNU Parallel: http://www.youtube.com/watch?v=OpaiGYxkSuQ
Our problem was that we were making a composite image but sometimes the 'background' image content was shifted to the left or right. (e.g. All images were the same size with a white background but some were made from different source template and so the actual content sometimes started at pixel 25 and other times at pixel 35.) To make it worse they were all JPEGS.
I ended up writing a program to find the bounding box of the content (with a fudge factor to account for the JPEG dithering) and identified the bad images to be fixed by hand. It was a nice diversion from SQL and ASP.
It links directly with libjpeg, libpng, etc. instead of via some imagemagick bullshit. Instead of being fucked if what you want to do doesn't line up with an existing magick command, it gives you the tools to do it yourself, and even pretty performantly since it can give you numpy arrays.
"In fact, the research phase of this project took longer than the batch processing itself. That was clearly time well spent."
I'm not so sure about this. A sufficiently parallelized but otherwise unresearched script could have been created in a fraction of the time. So you would just generate the correct thumbnails on new uploads and then let the batch job run for, say, two weeks. That naive approach will take you around a days worth of development time.
The time you gained by doing it the naive way you then put into the rewrite of that legacy component that prevented you from on-the-fly generating the images.
What you have done here is, IMHO, wasted time in a legacy solution and you wasted hardware resources for storage of the additional pictures of which probably only a minority will actually be seen anyways.
Don't get me wrong: I'm sure you had a lot of fun and there's little more exciting things than seeing your code perform magnitudes quicker than the initial approach. But in the context of your quote, I have to disagree.
Unless there's more background you didn't tell us about.
One thing I'm skeptical about:
"We found out, almost by accident, that using the previously down-sized “large” images resulted in better quality and faster processing than starting with the original full-size images."
Obviously, the processing speed will be faster with a down-sized image, but I can't see how starting with a smaller image will give you better quality. Unless the original image was so large that downsizing caused a lot of detail loss (in which case the image should have been sharpened).
I'd almost be tempted to spend a few more hours and hook it up to mechanical turk rather than actually doing the classification myself. (Partially because I get bored easily and partially because I have all the artistic appreciation of a mole rat.)
This system doesn't scale out without ad-hoc partitioning. I think Etsy's approach here should have been informed by the New York Times' project a few years ago. They converted 4 TB of scanned tiffs (their article archive) into PDFs on a hadoop cluster running on ec2 and s3. Parallelizing the process across hardware nodes is the scale free way to do this, exactly what hadoop was intended for.
All that said, I know there are smart folks at Etsy. I'll give 'em the benefit of the doubt; there may be a good reason not to go that route but this write up didn't make that clear.
This looks like a problem for which I could write the code in 15 minutes. Assuming Markham's Rule of Estimating (double the number and go to then next larger unit), that's about 30 hours of work.
This is a problem just begging for a functional solution. If you've got a language that already has a bunch of libraries, like .NET, you just wire it up, pipeline it, and send it out to as many workers as you want. The image tweaking, worker nitpicking stuff just isn't worth the time. If you're doing millions of everything, then I'm sure you've already got some broker parallelization thing going on. Shouldn't need to redo that every time. If you want to be truly anal, do some tests to determine the fastest way to compress. But aside from that, it's all big old hunks of immutable data splaying out into the universe, running on as many cores as you need.
I probably missed something. These things always look simpler from the outside.