Comment by skuhn | Hacker News Reader

Comment by skuhn | Hacker News Reader

Parent Full thread

(I'm the datacenter manager at imgix and wrote this post)

From a pure hardware perspective, I would love to move this part of the service to Linux systems with GPUs. I spent some time evaluating this before we committed to the Mac Pro solution -- built some prototype hardware and did a cost analysis. It just wasn't the right move, because of the engineering cost for us. OS X's graphics pipeline is really strong, and we've built a lot of cool things with it. There is no analog whatsoever on Linux -- we would have to commit a lot of resources to re-build what we already have, and it would in the best scenario not be a customer-visible change. As a lean startup, we have to be ruthless with the work do: if it doesn't move the needle for our customers, it's probably not the right thing to do right now.

So instead, I've spent some time (and engaged with partners like Racklive) to get the Mac Pros to be as operationally acceptable as possible. This rack design and the chassis we designed go a long way towards achieving that goal. Airflow is taken care of, and the rack hits my power quota almost exactly (at full load). Cabling and networking and host layout follow our patterns from our conventional server racks. USB and HDMI ports on the front allow me to easily use a crash cart.

The lack of IPMI is my biggest operational headache. We have individual power outlet control and can install the OS over the network, so that's something at least.

The OS itself is also challenging. I'm not a fan of launchd. Finding legitimate information about how to do something on OS X is pretty tough, given that most of the discussions are focused around desktop users (who may be prone to pass on theories of how things work rather than facts). We've gotten it to a point where things work pretty well -- we disable a lot of services, run our apps out of circus, use the same config management system as on Linux, and so forth. We treat the Macs as individual worker units, so they're basically a GPU connected to a network card from the perspective of our stack.

> who may be prone to pass on theories of how things work rather than facts

This is the biggest nightmare about working with OS X, to me.

Any forum discussion you find on Macrumors or the Apple forums is hilariously misguided with pathetically bad "theories" on why something isn't working and how to fix it.

"Zap the PRAM!" can be found in any/every thread, and that's a mild example.

Zapping the PRAM is a pretty frequent joke around the office.

There are some OS X groups that are more focused on automated deployments for IT type stuff, so those can often be a source of more enlightened discourse, even though it still isn't exactly catering to our niche.

Zap the PRAM is the mac version of "Have you tried switching it off and on again?"

(you should get Apple promotional sponsorship while you're at it. this blog post is great for the mac pro, which is a fantastic piece of hardware.)

This should be at the top of the article to hush folks like myself who might be curious from the get go why another approach wasn't taken and why the Mac Pros were selected. Thanks for the insight as to the why.

It was something that I struggled with while writing the article -- you don't want to introduce your solution as "well, this sucks in various ways, but keep reading...". It does boil down to viewing things pragmatically though. The Macs present challenges, but by our math, they're worth it relative to the cost of doing something else and the benefits they provide.

And we actually run a lot of them in production, so I've figured out how to do it and not pull my hair out constantly. That's something I'd like to write on as well, but it would be in a different medium. More technical depth, less pretty pictures.

You do want to introduce your solution with "this has problems A, B and C, but we went with it because of X, Y and Z." It becomes much more interesting, because you're explaining not just your solution, but the problem space. Just the solution is usually less interesting than why the solution, even with its downsides, solves your problem.

By the way, thanks for clearly, completely and patiently responding to people in this thread.

Agreed, I didn't want to sugar coat things either. I think there is a bit of time given over to the downsides, but finding the right balance is tricky, and perhaps I erred by focusing too strongly on the "this is awesome!" side of things.

I want to explore the design decisions around the chassis in a follow-up, and we have one interview in the can already with the industrial designer. Hopefully that article will be a little faster to get out; this one was written about 3 months ago.

The other angle that I'd love to explore in a more in-depth article is how we actually do this stuff in production, and what we've learned about it. This would delve more into the ugly OS X stuff that we painted over to get things nice and pretty in production.

Reply on news.ycombinator.com