Hacker News Reader

Comment by tinco | Hacker News Reader

Can confirm cloud GPU is way overpriced if you're doing 24/7 rendering. We run a bare metal cluster (not VFX but photogrammetry) and I pitched our board on the possibilities. I really did not want to run a bare metal cluster, but it just does not make sense for a low margin startup to use cloud processing.

Running 24/7 for three months, it's cheaper to buy consumer grade hardware with similar (probably better) performance. "Industrial" grade hardware (Xeon/Epyc + Quadro) it's under 12 months. We chose consumer grade bare metal.

On thing that was half surprising, half calculated in our decision was despite the operational overhead how much less stressful running your own hardware is. When we ran experimentally on the cloud, a misrender could cost us 900 euro, and sometimes we'd have to render 3 times or more for a single client. Bringing us from healthily profitable to losing money. The stress of having to get it right the first time sucked.

View on HN

I've had renders cost $50,000! Or CTO was less than amused

I hope you didn't have to bin them, as Vogue did with one of their photoshoots costing a bank:)

Sounds interesting: reference?

In the documentary: The September Issue

> Running 24/7 for three months, it's cheaper to buy consumer grade hardware

If you have a steady load cloud makes little sense. It only makes sense if you have a tight deadline (as is not that uncommon with video and VFX) and can't fit it within your deployed capacity.

How do you manage the bare metal cluster? (E.g. apt/yum updates but also networking and such)

I'm a bit out of date but if we are talking about rendering (not data retrieval workloads) I believe the best way is fundamentally the same as it was 25 years ago: network boot, mostly network storage, and applying local config overlays based on MAC address or equivalents. Exactly what push or pull techniques are in vogue I am not sure but definitely no running package managers on each node. You want as little as possible locally -- just a scratchpad disk that can be rebuilt automatically in minutes.

When it was 3 nodes, and then 6 nodes, the answer was very unprofessionally. I didn't get the budget for a system administrator, and I spent all my budget on developers that could build our application and automate our preprocessing, overlooked system administration skills. So besides the DoE, managing 3 small teams and being the lead developer, I also am the system administrator.

So no fancy answer, our 3D experts got TeamViewer access to the nodes running Windows Pro. Sometimes our renders fail on patch Tuesday because I forgot to reapply the no-reboot hack.

We're professionalizing now at 12 nodes, we got to the point where the 3D experts don't need to TeamViewer in, so we're swapping them to headless Linux. No idea on the update management yet, but they're clean nodes running Ubuntu server.

Network solutions highly depends on the physical infrastructure, but for setup maintenance, you can often see SaltStack.

Reply on news.ycombinator.com