Hacker News Reader

Yep. For my own company I used a bare metal machine in Hetzner running Linux and a Windows VM along with a bunch of old MacBook Pros wired up in the home office for CI.

It works, and it's cheap. A full CI run still takes half an hour on the Linux machine (the product [1] is a kind of build system for shipping desktop apps cross platform, so there's lots of file IO and cryptography involved). The Macs are by far the fastest. The M1 Mac is embarrassingly fast. It can complete the same run in five minutes despite the Hetzner box having way more hardware. In fairness, it's running both a Linux and Windows build simultaneously.

I'm convinced the quickest way to improve CI times in most shops is to just build an in-office cluster of M4 Macs in an air conditioned room. They don't have to be HA. The hardware is more expensive but you don't rent per month, and CI is often bottlenecked on serial execution speed so the higher single threaded performance of Apple Silicon is worth it. Also, pay for a decent CI system like TeamCity. It helps reduce egregious waste from problems like not caching things or not re-using checkout directories. In several years of doing this I haven't had build caching related failures.

[1] https://hydraulic.dev/

> 2-5x pipeline performance at 1/2 cost just by using self-hosted runners on bare metal rented machines like Hetzner

This is absolutely the case. Its a combination of having dedicated CPU cores, dedicated memory bandwidth, and (perhaps most of all) dedicated local NVMe drives. We see a 2x speed up running _within VMs_ on bare metal.

> And knowing how to deal with bare metal/utilize this kind of compute sounds generally useful skill - but I rarely encounter people enthusiastic about making this kind of move

We started our current company for this reason [0]. A lot of people know this makes sense on some level, but not many people want to do it. So we say we'll do it for you, give you the engineering time needed to support it, and you'll still save money.

> I just don't see why going bare metal is always such a taboo topic even for simple stuff like builds.

It is decreasingly so from what I see. Enough people have been variously burned by public cloud providers to know they are not a panacea. But they just need a little assistance in making the jump.

[0] - https://lithus.eu

At the last place I worked at, which was just a small startup with 5 developers, I calculated that a server workstation in the office would be both cheaper and more performant than renting a similar machine in the cloud.

Bare metal makes such a big difference for test and CI scenarios. It even has an integrated a GPU to speed up webdev tests. Good luck finding an affordable machine in the cloud that has a proper GPU for this kind of a use-case

Is it a startup or small business ? In my book a startup expects to scale and hosting bare metal HW in an office with 5 people means you have to figure everything out again when you get 20/50/100 people - IMO not worth the effort and hosting hardware has zero transferable skills to your product.

Running on managed bare metal servers is theoretically the same as running any other infra provider except you are on the hook for a bit more maintenance, you scale to 20 people you just rent a few more machines. I really do not see many downsides for the build server/test runner scenario.

Every business wants to scale, not many do. There's no real difference between a startup and a small business, except perhaps how strong or realistic the dreams are.

Consider that even VC-backed Google started out with lots of cost saving measures in place. Once they brought their second datacenter online, the way they uploaded the index to it was Lucas putting a big stack of hard drives in the boot of his car and driving across the USA to plug them in. This continued for a while until they were able to strike a special deal on bandwidth.

That was just one of the unconventional things they did to save money. Using Linux was another. Although we now think of what Google did as obvious, at the time it was considered bizarre and radical that a search engine company - a software company - would rack bare motherboards onto corkboard bases, just to save money. All their competitors were running on commercial UNIX big iron.

https://www.datacenterknowledge.com/hyperscalers/looking-bac...