We've found that distributed building has pretty much eliminated the need to upgrading developer workstations. Super easy to set up, too.
The distributed build systems only really benefit from aggressively caching the modules that are built, right? But the majority of the builds we do are almost fully cached, having changed just one module that needs recompiling then the linker sticks everything back together, which the machines would then need to download from the distributed builder and at 300MB a binary that’s gonna take a while.
I may have this totally wrong though. Would distributed builds actually get us a new binary faster to the local machine?
I suspect we wouldn’t want this anyway (lots of our company work on the go, train WiFi wouldn’t cut it for this!) but interested nonetheless.
Not really, you have more cores to build on. Significant difference for slow to compile languages like C++.
> I may have this totally wrong though. Would distributed builds actually get us a new binary faster to the local machine?
Yes, again, for C++.
If you want a GCP instance that is comparable to an M3 Pro 36GB, you're looking at an n2-standard-8 with a 1TB SSD, which comes out at $400/month.
Assuming you have it running just 8 hours a day (if your developers clock in at exact times) and you can 1/3 that to make it $133/month, or $1600/year.
We expect these MacBooks to have at least a 2 year life, which means you're comparing the cost of the MacBook to 2 years of running the VM for 8 hours a day, which means $2800 vs $3200, so the MacBook still comes in $400 cheaper over it's lifetime.
And the kicker is you still need to buy people laptops so they can connect to the build machine, and you can no longer work if you have bad internet connection. So for us the trade-off doesn't work whichever way you cut it.
1. With a savings plan or on-demand?
2. Keeping one instance on per developer indefinitely, or only when needed?
3. Shared nodes? Node pools?
4. Compared to what other instance types/sizes?
5. Spot pricing?
2. The issue with only-when-needed is the cold-start time starts hurting you in ways we're trying to pay to avoid (we want <30s feedback loops if possible) as would putting several developers on the same machine.
3. Shared as in cloud multi-tenant? Sure, we wouldn't be buying the exclusive rack for this.
4. n2-standard-8 felt comparable.
5. Not considered.
If it's interesting, we run a build machine for when developers push their code into a PR and we build a binary/container as a deployable artifact. We have one machine running a c3-highcpu-22 which is 22 CPUs and 44GB memory.
Even at the lower frequency of pushes to master the build latency spikes a lot on this machine when developers push separate builds simultaneously, so I'd expect we'd need a fair bit more capacity in a distributed build system to make the local builds (probably 5-10x as frequent) behave nicely.
While I had no power over changing the laptops, I was co-administrator of the dev datacenter located 20 meters away and we had our own budget for it. Long story short, that dev datacenter soon had a new, very beefy server dedicated for CI jobs "and extras".
One of said extras was providing Docker containers to the team for running the product during development, which also happened to be perfectly suitable for remote development.