I'm inexperienced in the hardware front, would that machine likely not break down under a large load for 3 years straight? Nothing is set aside for hardware failure etc.
Every 6 months or so a hard drive fails (out of 16 per server), no other components have failed. 10 machines are 6 years old (test system as it's out of warranty), 10 3 years old, 10 new.
There's also 20 or so other servers under different loads, I've not had anything fail other than hard drives.
When I started the job, there were spare power supplies for some 10+ year old servers in storage, so I'm either lucky or reliability is improving.
I used to work for a VFX company where we would try and get as close to 100% utilization out of the farm.
We had machines that were 4 years old, still merrily plodding away. (not too many mind, they ate more power than its worth.) The things that tended to fail were Harddrives and fans.
They covered that I think.
Between two data centres, lets assume I did 6 visits a year, and that we had one ~30/min incident a month at $50/incident (it was less, but I don't remember the exact details, and it doesn't matter for this exercise). Let's assume I lost a whole day every visit (I didn't, though it got close at times), and "charge" $1000/day for my visits. That adds up to $6k/year for my time, and $600/year for remote hands, or ~$110/year per server. For comparison the colocation cost us ~$17k/year, or ~$283/year per server. These costs were pretty stable by number of servers, and so favored using fewer, more powerful servers than we might have otherwise.
So that added the cost of renting space at a manned colo facility instead of having the servers in the office (we did have a rack of servers that didn't need 24/7 attention at our office as well).
The rest of my time was spent on devops work that in my experience tends to be more expensive (on the basis of having contracted to do this kind of work on AWS too, and know the difference in billable hours I'd typically get per instance on AWS vs. per physical server on colocated setups) on cloud setups because complexity tends to be higher.
Didn't this make each failure a bigger hit to your overall capacity? How much redundancy did you have? I used to work in adtech with colocated hardware, and it was old and failed a lot, but they had enough it didn't matter ("we're down 5/120 in Germany but we can swap them out while we're there next month").
It was also a fully virtualised setup that could also tie in rented dedicated servers or cloud instances via VPNs as needed. So where it made sense or if we had an urgent need, we had the ability to spin things up as needed.
E.g. we had racks in London, but rented servers at Hetzner in Germany (Hetzner got close to the cost of the colocated servers, though mostly because rack space in London is ridiculously expensive; it might actually have saved us money to put servers in their colo facilities in Germany, even with the cost of travel to/from them occasionally)