It's like comparing leasing a Mercedes-Benz with full service and all inclusive insurance to building your own bicycle. Some people don't need the car, sure, but it's not the same thing.
Although having to spend some time to attain good enough availability/durability is not a big deal when the rush to launch was over long ago. Unlike in your example, you can turn the bicycle into a car.
All it needs to do is to get you the data faster than the compilation itself - most of the time - to be useful.
I also wonder at the original thinking, given the use case. Cache usually has high ratio of download to upload and doesn't need high reliability. S3 doesn't excel at the costs of fetching the data from it, at all.
Do you ACTUALLY think you can communicate with people at Amazon who truly know what's going on? If so, why do we see countless stories about people who have problems with Amazon, Google, et cetera, who can't communicate with them? Do you know of some special, magic way to get the skilled people to talk with you that the rest of us don't?
You act like being able to call Amazon, even if you could get someone on the line who knows something, is somehow better than having even just one good systems administrator. All of your ridiculous assumptions go right out the window when you have a sysadmin build, test and deploy your own hardware that you (via your sysadmin) have complete control over.
Security? You can't secure Amazon. You literally can't. If you wanted to, you'd have to buy the company, then you'd have to evaluate your whole S3 staff. If you hire a systems administrator, you have to vet your systems administrator and anyone else who has admin access, not a collection of tens of thousands of employees across the planet, any of whom could be stupid enough to visit the wrong web site using the wrong browser on the same computer as their keys.
Availability? My side projects have better uptime than Amazon.
Durability? Two geographically disparate locations and LTO, plus many mirrors, make things plenty durable.
When you're not a systems administrator, you can be forgiven for not understanding the big picture, but not for acting like you do.
And for the point in question, you don't need to talk to anyone, the point is there's thousands of datacenter, networking, app-level layers of staff all working to make service stable for you without you even knowing. I don't need to talk to anyone if an S3 server, rack, or even full DC goes down.
If this is threatening in a serious countenance, our growth and ability to conduct our project. I'm not sure if we need all the 9s of durability of S3, given that people already lost all their buckets on S3.
We do not have the money to store 5x or 6x the contents of the cache.
So we much prefer to have control, and we have enough skills to run all that shit easily, the problem is that the skilled people are in rare availability and are usually working on harder problems than running that. So ultimately, this is a balance problem.
Disclaimer: NixOS developer, 23.05 Release Manager.