> With numbers like that Twitter could, profitably, hire dozens of engineers that do literally nothing else. Just tweak thread pool sizes all day, every day, for service after service. Even though it's a boring, manual, "noncomplex" thing, this type of work is clearly valuable and should have happened years ago.
The issue is that once you hire a dozen engineers to do this (say for 5M a year in total), and they do it for a year, they save mid 8 figures (keep in mind this was the largest service, so the savings across other services will be smaller).
Then can they keep saving mid 8 figures every year?
I'll paraphrase something I previously wrote privately, but imagine you have some team that's able to save 10% of your fleetwide resources this year. They densify and optimize and improve defaults. So you can now grow your services by 10% without any additional cost increase, and you do! The next year, they've already saved the easiest 10%. Can they save 10% again? Can they keep it up every year? How long until they're saving 3% a year, or 1% a year? And that's if you keep the team the same size, where its clearly loosing money! If you could afford a dozen people to save 10%, you can only really afford 1-2 to save 1%, but then you're likely to get an even smaller return.
Unless you expect to be able to maintain the same relative value of optimizations every year, 3 or 5 years out, its not worth it to hire an FTE to work on them.
I should note that I've experienced this myself: I was working in an area where resource optimization could lead to "significant" savings (not 8 figures, but 6 or maybe 7). My first 6 months working in this area, I found all sorts of low hanging fruit and saved a fair amount. The second six months, 5-10x less ROI. I gave up even trying in the third six moths, if I come across a thing, I'll fix it, but its no longer worthwhile to look.