> During the Q&A an attendee asked about Kubernetes CI. The audience member said that CI for the project cost "between $100,000 and $200,000 a month"
wtf?! That's a crazy amount of money. At 10,000 PRs a month, that's $10 to $20 per PR(!!)
Here's our GCP spend for the past month: https://imgur.com/a/VVJTSKx. Note that does not include a separate AWS cluster that we are migrating jobs too.
A large chunk of this comes from the nature of distributed tests. We need to reproduce the environment, spin up compute, etc. We do have a large problem with flaky tests on the project as well. Whether that's timeouts, memory/cpu consumption creep over time, loads of other things. We talk about how one day we'd like to get to the granulairty of being able to go to a SIG and say, "this flaky test of yours is costing the project $x in retries. Please dedicate some resources to fix it".
How we distribute the artifacts is a whole different conversation. The container world is unique in that voluntary mirrors are not as possible as with linux packages and other binaries.
If this space interests you please join us at either [SIG K8s Infra](https://github.com/kubernetes/community/tree/master/sig-k8s-...) or [SIG Testing](https://github.com/kubernetes/community/blob/master/sig-test...)!
I have seen in lot of organisations, every push triggers a build. That is wasted compute.
I don't think I've ever seen the "developer time is worth more!" true-ism being uncritically repeated than here. They can easily hire over 12 full-time devs for this money.
You're still ignoring it. They've got an issue in their CI, it's not that they're triggering it to often. It's because each build costs way too much. Though we technically don't know that either, as the attendee didn't specify how many builds they did, technically. (At least I didn't see it in TFA)
It was also explicitly stated that the goal was to have the Dev run the build locally and sign the commit with the build status so they could skip CI on the server and save cost there. Considering that context, I believe that the Dev time = money argument is justified.