It’s why I’ve started making CI simply a script that I can run locally or on GitHub Actions etc.
Then the CI just becomes a bit of yaml that runs my script.
Then the CI just becomes a bit of yaml that runs my script.
In my experience these are the bits that fail all the time, and are the most important parts of CI once you go beyond it taking 20/30 seconds to build.
A clean build in an ephemeral VM of my project would take about 6 hours on a 16 core machine with 64GB RAM.
It sounds like you've got hundreds of millions of lines of code! (Maybe a billion!?) How do you manage that?
Our cached builds on CI are 20 minutes from submit to running on steam which is ok. We also build with MSVC so none of the normal ccache stuff works for us, which is super frustrating
I am fortunate in that the only things I want to reuse is package manager caches.
The complicated part comes when you have job A that builds and Job B that deploys - they run on two different machine specs so you’re not paying for a 16 core machine to wait for helm apply to wait for 5 minutes - they need somewhere secure to shuffle that artifact around. Their access to that service is likely different to your local access to that service, so you run your build locally and it’s fine but then the build machine doesn’t have write access to the new path you’ve just tested and it fails.
90% of the time these are where I see CI failures
At my place, we have ~400 wall hours of testing, and my run begins by figuring out what tests should be running and what can be skipped. This depends on many factors, and the calculation of the plan already involves talking to many external systems. Once we have figured out a plan for the tests, we can understand the plan for the build. Only then we can build, and test afterwards. I haven't been able to express all of that in "a bit of yaml" so far.