If you're OK with waiting multiple hours for CI, I suppose these issues wouldn't affect you very much. Where I work we make relatively small PRs and deploy to production tens of times per day. The recent outages have been a nightmare for us.
We do that, sort of. Obviously we're not PR'ing to get things in to production. Changes go through QA, Staging, UAT, etc first. The final point is that they go to prod though, and if a few hours of changes end up queued to all go at once that's fine.
If not seeing a change in production for a few hours counts as "a nightmare" for you then I certainly don't envy you. That must be incredibly stressful. How do you do good work under that amount of pressure?
Any chance you're willing to share more?
It sounds like there's something interesting to learn here.
At the company I work at, we have about 160 full time engineers with about half our repos on continuous release. We average right now somewhere around 50-55 releases per day, with some hot repos getting up towards 20 releases per day on their own.
The point of small releases is that it makes rollbacks super easy (you always know what broke), roll forwards easy (you can easily test post rollout to find out if things aren’t functioning as expected), and you can mentally move on to your next task sooner because your code gets released when you want it released- not when the next “big develop release” happens.
To pull this off requires a few things. You need good tests that run against PRs. You need fast deploys (under 10 minutes is a good starting point, but get them faster if you can). And, ideally, you have ephemeral environments for each PR so that changes can be tested fully in isolation without requiring a shared staging environment.
This is legit what I do for a living, so AMA.
Can you describe at a high level your development toolset, CI/CD toolchain and workflow, and deployment environment?
Devs develop locally. Can test their code (manual test and most automated tests).
They push and open a PR. They can deploy ephemeral environments with any dependent microservices spun up along side. This is an in house tool we developed that builds and deploys ephemeral docker, CDK, and terraform based apps.
We then run CI automation against their PRs. This can include E2E tests courtesy of these ephemeral environments.
Once their PR is green and signed off, they merge to master. We’ve done some wizardry behind the scenes so that their code is live in production in around 3 minutes.
They verify their changes in prod and we run any automated regression tests against prod.
Next person merges as they see fit.
That’s it! It’s not easy to get to that point, but once you’re there it’s a great process. We’ll have capacity issues once we get up to 30 or 40 releases per day in a repo, but I’ve got some ideas about how to work around that.