I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.
I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do that, it would appear that the code has to be written much more defensively than what they're doing it right now.
Blue/green and temporarily ossify capacity? Regional?
- Push a version that enables new logic for 1% of traffic
- Continue rollout until 100%
Unit test, Integration Test, Staging Test, Staging Rollout, Production Test, Canary, Progressive Rollout
Can all be automated can smash through all that quickly with no human intervention.
I really don’t buy this requirement to always deploy state changes 100% globally immediately. Why can’t they just roll out to 1%, scaling to 100% over 5 minutes (configurable), with automated health checks and pauses? That will go along way towards reducing the impact of these regressions.
Then if they really think something is so critical that it goes everywhere immediately, then sure set the rollout to start at 100%.
Point is, design the rollout system to give you that flexibility. Routine/non-critical state changes should go through slower ramping rollouts.
They need that same mindset for themselves in config/updates/infra changes but probably easier said than done.
https://blog.cloudflare.com/deep-dive-into-cloudflares-sept-...