Can't those first 2 steps be combined together? Why do they need to be shipped separately?
1. Have the code recognize the new value. Get that shipped everywhere.
2. Have the code do something reasonable when the new value appears. Get that shipped everywhere.
3. Start emitting the new value.
My proposal is
1. Have the code recognize the new value. Have the code do something reasonable when the new value appears. Get that shipped everywhere.
2. Start emitting the new value.
If the user has to manually update, and might refuse to do so, the problem exists under both the 3 step and the 2 step process. So nothing is gained by using the 3 step process.
If the client and backend are updated independently, that should still be fine with the 2 step process. The 2 step process says "Get that shipped everywhere", meaning shipped to both the client and the backend.
1. Support the new value in your schema
2. Support the new value in the client
3. Emit the new value from the server
Combining 1 and 2 is probably possible, but not a great idea. Imagine if you end up rolling back 2 and 3, but there are still potentially new values in flight. If 1 is still there, you're good. If 1 and 2 are combined, you rollback all 3, but the new value is still in flight and your client crashes.
You have to
* Ship the new version
* Wait until you are sure you don't need a rollback to an old version
* flip the switch
If you don't wait, you might need to roll back to an old version that doesn't support the new state, which then blows up.
The third can be a flag flip instead of a code push, but it still needs to be a discrete event to start generating the new value that triggers new behavior.
But also there could have been a bug in 1/3 that wasn't exposed until step 3/3. Meaning even rolling back to 1/3 won't solve the problem.
The 2 step rollout is safe if we assume there are no bugs in the code, but problematic if there are bugs. But the 3 step rollout is also problematic if there are bugs. I guess the 3 step rollout has the advantage that it partitions off a section that might be buggier than other sections so that it can be individually disabled. But that'll only help sometimes, and I'm not sure if the additional complexity is worth it.
Flag flip vs code push doesn't seem to make a difference to me. All 3 rollouts could be flag flips enabling code that was written much earlier but hidden behind flags.
Another problem is doing a double rollback like that seems a bit risky to me, because other features that were being deployed simultaneously might not have been designed to handle a double rollback, only a single rollback, so they could break. If we want to allow double rollbacks, we must require that all development not just handle single rollbacks smoothly but also double rollbacks.