Hindsight is 20/20
Deploying in such a way that all your servers are not running the same codebase is obviously bad.
Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad.
Not having anyone monitor things closely enough, including the hundreds of warning emails they got before the market opened, is obviously bad.
There is no hindsight necessary here. You could look at what they were doing and predict a catastrophe.
I think the main problem here is nobody at this company pushed back on this stupid development plan of reusing a flag for a different purpose. There's no excuse for that (or maybe there is, they had run out of fields in some fixed-width message format or something dumb like that). Also apparently the use of the flag was not tied hermetically to the binary in production; when they rolled back the binary the flag was still there but it meant something different to the old software.
The correct way to roll this type of change out is for the new input (the "flag" in this case) to be totally inert for the old version of the software, and for the new version to have a config file or command line argument that disables it. So _first_ you start sending this new feature in the input, which is meaningless and ignored by the existing software, and then you roll out the new software to maybe 1% of your fleet, and see if it works. Then roll it out to maybe 10% and leave it that way for a week. Insist that your developers have created a a way to cross-check the correctness of the feature in the 10% test fleet (structured logging etc). If it looks good roll it to 100%. You now have three ways to disable it: turn it off in the input stream, turn it off in the new software with the config or argument, or roll back the software.
Doesn't look like these guys really knew what they were doing.
Trading floor is only open a few hours every day, the functionality being rolled out required the markets to be open. Furthermore, since the changes were all for new functionality they rolled it out in stages days ahead of time (good move).
Roll out the code in advance, and have the production machines switch to it at a defined, synchronized time?
I mean, imagine you only have one production machine. If you're willing to admit that you can have it switch from version X to version Y with no downtime, then synchronization is the only barrier to doing the same on n machines. Why would you need scheduled downtime?
But there's at least 30 minutes of downtime per week per market (usually per day), and the vast majority of downtimes coincide during the weekend - so this is all moot discussion and needlessly complex solution. If you can afford the downtime, switch midnight GMT between Saturday and Sunday, when all markets are closed.
If you plan is for the software to be on all servers, it needs to be on all servers.
What sometimes happens when you want to decommission features is you just turn the flags off rather than remove the code. There's an obvious allure to this as you have already tested the on/off functionality of the switch when you did the original roll-out so you can avoid having to test whether you have removed the code correctly. It sounds like in this case they removed the code and repurposed the switch that disabled said code (it may really be a shared memory system and they are running out of flags), but they fucked it up. The old code was still there on some servers and the switch was turned back in intent to enable the new feature it was re-purposed for, re-enabling this old code.