Yes, the deployment practices were bad, but they still would have had an issue even with proper practices.
The real issue was re-using an old flag. That should have never been thought of or approved.
The real issue was re-using an old flag. That should have never been thought of or approved.
There are always two error rates.
- how many orders per minute do you expect to create?
- how many orders per minute do you expect to cancel/amend?
- what’s your max per-ticker position?
- what’s your max strategy-level GMV/NMV?
Etc.
Any one of those questions can be used to set up killswitches.
[edited for formatting]
There are always two error rates.
Defining behavior is great for retrospective analysis but would you really feel comfortable putting hard cuts into production based on the answers to those questions? I’m genuinely asking, because IME I wouldn’t be.
Seriously, suppose you have a truly awesome system making $100B per year of revenue. If you unnecessarily shut down 0.1% of the time, that’s only $100M per year lost, and an 0.1% unnecessary shutdown rate seems pretty high.
IME that last 9 is where all the action happens
> unless you have customers who care quite a lot
All customers care about their trades. I’ve worked with these systems. You can’t treat smaller traders as less-than.
> only $100M
How far removed from the problem do you have to be to think one hundred million dollars is not going to effect anyone?
If $10b is at risk, $100m is not a lot for an insurance policy.
Estimate what a real human can do in a day, and use that as the limits. Verify that the system behaves ok for some time, then scale up the desired trading volume and limits, observe, scale, repeat.
But you don't do it by making a (bad) guess up front and then just leaving it at that.
Hopefully the kill switch system is reasonably easy to code review and test :-)
It's a set of failures in a system that all had to happen for the failure to occur.
it reads less to me like a case for devops as it does a case for better practices at every stage of development. how arrogant or willfully ignorant do you have to be to operate like this considering what’s at stake?