How to lose $172k per second for 45 minutes (2013)

This. Don't except things to change if you happily stay in a workplace where things are done in a half-assed way. There's endless supply of jobs for software people nowadays.

Changing the status quo is kinda fun in these environments. Low hanging fruit, easy work. As long as your head isn't on the chopping block.

And everyone benefits. You, the employer, your colleagues, the fresh grads with starry eyes

I wish I hadn't experienced this exact scenario as well. I actually worked at a publishing company where I discovered the previous team building their major education platform didn't know there was such thing as source control. Their method for sharing multiple team members changes was to email the full project to one person every Friday who would manually open a merging tool and merge all changes themselves. They would then send out the updated code base to all people again. Because of this method, they were afraid to delete any stale code, and just prefixed old versions of functions with XX. As you can imagine, inheriting that code base was a nightmare to deal with.

>I've seen companies not using source control

Well I've seen companies that have their own idea of source control. Which is lots of copies on the network drive, and an Excel registry with what is in which file.

It is source control. Just bad source control.

> I've seen companies not using source control

> ...all changes are being made directly on the production servers via SFTP

I know this used to be common, but recently? Curious how often this is still the case.

This comment is resonating very well with that I've been experiencing at my current job, and with our new head company. It's always nice if you can take a new application, throw it into a clean container build chain with an orchestration - or some AWS stack, and there's a cheap, low-maintenance, HA production system.

However, there's also the skill set of taking such a ... let's call it well-aged development team and approach and modernizing and/or professionalizing it. And yes, sometimes this means to build some entirely lovecraftian deployment mechanism on mutable VMs because of how the application behaves. But hey, automated processes are better than manual processes, which beat undocumented processes. Baby steps.

> I've seen companies not using source control with no staging environments and no local development environments where all changes are being made directly on the production server via SFTP on a basic VPS.

Omg. And here I am feeling ashamed to tell others about my small small personal website with separate dev, qa, production environments on same server (via VirtualHost), code checked into github, deployed via Jenkins self hosted on another VPS, which was initially spun up with Ansible and shell scripts. All done by me for self training purpose. All because I thought businesses would have something more sophisticated with bells and whistles.

And then I hear there are businesses that make changes directly on live production servers...

But I'm not surprised by such stories as I have seen some bad workflow in real businesses that deal with tens of millions of dollars a year.

Years ago, I worked in the NOC of a company that's top in the small niche. They have dozens of employees, and been around for years.

Part of the job responsibility was rolling out patches to production servers. The kicker was the production servers were all Windows servers, running various versions, covering practically all Windows versions ever released by Microsoft. You can see where this is headed.

Rolling out a patch was all done manually, 1 Windows server at a time. Everything was manually done.

The instruction for deploying a change was easily multiple lines long, each with different style of writing/instruction. Often in plain text file format. We would print them out so we could check them off as we went down the list.

The CTO is still there, but everyone IT person under him has left or been let go. Working in the IT there is a struggle because lack of automation and old old stuff, but the CTO just blames bad employees and keeps churning them in/out. When the real issue is the decade or 2 worth of old legacy stuff that need to be cleaned up and/or thrown out, which can only be done by the direction of the CTO. But he knows he won't get that kind of budget from higher-ups so he will just keep hiring/firing employees and/or bring in some H1B workers who's basically trapped once they join. And of the few H1B workers I met there, they were truly completely non-technical. One did not want to learn how to use keyboard shortcuts to do common tasks... Good guy though.

This kitchensoap article may be more correct, but man, it's a arduous read. A bit of brevity and humour can work wonders, conversely complete correctness can be detrimental if people lose attention.

> He also makes an interesting point related to what you're saying: the SEC says the risk management controls were inappropriate, but clearly Knight thought they were appropriate or they would have fixed it.

> This would mean that Knight Capital did have appropriate controls the day before the accident.

I think these claims are seriously confused.

SEC fines don't require mens rea, so Knight is simply being punished for having inappropriate controls, their view on the matter be damned. Kitchensoap rightly observes that "this event was very harmful" does not imply "this event was caused by extreme negligence". But the SEC filing focuses on alerting and controls; position limits don't prevent a specific misstep, but they limit the maximum size of any error that does occur. (Knight had position limits on accounts, but didn't use them as fundamental boundaries restricting actual trade volume.) The thesis is that Knight should have prepared to mitigate "unknown unknowns", in which case the size of the error is relevant because the size was exactly what should have been controlled for.

On appropriate controls, SEC fines are certainly outcome-biased, but the claim is obviously that these controls were always inappropriate, and the disaster simply revealed them. Post-disaster punishment creates an ugly system where people who don't take excess risk can be outcompeted before their competitors crumble, but the rule isn't actually conditional on failures.

Kitchensoap asks whether Knight would be judged so harshly if they'd only lost $1,000. Socially, perhaps not, but legally they actually would have! The SEC isn't just punishing Knight for losing money but for disrupting the market with improper trades; it specifically notes that for some "...of those stocks, the price moved by greater than ten percent, and Knight’s executions constituted more than 50 percent of the trading volume. These share price movements affected other market participants...". A smaller loss wouldn't have defended against that charge, while a smaller trade wouldn't have violated SEC rules.

I think the author is basically aware of this, since his fundamental point is that the SEC is describing the legal wrongs rather than the technical mistakes. That's a good point and I'm glad you linked this. But I think his focus on the specific deployment error neglects the fact missing position controls were the larger legal and technical failure.

> SEC says the risk management controls were inappropriate, but clearly Knight thought they were appropriate or they would have fixed it.

By this logic, we can claim that if a hospital is storing all its employee passwords in plaintext, that's "appropriate" because if it was inappropriate they wouldn't do so.

Or that if a company is neglectful about offsite backups, that's an "appropriate" data retention strategy because if it wasn't, then the company would be taking backups.

In this case, if Knight thought its controls were "appropriate", then that's the problem that needs fixed.

You live by the AI stock trader, you die by the AI stock trader.

I've seen a lot of companies and teams with hair-raising practices, but most of them still manage some basic business-logic safeguards like "don't buy for more than $X" or "if we don't sell anything for 10 minutes, shut it all down and sound the alarms". Generally they can't do worse than "making no money" or maybe "paying full expenses without making any profits".

The link here excerpts the full filing, but the real meat wasn't included:

> 16... Because the cumulative quantity function had been moved, this server continuously sent child orders, in rapid sequence, for each incoming parent order without regard to the number of share executions Knight had already received from trading centers... 17. The consequences of the failures were substantial. For the 212 incoming parent orders that were processed by the defective Power Peg code, SMARS sent millions of child orders, resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes. Knight inadvertently assumed an approximately $3.5 billion net long position in 80 stocks and an approximately $3.15 billion net short position in 74 stocks. Ultimately, Knight realized a $460 million loss on these positions.

Knight did $1.4 billion in revenue per year, but there was no safeguard against adopting $7 billion worth of positions in 45 minutes. The end result was that they lost 4 years of profits in under an hour. That's crazy, and not at all standard.

> bottom ~80% of the industry. The top ~20%

How are you defining "top" and "bottom" here? I can agree with the 80/20 split, but I don't think it necessarily tracks with, say, name recognition or market cap. Amazon, for example, utterly failed on their last Prime day, and from the discussions of those Amazonians I know, it was pretty much inevitable.

The REASON these fast-and-loose habits are habits are because, when they work, they make money. You can't look at a lottery and say the winners "did it right" and the losers didn't - the winners just haven't failed yet. Those companies that AREN'T trying to grow at ridiculous speeds but ARE trying to maintain quality (and are fortunate enough to be in a market niche that supports that) are the ones most likely to have bulletproof reliability. Those won't be the "top" companies by most peoples definitions.

> The top ~20% of tech companies actually care about quality

Most companies are not tech companies, fyi. So you're talking about the top 20% of a very small subset of business.

The team of MBA's can probably help avoid 172k$/second types of mistakes no?

There’s nothing explicitly wrong with this. MBA’s aren’t intrinsically stupid, and excel is a reasonable tool for calculations. It’s just a pure functional language using geometric space instead of a namespace.

Most business deals are done in part based on results from excel spreadsheets. What other tool would you expect to be used?

This must be ignorance speaking, but what is the issue with that?

+1 on this... there's a huge difference in code that may be used by a dozen users to pick where to have lunch and HFT systems that handle millions of dollars worth of capital investment a day. There also is a correlated high burnout.

About a decade ago, I was a contractor in a security services team at a large bank. There were systems in place to minimize the impact of our software, which handled requests for access, and the actual access granted. Our software was developed like most, or worse in the beginning. In the end, by separating workflows and agreeing on interfaces, a lot of issues were prevented altogether.

Other systems should be designed with similar safeguards and separate certain control flows behind well thought out APIs.

Others still should be created much closer to classic waterfall, with adjustments meaning a stop-work, re-evaluate, update design, and proceed process in place.

It should really depend on the situation.

There's nothing particularly wrong with e-mailing (or using other push notification mechanisms) when a warning or error occurs. When those logging levels are properly used it means some sort of administrator intervention is needed. But send the notification only once. Don't keep spamming out the same notification over and over.

Don’t mean to sound snarky, but there are tools that do this and have been for years. If you’ve been grepping through logs for the last 3 years, you’re doing it wrong for the cloud era.

Often times the answer is writing better alert triggers that take historical activity into account to cut down on false positives. Other times it’s simply to reduce the number of alerts. In every case you need an alerting strategy that takes balances stakeholder needs, and you need to realign on that strategy quarterly. It’s ultimately an operational problem, not a technical one.

Alas, back in the real world, logging is always the last thing teams have time to think about...

I once worked in an envrionment like this that saw about 1k emails a day when I started... What I started doing was trying to resolve one bug a day... I'd look at an error, do a search in my email filtered folder for that error, and manually count them, whichever had the most emails, I took care of it.

About 2/3 just came down to filtering certain classes of errors. 4xx errors, I stopped email notification altogether, since they were already being trapped/handled by the system. Others were a little more specific. Ironically, .Net tends to handle some things that should be 4xx errors as 500, so reclassifying those took out a lot as well.

In the end, within about a month, the emails were down to a manageable 20 or so a day and got more visibility as a result.

Honestly, it sounds like there just needs to be some higher discipline here around what the different log levels mean. ERROR should only be used if your system failed to service a request. Any other errors are WARN. Then you tell the system to notify you on any ERROR log, because that means that requests are failing to be processed.

In your example, the `NumberFormatException` is a bad ERROR entry, because it's covered by a WARN entry right below it. Meaning that the request was not failed -- it entered a default value instead of the bad parsing. So that exception should also be at WARN level.

(Arguably, overwriting input values with defaults because of a parsing error is probably a bad idea and should be an ERROR due to rejecting the request. But I'm rolling with what we got here.)

Was the deployment process of your parent post actually error-prone? From what I gathered, the developers were unaware of the lost handheld scanners. I imagine if they did they could've proactively put them out-of-service until found.

IMO that's too late of a condition check. You should keep a margin of free space for OS/Background applications just so you have enough life left in the application to detect and log it.

I think the parent did well openly and honestly raising a personal example where missing a "basic" check caused near career changing problems - I applaud them for sharing a difficult situation.

I was concerned that it's possible to read your comment as if it was critical of the parent - was that your intention?

Big bank is not the same as a small/mid sized trading firm. Banks have regulations they need to meet, and typically do things by the book.

I'm not an expert here. Part of what I said is based on the 6 different people I've met who have worked in the industry. I'm just saying if you have $400+ million to lose and you rely on the IT infrastructure allows you to make that money then you can spend a few million on top notch people and processes to prevent this kind of thing. I worked at a relatively large media company, and every deployment has a go/no-go meeting where knowledgable professionals asked probing questions, you defended your decisions. I've love to know what they did in Knight Capital. The idea of re-using an existing variable for code that was out of use strikes me as a terrible idea.

Maybe I was using the wrong terminology, but when I said "CI" I meant that to include automated, atomic deployments.

In the last 16 years I've worked in software the last 10 or so of that has not included manually copying files to production

The bug report describes how they reuse a flag for new code that triggered old code.

"CI" would generally help keeping things tidy and transparent. It would be easier to find out what branches or flags or parameters already exist, which ones are not relevant any more; someone could rename them with a prefix "OLD_" or otherwise clean up, so that you typically pick from a list of options instead of setting a flag by accidentially copying it from one script to the other.

It is also easier to have error visualization plugins, or to look around previous deployments or test runs and get a pretty good idea if any new errors have shown up, even if you haven't analyzed the old ones.

I disagree that it isn't a software bug.

"The new RLP code also repurposed a flag" - this is the moment when terrible software development idea was executed that resulted in all of the mess.

Of course I don't know the full context and maybe, just maybe there was a really solid reason to reuse a flag on anything.

What I observe more often is something like this though:

  1. We need a flag to change behaviour of X for use case A, let's introduce enable_a flag.
  2. We want similar behaviour change of X also for use case B, let's use the enable_a flag despite the fact the name is not a good fit now.
  3. Turns out use case B needs to be a bit different so let's introduce enable_b flag but not change the previous code so basically we need them both true to handle use case B.
  4. Turns out for use case A we need to do something more but things should stay the same for B.
  5. At this point no one really knows what enable_a and enable_b really do. Hopefully at least someone noticed that enable_a affects use case B.

If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag even if they do exactly the same thing as more than likely they do exactly the same thing only for now.

What would probably be even better is separate, properly named config flags for each little behaviour change and just use all 5 of those to handle different use cases.

edit: formatting

One of the tests in my CI pipeline is verifying that my new code works with previously released code. It gets painful when these tests fail, but it is a valid configuration so I have to work around bugs in the old code from time to time.

That's assuming that business-focused people have any interest in determining the amount of technical debt at all. Generally they're content to let other teams pay the debt, and many of them are quite happy to complain when it doesn't get paid fast enough.

While I mostly agree, many companies have a tech department of half a dozen people, and implementing and enforcing every good practice devops isn't always realistic.

That said, I'd expect at least a backup of production, then again he said he lost 1 hr of data so it was likely between backups.

Even cowboys don't herd cattle by themselves...

No I totally agree, looking back on my actions, rm -rf'ing the production database was not a productive use of my time.

SQL fixes in production don't have to be that risky. Just run a SELECT first with the same predicates and be sure that's what you want. In some cases this can be better than trying to develop a fix using a test system which may not have the same data.

Same with "rm -r" --- run an "ls" first.

Sure it's not ideal but life isn't ideal. When you have a production problem, you need to fix production.

I think you're making a number of assumptions about the size and maturity of the parent's organization; it could be just one person.

With the right tools and processes, it's possible to build very successful companies with tiny engineering teams. Developers run queries against prod because there's nobody else to do so. The risk of mistakes is mitigated by the increased situational awareness and the developer quality and communication in 3-teams vs 30-teams.

Neither approach is inherently 'wrong', although running a 30-person team the same way as a 3-person team (or vice versa) can have nasty consequences.

I worked in a place for years where we had access to production databases, to do updates, deletes and so on. We had a certain stylized way of doing things to prevent irreversible errors. If you were going to delete data for instance, you always created a backup table first with 100% of the columns for the rows of interest, then selected from it to verify it looked reasonable, and then deleted using the backup table to correlate rows to the production table using the primary key. And then another analyst reviewed your work (originally two levels of review, later reduced to one). The backup tables and comments in a ticket including a copy of the SQL run were kept indefinitely, so even if the review missed something it generally was correctable.

The DBAs in the company did think SQL should be reviewed in advance, but that's not how our department did it. I think it's arguable that, given that reviewers before or after doing something dangerous may miss things, it's better to establish safe practices and if you do that, then you don't really need a review in advance.

Pretty sure Google cloud does support it

Pretty sure aws still doesn't

Pompei?

I've thought before that transforming our soft grey matter into gold is basically what the Rosicrucian alchemists were on about.

People are responsible for making the processe, no? It’s still a people problem.

Yeah there's a lot of macho, procrastinatory bullshit that gets in the way of having a good process. There's always someone who thinks that each step is 'simple' and refuses to believe in death by a thousand cuts. They blame people who make mistakes and feel better about themselves.