How Power Peg Brought Down a Knight | Hacker News Reader

How Power Peg Brought Down a Knight

futuresmag.com

18 pointsminimax5 comments

This is a little long, but it's a must read for anyone interested in real-life consequences of computer bugs.

Basically, new software was pushed to seven of eight servers at Knight, one of the biggest players on Wall Street. A flag activating the new software was repurposed from very old inactive software that was still on the servers.

The new software ran fine on seven servers, but the flag activated the old software on the eighth server, which promptly started doing something completely insane as far as financial markets are concerned. A few minutes later people removed the new software. But didn't change the flag. So then:

   (the old software) jumped in to fill that void
   and was now running on all eight servers
   simultaneously, buying on the offer,
   selling on the bid.

Knight lost over $400 million in under an hour, and was forced to eventually sell out to one of their biggest competitors.

45 minutes to be precise: http://www.sec.gov/litigation/admin/2013/34-70694.pdf

A little surprising to me that when you're trading hundreds of millions of dollars an hour that the software wasn't built with a NASA style of redundancy built in that would have caught the mistake.

NASA style redundancy comes with a significant cost, that is delivery times. The opportunity costs of time to markets in electronic trading are huge. Some opportunities disappear in weeks (though these systems were not targeting those sorts of opportunities).

Also, redundancies/qa are hard to quantify from a cost benefit perspective and trading firms are very short term P&L driven (pay is frequently heavily bonus based).

> The opportunity costs of time to markets in electronic trading are huge.

That's probably 90% of it right there. But perhaps the remaining 10% is that the "big swinging dicks" that run the brokerages have nothing but contempt for anyone who isn't a broker or a trader. IT is a cost center, nothing more.

Even after the debacle, one of the executives said:

   “The IT guys were arrogant.”

Well, maybe they were. And, maybe, because of the arrogant attitude the executives had, they couldn't attract any non-arrogant IT employees?

Reply on news.ycombinator.com