High-Speed Trading Firm Deleted Some Code by Accident
bloombergview.com
bloombergview.com
I've seen this at countless finance companies, and from interviewing people across the entire spectrum of finance. It is a rare corner of the industry where you'll find quality engineering. Usually everyone in a finance company thinks they are amazing engineers, and are just plain wrong.
My advice: work smarter, not harder.
In finance technology is seen as a cost center, not a profit center. They want the minimum needed to ship. Early on this results in a lot of tech debt. Later on, fixing the system is too hard and thus they need to hire super smart people to figure out the mess they are in for any code change.
Enforcement actions are INCREDIBLY rare. The penalties are shockingly low. Thus the cost of failure to comply is low, and the businesses are not prioritizing compliance.
If enforcement actions are rare, it's probably because these firms are playing clean for the most part, contrary to popular belief. If you trade 5-10% of the market, regulators are going to audit you every year. They are looking for even the smallest issues to give a fine, even minor things like not having the right Head Trader's name on your supervisory forms after someone quits.
THIS IS A TOWER. PHOTOGRAPHER: DAMIEN MEYER/AFP/GETTY IMAGES
The article mentions "Tower Research".
Is the image/caption a joke, auto-generated by content, or just plain laziness?
>When they do start doing something stupid, then they will lose money and be punished
hehe no, trades will be reversed, people that "maliciously took advantage of poor algo error" will be punished instead
Not only does nanosecond resolution provide zero benefit, models indicate that it actually harms liquidity. To be specific, serial order processing in continuous-time is more efficient in "time-space", but less efficient in "volume-space". The better mechanism is batch order processing in discrete-time (i.e. process all arriving orders simultaneously in batch every 100 milliseconds, rather than one-by-one every nanosecond): http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
Because the markets are fast, there is less risk that their hedging product will "run away" from them before they can execute. Since the risk is low, they can make a very competitive and tight market. If they had a 100ms delay to hedge, they would need to make a bigger spread to compensate for the risk.
You may ask, well, why do we need market makers at all? Well, you don't have to use them. You have other options for trading. Call up your broker and ask. What you'll discover is that these other options have far larger transaction costs.
A lot of the complaints about HFT I've seen on Hacker News seem to think that no one should get paid for market making. I don't think that's possible. (And if it is, I'd love to know how.) Transaction costs are an inescapable part of trading; ultimately, you're faced with the choice between tilting your hand to other traders or paying some intermediary to take on risk for you. Generally, it costs less to choose the second option.
You argument is that Vanguard is a market maker, Vanguard likes HFT, thus HFT must be good. That doesn't really follow at all. Vanguard could be arguing in favor of HFT for all kinds of reasons, one of them could be that they have the money to rent server space right next to the main exchanges, and pay for the ludicrously low latency connections to the exchanges, so they benefit from HFT by being able to be quicker than others, and they don't want to loose there edge.
Now a small team of researchers can make markets on many securities at once. They don't have to make as much money on each, so the spreads get smaller. Additionally, instead of a monopolistic specialist, these professional traders compete to win transactions by making better prices (higher bids and lower offers) than others. Many market makers try to keep a neutral "book" of positions, so if they sell some GS they may bid higher in other bank stocks to reduce their exposure to market risk. The faster a market maker can update his quotes or hedge when conditions change, the less spread he needs to charge.
http://meanderful.blogspot.com.au/2013/01/hfts-dirty-little-...
See, again, per the article, Chesterton's Fence.
Another thought on HFT's value to society: I generally agree that HFT does not provide "direct" value. There are clearly "arbitrage" opportunities that HFT firms consistently profit from. However, by definition, those arb opportunities will either be taken by someone else, or enough people will run into grab them such that the arb goes to zero. If there are consistent arb opportunities then something needs to change - namely the regulation which structures the market and allows people to take consistent advantage of it. Thus new regulation is issued and the market, inherently a complex and intricate system, becomes stronger as a result. This is maybe where the value to society lies. and maybe this is an idealistic view of how regulation, society and HFT interact in this case.... ?
Like you said, if enough people do the arbitrage, it converges to zero very quickly. There's little profit in it and everyone reaps the benefits of near-instantaneous accurate pricing.
Commenting tricky / non-obvious code is well recommended.
Writing this software would be very scary for me -- I'm confident in my work for the most part but I'm only human. To then let a "for-loop" go wild making thousands of trades per second on the entire market with real money against other high frequency traders no less.. man... you gotta be really really really really sure of your code!
In the HFT world, you may have only days, or sometimes hours, to deploy. Yes you still write tests, and a bazillion safety checks when you deploy, but if you manage your risk down to zero then this is definitely not the most profitable way to go.
Then if the regulation changed, someone would grep through the use cases, and see what behavior is occuring/ how to modify the code base to cope with the change in regulations (this occurs a lot). Of course most finance firms don't do this and actually violate regulations all the time. There is literally a gold mine for the SEC/FINRA to go after, but the government is fairly incompetent at mining through the data and holding companies accountable. The reason being that they don't pay enough to hire the talent that would uncover the violations.
My understanding is that companies such as this rely purely on manual testing. Thus it is easy for things to slip through the cracks. Usually they never get hit by penalties or get caught so the pressure to do better is not there.
In this case, of course tests really do prove the absence of bugs rather than just (not) proving their presence.
I really don't like the whole "unit tests lets you not worry about breaking things" view. That's not confidence, it's overconfidence. And it doesn't even mostly work for things where "correct" is more fuzzy than "eventually produces this exact output".
What would you propose as an alternative?
With that said, libraries that send orders to exchanges are required by the exchanges to be certified against their expected behaviors. If you want to send routed ISO orders, you have to test that with them. When making changes to said libraries, it can often be good to re-certify to ensure things are still working as expected.
And retail brokers already do what you suggest. All their marketable flow gets sold to off-exchange market makers, while their limit order flow gets routed to exchanges that pay the highest liquidity rebates, Reg NMS just mandates what price it can trade at: http://news.indiana.edu/releases/iu/2014/02/study-of-potenti...
http://www.bloomberg.com/news/articles/2012-08-02/knight-has...
The SEC order there is an interesting read for anyone involved in large scale transactional systems: https://www.sec.gov/litigation/admin/2013/34-70694.pdf
When something like this happens, it's always a series of oversights.
1. Someone forgot to add comments or test cases while writing the code (or perhaps he wrote it but people are not running the test cases before commits).
2. Someone else thought that the code is dead and deleted it (and skipped the test cases if there were some covering that particular scenario).
3. Either the person deleting the code didn't wait for the code review or code reviewers missed that as well. (If code reviews are non-existent, it's a disaster waiting to happen).
So in the hindsight, write comments, write test cases, run test cases and do code reviews carefully.
Fruit stand 1 is the cheapest, fruit stand 2 is the 2nd cheapest, and fruit stand 3 is the third cheapest.
You want to buy all the apples out there.
The rule is that you have to go to the cheapest fruit stand first, and exhaust their supply before moving onto the next cheapest fruit stand. If you want to go to ALL the fruit stands at the same time you need to tell them you are an 'OK' guy and are really buying fruit everywhere.
Well, someone deleted code and ignored one of the fruit stands. This violated a rule and cost the fruit buyer a lot of money in penalties.
In Europe and Asia, there are competitive exchanges, but no rule to force people to route there. There are some guidelines for brokers with customer orders to have a duty of "best execution", but for proprietary traders or professionals, they can do whatever they feel is ideal.
And guess what, there are very few "trade throughs" or cases where markets invert with each other despite lacking such a rule. People usually act in their economic best interest without a law mandating they do so. If it happens, it's an arbitrage that someone will eliminate very quickly.
Intermediaries make up 99.9% of the market because very few individuals would bother paying to connect to a dozen stock exchanges if the can pay a broker a few bucks to do their trade instead.
This rule seems designed to protect the consumer when the consumer and the stock brokers goals are not aligned.
If they weren't a BD, their BD (which is required to ultimately reach the markets) would be on the hook for letting this happen. ISOs are something that get a lot of compliance/regulatory scrutiny and most BDs don't let their customers use them. If they are allowed, there's a lot of reporting/transparency/surveillance to ensure they are issued properly. And as you can see from this fine, there are a lot of rules and corner cases one must cover and apparently even the best firms mess it up one way or another.
Latour's previous fine (read Matt Levine's continually excellent articles about this stuff) was for "Net Capital Requirements" which are designed to keep customer money safe, particularly in the presence of other customers who might blow up ... of course Latour has no customers, but it is still the rules. [Although IMO this one was a grey area where Latour probably had better models than what the rules required, but since they didn't match the rules properly they got fined.]