The Accidental HFT Firm (2018)
web.archive.org
web.archive.org
I never came across isolating CPUs, TCP offload, FPGAs, layer 1 switching until I joined a HFT shop. Its like a whole different way of looking at computing system architecture.
We spun them into their own org, and now its "their money" they are spending. Their bill is $2k/m! Amazing!
Making things fast though is a whole other world.
They are optimized for throughput, not latency. If you want to do things in microseconds instead of anything from milliseconds to seconds you look at all the lovely code everyone has written in the last 40 years then throw it on top of the burning pile of cash.
I work in an area where microseconds matter and have to write everything from scratch in kernel mode C because nothing else cuts it.
Neat. Is that all for custom processors too?
However, when Google buys a WAN link from NYC to CHI, they pay $XX dollars because they don't care about the latency too much. When a HFT buys that same link, they'll spend three times that much just to get a link that is a few milliseconds faster.
Also, the colocation costs at the major exchanges (NASDAQ, NYSE, CBOE) are insane. A cabinet in one of those data centers can be 3x as much as a cabinet somewhere else.
I guess what I'm saying is there are a lot things in the HFT world where you have to spend a lot more money than you would if you were doing similar amounts of work at any other tech firm.
Getting latency in the micros with low jitter is extremely difficult and requires a bunch of custom software. And getting into the nanos requires a bunch of custom hardware.
For me, HFT is the more honest, and less evil, way of making a buck.
If you don't know how that works, just look e.g. at TicketMaster:
https://en.wikipedia.org/wiki/Ticketmaster#Criticism_and_con...
> People would just cut them out if all they did was skim a bit from everyone.
The point is that "just cut them out" is not so simple in many cases. See TicketMaster and the fact that even artists hate them but have to deal with them.
Presumably you buy all your goods directly from growers / manufacturers, rather than using retailers who are simply there to skim off the top?
> Also, HFT has contributed to a flash crash before, so it's not free of externalities either.
You're confusing the effects of the industry with the effects of incompetent actors within that industry. Besides, the negative externalities are trivial. Institutional holders barely notice or care about flash crashes.
> You're confusing the effects of the industry with the effects of incompetent actors within that industry
Doesn't that same point apply to Cambridge Analytica, the singular example of a negative externality from ads in the comment I replied to?
Whether HFTs and/or the broader concept of middle-men are better or worse than ads, they're at least still in the same ballpark. I definitely agree with the GP comment's sentiment:
> It fits in the "our smartest people are making people click ads" category.
What if you're in the northern hemisphere and want to buy strawberries in December? Or you want a pint of milk, but your local dairy only sells it by the tanker?
The point is effective middlemen help markets function even better than if it were just people with things to sell and people who wanted to buy them. They provide a valuable service, which the market also coenienhly ascribes a value to.
> > You're confusing the effects of the industry with the effects of incompetent actors within that industry
> Doesn't that same point apply to Cambridge Analytica, the singular example of a negative externality from ads in the comment I replied to?
I think there's a difference between incompetence and malice. But broadly, yes. I don't think you can write off the ad / data broker industry because of a few bad actors.
> Whether HFTs and/or the broader concept of middle-men are better or worse than ads, they're at least still in the same ballpark. I definitely agree with the GP comment's sentiment:
> > It fits in the "our smartest people are making people click ads" category.
The difference I see is that it's possible to externally measure and quantify the benefits / externalities of HFTs to the market participants. It's much less clear cut in ads, which is why some people feel ickier about that particular industry.
But I don't buy the whole "smartest people doing useless things" argument. You can't make people work on "useful" things, and you can't discount everything that a smart person does in the ad industry as useless to humanity. It's much more complicated than that.
You do understand how that's different from HFT, right? That's more of a market making job. HFT is like hiring The Flash to be your grocery store clerk, who notices you reaching for the milk, and runs in to empty out the shelf and raise the price by 0.003 cents by the time your hand gets there.
HFTs provide liquidity, which by definition is providing what people want to buy/sell, when they want to trade it and it the desired quantity.
> HFT is like hiring The Flash to be your grocery store clerk, who notices you reaching for the milk, and runs in to empty out the shelf and raise the price by 0.003 cents by the time your hand gets there.
This is not what HFTs do. They can't see your "hand reaching", they see your quote hit the book at the same time as everyone else. It's not magic.
They're literally the market makers. Your complaint appears to be that they're good at reacting to new information, i.e. being good market makers.
Every exchange gives "good standing market participants" aka HFT market makers first dibs on orders. If a gallon of milk goes on sale and I'm waiting in line with a market maker I will never make the sale, it'll go the the HFT firm every time.
Or are you talking about exchange hosting, where you get lower latency if you pay more to colocate?
Grocery stores to the things that are difficult for me to do, like sourcing and sorting a wide variety of produce and products. Also, they use economies of scale to make it cheaper than if I were to do the same thing myself without an intermediary.
HFT firms neither enable customers to do things they can't accomplish on their own, nor do they save customers money by exploiting economies of scale.
Study after study has shown this empirically the be false. It's uncontroversial and established that spreads have significantly tightened (that means it's cheaper for market participants to trade) because of the rise of HFT market makers.
In more concrete terms, what does it achieve? Genuinely curious. It seems to me that "enabling capital to get better returns" is not a noble goal in itself (especially considering the gilded age levels of inequality we experience in the developed world today).
The liquidity from market makers and HFT firms have ushered in this age of free trading for the lay man. This was not possible even 10 years ago.
> It seems to me that "enabling capital to get better returns" is not a noble goal in itself
The majority of Americans own stock. Sabotaging the middle class's retirement to give a middle finger to some billionaires does not seem noble to me.
Okay, so it makes it easier for some people to become market traders.... but is that beneficial? The money that HFT firms are making has to come from somewhere. The only way an average person can be a market participant and come out ahead, is if they have some inside information that the HFT firms do not. Risky stuff. And the ability for people to make trades that turn out to be winners, is reduced by the massive and aggressive trading done by HFT firms.
The massive profits from HFT come from somewhere. Someone is making less because the HFT guys made more.
HFT firms do not benefit the average person- unless you define "benefit" as allowing them to have an easier time gambling at a game that has become harder to win.
Secondly, their income comes from the spread, same as any other market maker. They're able to operate at much lower profit margins exactly because they're HFTs. Making a cent a trade makes no sense if you're doing 3 trades a day. However, it'll quickly become an attractive business if you're doing millions of trades per day. Because spread is what they compete on (and competition is pretty healthy), this drives trading costs down for everyone else.
From places like the Tabb Group? "According to research conducted by the Tabb Group, the top 5 High Frequency Trading (HFT) firms make an average of $2 billion a year in profits."
> However, it'll quickly become an attractive business if you're doing millions of trades per day
Attractive as in profitable? Honestly I don't know what you're trying to get at here. Yeah, their margin is small, and they make massive profits by replacing that with volume. Why is that a controversial claim?
> this drives trading costs down for everyone else
It really feels like you didn't even read my last comment. Okay, let's grant that they lower trading costs, while extracting many, many billions of dollars from the market.
I have a great idea for a business. I'll lower the door charge at vegas casinos, in exchange for bumping the house edge up by a fraction of a percent. There you go, I have made it a lot cheaper for everyone to get in and participate in the craps market. I have offered a valuable service and am in no way a parasite.
What's the difference between my business plan, and what you're describing?
Even ten years ago, commissions were sky-high in comparison. Twenty years ago, it wasn't possible for most investors to access markets directly at all. You had to go through several layers of intermediaries just to get market exposure (it isn't just execution costs, HFT is an enabler for other innovation like ETFs)...you paid your wealth manager, the fund manager, who paid the broker, etc. Your returns were getting cut by a significant amount.
This trend is still working it's way through the industry but even versus ten years ago, you have seen massive improvements for consumers.
For retail investors like me, HFT firms allow me to get better trade execution.
Avoid the tendency to exaggerate your feelings.
Targeting a non-deterministic venue comes with its own set of challenges which are not purely technical. You don't know how long it will take for your order to reach the matching engine nor the likelihood of things changing before it gets there. Simulation is also more difficult. So you have to rely on alpha instead, which is a different game.
The CME is also quite deterministic (largest derivative exchange in the world, big names there are the S&P, treasury notes, eurodollar, wti..)
Off-the-shelf best-in-class solutions run at 30ns (see STAC T0). The fastest people are faster than this.
So, when the underlying moves (the stock) the option should move according to the model. But there are time inefficiencies in the market so a HFT can trade against the misplaced option and lock in a bit of profit. Do that a million times a day any you may make a few bucks.
Determinism is achieved there by ensuring there is only one path to the order gateway with everyone having cables of equal length, and that the order in which network packets hit the gateway is guaranteed to be the order the matching engine processes the packets in. Likewise you need to ensure everyone sees the market data at the same time (equal length cables, multicast). Most exchanges fail hard at guaranteeing this.
For your option example, I think you paint a somewhat misleading picture of options trading. You can't trivially arbitrage the spot and the option. There are several unknown factors between the two. An options market-maker works by taking on risk, decomposing it based on the sensitivities of the price to the various inputs, spreading it across the option universe and biasing its prices so that trades statistically take them back to zero. It's closer to a dispersion strategy. For some markets, delta (sensitivity to the underlying) risk is bad to take on so you try to hedge out of it fast, or you avoid trading options with too much delta in them. Trading high-delta requires being fast to react on underlying moves. But that's just an extra requirement when operating this kind of strategy and isn't how you make profit. Even taking out mispriced options after a move is usually mostly done as a way to get out of risk for cheap, and what you'd care about most there is modelling the spot-volatility dynamics.
On the other hand, your edge can be explained in very simple terms: in the article, their edge comes from the fact that they had a closer colo than other market participants. ITG made money by front running customers[0], cheaper financing means you can arb an ETF more aggressively than the other guy etc.
Like, there are straightforward mechanical reasons why you're making money. No amount of clever hacks can help you if you're at a structural speed, informational or cost disadvantage.
[0]: https://www.reuters.com/article/us-securities-regulation-itg...
However, there are plenty of firms that operate in the low frequency space. Long/short shops are typical for this type of trading. They don't care about order speed. They spend all their time trying to find inefficiencies or hidden correlations between financial instruments and profit from them.
Low frequency tends to be highly research compute focused, while HFT is highly trade compute focused.
Source: hft quant
For me HFT is the most interesting job one can have and it requires you to be good at a variety of things to make money: low level tech, quant modeling, risk management, business sense so you can cut good deals, etc.
Unfortunate the industry is unfairly maligned by people who really don't know what they're talking about (like Michael Lewis in Flash Boys). It's cheaper to trade now (especially for retail) than it ever has been in history and the positive effect this has had on the capital markets and the economy in general cannot be understated.
So that's a huge part of the game, try to detect toxicity early and widen your spread as much as needed to stay out of it. In practice it means that whenever toxic flow hits a book the book's spread (the distance between the bid and ask) will widen as well as the new price is sought.
Pay for Order Flow is a big tool here because you pretty much know that all trades done on free brokers are non-informed. If you subtract that out you increase the signal-to-noise ratio on predicting toxic flow.
Finally, lets say you are hit and take on a bunch of toxic inventory. Now you need a place to dump it to. Your options depend heavily on your situation, but you might either hope you have complementary non-MM strategies which can absorb this inventory or you might hope that you can quickly shovel it off to another venue (another exchange, a dark pool) where you won't get hurt so badly.
And if you can't do any of that, you take a loss.
1. https://en.wikipedia.org/wiki/Spoofing_(financhttps://en.wik...
However, if you place an order that you do not intend to get filled to make the market think there is demand when there actually isn’t, that is illegal.
So, how do they prove this? Well, proving intent is hard, unless you’re dumb enough to send emails saying, for example, you want a smart order type that won’t get filled.
Some context here. Software design-wise, Orc was stuck in 00s. Not scalable, hard to support, hard to add new features, etc. And the corporate structure did not allow for innovations.
Somewhere in the end of 00s group of engineers left Orc (the company) to rebuild the trading platform from the ground up, and formed a new company, called Tbricks. Tbricks was such a success that Orc (again, the company) spend a lot of cash to acquire it, put key people from Tbricks to the leading positions and sunset legacy (classic, as we called them) Orc products.
After the merger, the company was renamed to Itiviti. I came there ~a year after the merger and worked for five years as an SWE in the Tbricks team. The goal of the merger was to migrate all Orc customers to Tbricks in a year or two. Five years later I left the company, and there still were a few of major customers not migrated yet. Orc was supported by one or two persons full-time, and a few of greybeards who moved on to the new projects, but still had the knowledge. And I think it is still bringing few millions USD a year in license costs.
"Mr. F., one of the original hires, coded what we called Strategy Zero. It was simple: have no position, wait for an option edge of a reasonable size, at least a tick. Then, hit it. Wait. Dump it out and take a tick profit."
In this case, the strategy involves 1) seeing that you can buy a security at $10, 2) knowing that the security is actually worth $10.01 and will move to this price in a few seconds, 3) lifting someone's $10 offer, 4) waiting the few seconds for the market to move, and 5) hitting someone's $10.01 bid for a $.01/share profit.
This liquidity taking strategy is not similar to a liquidity providing strategy. Liquidity providing is leaving orders on the order book for anyone to take and gaining money from one person who buys your shares for $10 and from another who sells shares to you at $10.01. This involves more risk to your business.
I think that if you want to debate the merits of whether working at an HFT firm is an actually useful service, you have to look at how often the HFT firm provides liquidity versus takes liquidity. Providing liquidity more obviously tightens spreads and improves prices, as you can't be a successful liquidity provider unless you provide the best prices first. Liquidity taking, however, involves lifting orders from exchanges that you know are incorrectly priced, which seems to me like a more obvious "bad" (or at least less good than liquidity providing).
I think you have that backwards. If you always sell at $10 and buy at $10.01 you lose on every trade.
I'd pay $100 for the book
I also used Orc back in the day to trade options, and it's complicated. The thing about options is you have a whole bunch of instruments that are closely related, priced off the same underlying. So there's a smoothness constraint in there that you need to think about (have a google for "non-arbitrage vol surface").
Quickie options tutorial: when you trade an option, you have curvature risk. Meaning that when the price moves, the hedge is not some fixed number of contracts. This can be good or bad for you, depending on whether it's concave or convex. On top of that, normally if your gamma is positive (ie you make money either way it goes) you are paying for it by the option losing value (theta is negative). There's a bunch of these greeks, which are just the sensitivities of the option price to various inputs like price, implied volatility, interest rates, etc. Different parts of the surface have different greeks.
So it's a bit of a balancing act. A sophisticated MM might decide to hold all their greeks within some range. This would mean slightly skewing your prices in some area of the surface in order to entice someone into trading with you, plus keeping an eye on the underlying.
As for market making, it would be nice if prices just stayed the same. People would come and buy things from you for 101 and they would come and sell things to you at 99. You'd just be transporting one order to the other one in time, the perfect middle-man business. The customers win because they don't have to meet each other, and you win because you have 2 more in the account.
Sadly that isn't what happens. Sometimes a guy comes by who wants to buy at 101, and then 102, and then 103. The market has moved, and you being such a nice guy have sold at those prices and are now short when the thing is more expensive. This not cool, but the real problem is that you were standing there the whole time offering product while not knowing what it was worth. What's worse is that you're more likely to get deals when this is the case. This is the adverse selection issue: you have a bunch of competitors when things are plain sailing but then when things move you "win" the deals.
So, what to do? One thing you can do is just not trade with that guy with the massive orders. Find a bunch of punters who aren't dealing massive orders, and deal in opposite ways to each other. Often what actually happens is some guy who is good at marketing builds a business doing this, and since he doesn't know how to also do the market making, he cuts a deal with a market maker. This is what you might have read about. You split the earnings between the marketing business and the risk business.
All of the above is of course helped by being really fast. If you see a bunch of sellers, you pull your bids quickly so they don't sell to you. If you see that you need to move your prices, you do it fast so that you're first on the new price.
There's even more to this game in the form of arbitraging other people's orders, but we'll leave that for now.
https://news.ycombinator.com/item?id=16222530 (5 years ago, 32 comments)
https://www.reuters.com/article/us-securities-regulation-itg...
Their MM operation was making lots of money according to someone I talked to. Just not enough to offset them setting it on fire.
Alameda did many illegal things, including lying to investors about their returns to get funding. And pump and dumping a huge lot of shitcoins which SBF created or funded.
So I don't doubt they'd also frontrun their customers.