High Frequency Trading and Finance's Race to Irrelevance
blogs.hbr.org
blogs.hbr.org
1. HFT does NOT take anything from most investors most of the time. Most investors trade on a much longer investment horizon than any HFT firm. If you are a normal investor, you can lump up most of HFTs into the same bucket as the exchange itself. Most of them engage in some combinations of (i) pure arbitrage (ii) very short-termed statistical arbitrage (iii) market-making based on (i) and (ii). None of this is really that relevant to most investors. The only time HFTs can screw over a lot of investors is when their software goes awry. But this is a risk that any computerized system has, high frequency or not. If there is a major bug in Chicago Mercantile Exchange's matching engines, that would be a total disaster.
2. HFT firms do have huge execution risks: if you are making a two-sided market, there is a chance you can get "swept", meaning when the value of the underlying moves faster than you can react, you get filled on one side of the order without a realistic chance of hedging for a profit. HFT firms do a ton of research into estimating their execution risks. (edit: This point is often glossed over, making HFTs look like this evil superpower group of nerds exploiting other investors. That's not the case)
3. Finally, most HFT traders don't go into the profession solely for the money, just like most people do not apply for YC solely for their passion. Most of my former coworkers weren't that greedy and led pretty modest lives despite making hundreds of thousands of dollars. For them, high frequency trading had a locally optimal balance of tackling intellectually challenging problems while getting paid handsomely.
I won't say HFT is the most valuable thing that its practitioners can be doing. I think the (trading) world would operate just fine without all these micro-second level transactions. But they are NOT the next subprime mortgage, and I just hope folks stop commenting on stuff they have no clue about.
Also curious on your perspective of their role in market making. Do HFTs ever end up buying a larger block than they know is being demanded and end up holding some of the remaining for a time?
> I won't say HFT is the most valuable thing that its practitioners can be doing. I think the (trading) world would operate just fine without all these micro-second level transactions. But they are NOT the next subprime mortgage, and I just hope folks stop commenting on stuff they have no clue about.
Your post is interesting and I learned from it. However you admit that HFT is not the most valuable thing smart people could be doing - this is the reason that many dislike it. HFT provides little value to humanity or society. I do not think this makes me 'misguided' or 'have no clue' and I don't like how this makes it okay for advocates to sweep away concerns about it. We could easily say the same thing: algorithmic trading in itself is misguided: it does not use technology to solve socially useful problems such as global distribution, food production or folding proteins to better understand drug interaction and find cures for disease.
The counter argument to this would be that said efforts help fund more genuinely beneficial work such as their recent work in robotics and proliferation of internet availability globally, but how the net balance of talent productivity plays out is murky. The ad team is a large, multifaceted team inside Google.
Finance guys are much more aggressive about capturing more of the revenue they create in salary and bonuses, while engineers seem to shrug at being "overpaid" while letting proportionally more of their value go to execs, investors and giant cash hoards.
Now if you are talking about ibank traders who HFT are replacing, that is true. Their salary bands are enormously higher than engineer pay scales. Sort of explains why they direct so much vitriol towards HFT.
Remember, that the vast majority of people who make large compensation in a finance firm are essentially sales force. They are selling complex products, financial services, or operations.
IE: You can purchase a stock in Euros in some European exchange, OR you can purchase a stock in Dollars on the NYSE. In both cases, you get the same value.
This equivalence in value only exists because traders perform arbitrage on fractions of a penny over the course of milliseconds. And thanks to HFTs, any divergence in value larger than a penny or so is arbitraged away within miliseconds.
This is a solid service for all involved, and HFTs are the ones who take the risks behind arbitrage. (IE: if the markets move in an unpredictable way)
Yet the HFT people are making tons of money. Where is it ultimately coming from?
Most of the money in HFT actually comes from those previously more rapacious middlemen's bonus.
That and for most people who actually understand the mechanics what is going on, the current system seems way better for the vast majority of people than the one it replaced, so we have little reason to change.
I'd say the financial markets have been there for 6-8 years already.
The first person to discount a bill at the very first bank was "abstracting itself from the creation of real-world value".
Later, I'm sure someone said the exact same sentence that you just did when the first exchange traded fund (ETF) was created. I know for a fact it was said throughout the 90s in relation to derivatives, etc.
In the future, HFT as it exists today will be a quaint feature of a bygone era that will be referenced to illustrate the problems with whatever new practice has superseded it.
What does matter is where you are on the spectrum and how fragile that makes you, but there is no natural law limiting how abstracted you can be and how fragile you can be. By some relative measures we're impossibly abstracted and fragile, but by other relative measures (as yet unknown) we're adorably simple.
Getting pennies skimmed off of my trades is not nearly as big a deal as a 0.1% yearly recurring fee on say... a $100,000 retirement account. (ie: $100 / year). Those tiny percentages add up to a huge amount of money in the long run.
I've heard of reports of 1.5% or even higher management fees (or ~$1,500+ / year in this example)
HFT seems like stealing only because they are using advanced technology methods to make money, whereas in the 1980s and before market makers routinely manipulated the spread and pocketed likely a similar percent of trading profits.
At least the online brokerages and pioneers of HFT broke down the antiquated 1/8 stock price ticks and lowered the spread and per share transaction costs.
Not quite. HFT uses information that you probably don't have, and acts on that information using software and hardware that you probably don't have[0].
You can obtain that information and that software/hardware - it's just expensive (high barrier to entry)[1].
As explained above, HFT exists because portions of the market are not perfectly competitive in the economic sense[2], but that's always been the case (and arguably was more so before the advent of computerized HFT).
[0] It's not so much that you don't have the information - it's that the latency is too high for most players.
[1] At least some portion of these costs are very real - the cost of maintaining low-latency connections between exchanges, colocating hardware, etc.
I read a copy of the Mentat prayer that had been written for traders somewhere but have lost track of it, that was something like 'by liquidity I give my transactions life, by speed I give them value, by value I make me wealthy' or something along those lines. It was much more clever than I could come up with, but modest google fu has not revealed its location.
As for the argument that the value they provide is "liquidity", that also seems dubious. I've seen that argument thrown around a few times now, but I'm not entirely convinced that HFTs, as they exist and work now, are achieving this goal fairly, nor that this is necessarily the best solution.
Unless you can say that the "price extracted" is fair and self regulating without any clear and transparent oversight, I think there is probably room for improvement and a benefit from increased scrutiny. However, as I said, my knowledge of the internal workings of the markets is still pretty limited.
(I'm told by knowledgeable people that there are indeed some shady things that happen in HFT, such as quote stuffing, that ought to be fixed, which makes sense. But I don't think that brings us anywhere near "shadow government", nor does it seem like a damning problem for HFT as a concept.)
The old time finance guys have no idea what a back-propagating multilayer neural network is and how it's eating their lunch, and there is resentment and suspicions of foul play. I am more suspicious about the suspicion. You can pitch a tent and set up a hedge fund like Renassiance or DE Shaw without the blessing of an investment bank, and perhaps the banks resent this, despite the fact that they're doing HFT as well.
What value did they add to the market? Bringing buyers and sellers together? Nope. Provide capital to the market? (Which is why stocks supposedly exist...) Nope. They end every day owning nothing.
Lewis's book also shows that many banks and brokers instead route some trades through their own internal 'markets' first, trying to find a match, instead of just trying to get the fairest price available.
Everyone gets screwed. There was a HFT trader who boasted that they went years without ever having a losing day. The only way to do that is in a market that isn't fair, where not all parties have equal access to information.
http://www.chrisstucchio.com/blog/2012/hft_apology.html
http://www.chrisstucchio.com/blog/2012/hft_apology2.html
http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html
https://news.ycombinator.com/item?id=7546104
Can you respond to my post in that subthread?
(1) Someone wants to move a block of 50 shares, but there are only matching orders in the order book for 25 of those 50 shares. The seller splits his order so that the 25 shares move; then either the other 25 sit there until a matching order appears, or the seller changes his offer price for the other 25 shares so that there's a match.
An HFT in this scenario is no different from any other investor deciding whether or not to match an order in the order book; he just moves faster. And since in this scenario there are no other matching orders in the order book without the HFT, the HFT adds liquidity in this scenario.
(2) Someone wants to move a block of 50 shares. There is no matching order in the order computer in which they place their sell order, but there are matching orders in other computers. The HFT sees those other matching orders before other market players, and matches them. Then he places his own order in the computer where the original 50-share sell order is, and matches it so it executes.
In this scenario, the HFT adds no liquidity; all he does is shift the profit on the trade from other market players to himself, by taking advantage of the latency between different computers on which orders are placed. The trade would have happened anyway once the two computers reconciled their orders.
The reason is precisely because HFT market makers are fast and sophisticated enough that they want to trade with all small block buyers, so they are resting orders at every trade-able price on every exchange they can.
The speed arbitrage comes from the fact that if they see a level being removed at one exchange it is a demand signal, so they change their own prices on the other exchanges. This is faster and cheaper than going out and sweeping another level (which requires paying the spread).
Essentially, there is no liquidity difference in this case, as all the same liquidity providers are there. They are just pricing their liquidity more accurately.
That seems to be a disputed issue right now; claiming that scenario 2 does happen in practice was, as I understand it, one of the main points of Michael Lewis' latest book on HFT. He may be wrong, but I don't think it's as simple as saying "they're so fast that they don't need to".
An HFT may have orders at every tradeable price at one instant, but what about the next instant when the set of tradeable prices changes? On an exchange with multiple computers executing trades, that is precisely the time when scenario 2 can come into play, and the fact that HFTs are faster than everyone else is precisely what gives them the ability to play scenario 2 in this situation.
> if they see a level being removed at one exchange it is a demand signal, so they change their own prices on the other exchanges
To be clear, my scenario 2 was not talking about matching prices on different exchanges. It was talking about a single exchange that has multiple computers executing trades. Conceptually, the single exchange is supposed to have a single order book, but because there is unavoidably latency between the multiple computers, computer A's current view of the order book may not match computer B's. An HFT that can detect the mismatch faster than the exchange's own computers can, can run scenario 2 in that situation.
Let me come right out and say it. Michael Lewis is not "wrong" because he never actually says that scenario 2 happens. The reason he doesn't say that, is that there is no proof that it happens. The reason he couldn't find proof is that it doesn't happen. Any cursory examination of the HFT side of the industry will show that it doesn't happen. He never even does this cursory examination. Go through the book, catalog when he talks to the HFT side. He doesn't do it. The closest he gets is interviewing a developer who is being persecuted by his own buy side firm. Michael Lewis implies that something happens but never actually shows it happening. He does this either due to incompetence, malice, or prejudice.
To prove my point, let's do a thought experiment. Let's say I have 2 HFT systems, each of which are equally fast and are equally good at determining if fills indicate true price movements or blips in the market, and are equally good at evaluating outstanding risk. HFT A's strategy is to quote "small" size in a single market and then when it sees fills it thinks indicate price changes to buy a level in order to raise the price in a particular market. HFT B's strategy is to quote at every level in every market and when it see's fill's that indicate price movements cancel it's order's that are at a bad price.
If their price discovery or risk algorithm are perfect the best case scenario is that HFT A and HFT B are paying the exact same price for the right to trade. If there is any slippage in their price discovery or risk algorithms, HFT B wins as it is not paying for the privilege to trade. QED, algos that don't pay up to jump priority queues do better than algos that do. In the blood thirsty world of HFT market making this inefficiently quickly leads to HFT A going out of business.
As far as your scenario about an exchange that is open to distributed systems attacks, that is akin to asking about a casino that has a secret flaw in their poker shuffling. Yes, that would give an unfair advantage to anyone that found the flaw, but it would also kill their ability to host poker games if it became public knowledge. IE exchanges have a huge incentive to make sure this doesn't happen.
> algos that don't pay up to jump priority queues do better than algos that do
It looks to me like all of this is addressed at the version of scenario 2 that I was not talking about, the one where the HFT is working across multiple exchanges.
> exchanges have a huge incentive to make sure this doesn't happen
I'm not sure the casino analogy quite holds here, but I'll have to defer any further comment until I can find the article I mentioned somewhere in this subthread, that went into how this scenario would work in more detail.
kasey_junk seems well-versed.
HFT is like a litmus test for HN. It is a renewable resource.
1) If you were doing this through a broker, and that was the state of the book, then they would just route your order to BATS (if I'm not mistaken that's a legal requirement for brokers). 2) If you instead placed the order to be on Nasdaq-only, that would lock (have buy and sell orders at the same price) the national book, something which is not allowed due to Reg NMS. So Nasdaq could not display that you want to buy at $10.00 anyway, and (I'm not sure how they implement their stuff) either hide your order or display it at another price.
So I'm not sure how exactly you're framing this situation. If the HFT's computer can see that your order exists, it's because it posted. If it posted, it could not have had the limit price that would've locked the book.
I may be misunderstanding what you mean when you say at the end that the "two computers reconciled their orders" - because nothing like that does happen. If they are computers at the same exchange, then no, that's simply not how matching engines work. If at different exchanges, then there is no reconciling, your order would not have posted at the right price, or your broker would have not tried to get you best execution on your order(which they are bound to doing).
No; as I responded to another post in this subthread, I mean multiple computers belonging to a single exchange, such as NASDAQ. In one of the previous HN threads on this topic, somebody linked to a post that showed the situation I'm talking about; I'll see if I can find it.
Remove the liquidity (which comes at quite a steep price it seems), now what? There seems to be plenty of capital looking for investments.
The immense effort spent on HFT could now be concentrated on a) finding things worth investing and finding better ways to do so b) create more things worth investing in
The price of liquidity is less than the inefficiency (loss) that comes from an illiquid market (people who mutually want to execute a trade but cannot). This is rather straightforward to demonstrate mathematically.
Just because HFT extracts a premium (similar to a transaction fee) on the spread doesn't mean both parties (and the rest of the market) can't still benefit.
> The immense effort spent on HFT could now be concentrated on a) finding things worth investing and finding better ways to do so b) create more things worth investing in
This is analogous to the argument often levied by non-techies at the entire tech sector: "They could be developing software to cure cancer, so why are they creating more apps to send selfies to their friends?"
The "premium" could then be split between the both parties. The cake is larger (but maybe not split as evenly).
The argument about webapps vs. curing cancer is also levied by techies right here on this site and might be valid, (yet weaker, since webapp-big data might eventually help with medical-big data?). Either way you have not made an argument for HFT with your last sentence.
And IMO that's often a good argument, though it's not really an argument against the techies but against our society as a whole. The fact that techies can get rich making selfie apps much more easily than by curing cancer, is a sign that incentives in our society are screwed up, in much the same way as the fact that smart people can get rich by running HFT firms much more easily than by running companies that directly produce goods and services.
It's a little bit like saying the guy who runs your 7-11 is a scumbag because he charges an extra 50c for a carton of milk, and "who needs that liquidity, they could just wait until <insert large supermarket chain> opens and get their milk there". Sure, but they might want the milk now. And no one forces them to get it from the 7-11. It's a free market.
The 7-11 provides the trade functions "assortment of goods" and "transport" and as such is valuable. The analogy to HFT does not hold.
"Large supermarket chain" also provides the trade functions "assortment of goods" and "transport", so what exactly are you paying your extra 50c for? Liquidity.
What a bunch of assholes those 7-11 types are. Charging us an extra 50c for milk. The nerve.
However, I'm afraid your proof-by-analogy does not work. If you actually know what you are talking about, why not explain the current financial system clearer and with more information?
In turn I promise not to burden this discussion with a dissection of your analogy, or even my own set of analogies which would beautifully fit my current bias, and which you would surely reject outright.
Market makers generally, smooth these time disparities. As long as the smoothing is cheaper than the price disparity then it is a net positive for the "real" participants. How do we determine if this is cheaper? Quite simply, the market! You can trade non-liquid products electronically. The price spread his much higher in those products.
But more importantly, HFT is not the same as liquidity. HFT only works in the first place because there are already market orders being placed; if no one but HFTs wanted to buy or sell stocks, no trades would take place. So the HFTs, by themselves, can't provide liquidity; they have to wait for someone else to do it, and then execute their trades to help narrow the bid-ask spreads.
I'll agree that there is some value to narrowing the bid-ask spreads; but is it enough value to justify the enormous effort and the enormous amount of human talent that goes into HFT, that could be doing lots of other valuable things? I don't think so; I think all that is a sign that the incentives in our society are screwed up.
HFTs by themselves can provide liquidity (like any other investor), namely by entering a limit order in the order book. They can also by themselves narrow the bid-ask spread, namely by entering a limit order at a price more competitive than the current first limit.
Of course, someone else has to send a market order to the exchange for a trade to execute.
Yes.
> HFTs by themselves can provide liquidity (like any other investor)
Exactly: like any other investor. Meaning, the "high frequency" part of HFT is not what provides the liquidity. (I admit that I should have made that clearer in my previous post.)
Suppose an HFT enters a limit order that is more competitive than the current limit. Either some other investor would have been willing to trade at the same price (as the HFT's limit order), or not. If not, then it doesn't matter how fast the HFT places the limit order; it's going to increase liquidity in any case. So the "high frequency" part is not necessary; the key is that the HFT is an investor willing to trade at a certain price.
But if some other investor would have been willing to trade at the same price, then the only difference the HFT makes is speed: the liquidity gets added sooner than it otherwise would have. So the "high frequency" part doesn't change anything except who the profits from the trade go to.
However, it's the category of sell-side traders in general that matters for liquidity.
You're using imprecise language here (for instance, referring to market makers as "investors"), but the clear implication is that you believe that normal, "buy-side" investors provide adequate liquidity. But they don't, and we know that, because the more competitive the sell-side gets, the lower spreads get.
Well, of course that depends on how you define "adequate". What is all this incessant trading of stocks for? What value does it provide? For example, if I'm an investor (a "real" one, not a market maker, to use less imprecise language) with a long time horizon, something like 30 years, because I'm saving for retirement, how does HFT make me better off?
In other words, brokerage fees, or the equivalent, should be lower, so the overhead to execute a trade is lower. That has some value, yes, but not a lot, because a buy-and-hold investor, of course, doesn't execute many trades, so the overhead cost of executing trades is already pretty small for him.
Also, most "buy-and-hold" investors hold mutual funds, not stocks, which changes things. See below.
> costs...can be very important even to value investors when there is some extrinsic prompt to trading
Yes, but as I just noted, most "buy-and-hold" investors are holding mutual funds, not stocks, so they aren't paying the direct costs of stock trading anyway. When I want to rebalance my 401k, I don't trade stocks, I trade mutual fund shares, and they're all shares of different funds offered by Fidelity or Vanguard or whoever my 401k provider is. (And with most 401k's, certainly with the ones I have, as long as you don't rebalance too often there is no fee for rebalancing.)
So decreasing the overhead cost of trading will only appear to me, if at all, as a decrease in mutual fund fees; but with most 401k's the individual doesn't see those anyway, because they're provided through employers. I don't know how much the fee question affects the negotiations between employers and mutual fund providers for setting up 401k's, but in any case that's at least two layers of indirection between me as a retirement investor and the overhead cost of executing individual stock trades.
So I can see some small benefit, yes, but is it enough to offset the high social cost of having so many smart people doing HFT instead of something more productive?
Whatever you pay to your brokerage, you pay on top of the spread. The spread is what you pay to place a market order.
In exchange for the privilege of buying right now, you pay the best offer price. In exchange for the privilege of selling right now, you pay the best bid price.
It's a commission you pay per share; the more shares you try to move, the more you pay whoever's providing you the liquidity.
I agree with you for the case of a "buy and hold" investor buying or selling actual shares of stock (with the proviso that the overhead to such an investor is already pretty low since he doesn't execute many trades, so the gain to him from something like HFT decreasing the bid-ask spread is small). But as I noted, many, probably most, "buy and hold" investors hold mutual funds through 401k's, not individual stocks.
For example, I have a 401k. I don't place market orders. I rebalance my portfolio every so often, which, after a lot of intermediate steps, might result in some mutual fund manager placing a market order. But I don't see any of the direct costs associated with that, whether it's paying the spread or anything else. I only see the net overhead of the mutual fund as a whole. (And, as I noted, for a lot of 401k's, like mine, I don't even see that, because the 401k mutual funds are no-load.)
Whether it is a direct line item on your prospectus or not is immaterial. No load mutual funds are more successful when they have lower execution costs. The biggest driver to low execution costs is an electronic market and then the bid/ask spread. HFT enable both of those low costs.
I see how HFT can narrow the bid-ask spread, but how does HFT enable an electronic market? Isn't it the other way around?
Imagine that there is a "true price" of some security at price x, and your bid and your offer are positioned around that true price at x - d (your bid) and x + d (your offer). So your spread is 2d. Then some external (public) event happens, which moves the true price by an amount greater than d. Let's say it is a big event, and the true price falls by 4d. Now there is a race. If you don't manage to cancel your bid in time, a predator will 'pick you off', take out your bid, and they'll sell to you at (x - d) when the true price is now (x - 4d). The predator has made an immediate (paper) profit of 3d, and you've made a loss of the same amount. If you're fast enough, you can successfully cancel your bid and repost it further down the book at x - 5d.
Now, as a market maker, a liquidity provider, you could increase your safety margin by quoting a wider spread, setting your bid and offer at +- 2d instead of +- 1d. Now it takes a larger move before you're in danger of being "picked off", but you'll also attract less customers and make less profits. Speed (latency) is directly correlated to how tight a spread a market maker can quote.
Is that enough of an advantage to outweigh the cost of the small number of intelligent people in HFT? I don't know, but it certainly seems easier to justify than the phenomenal amount of capital spent on internet ads. I'm a bit of a free market capitalist, so my biased response is how else should we allocate people's output?
I don't actually have a sense of how big or small the number is as a percentage of intelligent people. Particularly when you observe that, as you say, HFT is just a tiny part of the finance industry. There are many other parts of that industry that are also sucking up smart people to do things that, to me, have very little value compared to the other things those people could be doing.
> how else should we allocate people's output
Sorry to redirect again, but see my response to pas upthread. :-)
So it's hard to have the "it's terrible so many smart people are doing X" argument when our society has made it pretty obvious we value selling internet ads as high as any other possible commercial enterprise.
Me, I'll take the other position, which is that yes, the markets are bad at allocating capital, but they are better than our alternatives, and the silly little pocket change inefficiencies that go into speculative market making are a small price to pay for all the advantages of a modern liquid, price efficient market.
How are you getting that from what I said? You could s/HFT/internet ads/ in my post in response to pas and it would be pretty much the post I would have written about internet ads if that had been the subject under discussion. (Well, not really, because you'd have to change other things as well, but hopefully you see what I mean.)
> markets are bad at allocating capital, but they are better than our alternatives
This I agree with. But markets are as bad (or good) at allocating capital as the people who participate in them. In other words, markets reflect the values of the people who trade. I wasn't ranting about markets not being able to properly compensate curing cancer because markets are bad; I was ranting about people's values being so screwed up that the price signals they are sending into markets are making people rich for HFT and internet ads but not for curing cancer.
It's implied by what you didn't say. You don't require technologists involved in internet ads to justify their existence whenever a post about google comes up. Nor when news breaks of a new internet startup being acquired. I'm picking on you a little bit as a proxy for the entire HN community, as there seems to be a bias against HFT and towards internet ads that is a little unseemly.
Further, I'm not convinced you understand the scope of money involved in HFT vs cancer treatment for instance. When Roche bought genentech (a cancer research company) they paid 46 billion dollars for the privilege. Knight Capital Group, one of the biggest players in HFT is worth around 1.4 billion dollars. The market seems to be allocating resources correctly. The PR companies on the other hand...
Wow, you've gone through my entire HN corpus to verify that?
> I'm picking on you a little bit as a proxy for the entire HN community
I would prefer that you didn't do that, since I am not a good proxy for any community that thinks figuring out ways to get people to click on ads is a good use of smart people's time and energy. (Btw, I'm not sure that the entire HN community is such a community; a part of it may be.)
> The market seems to be allocating resources correctly
The numbers you give are interesting, but I'm not so much interested in money as in human resources. Genentech may have cost $46 billion, but are they paying smart people enough to get them to do cancer research instead of becoming quants?
Some of this may be a perception problem, as you imply by your comment on PR companies. What data I've seen on where smart people can get paid the most indicates that it's not cancer research. But I'm sure the data I've seen is incomplete; I would love to see more comprehensive data if there is any.
If you assume that the number of 'real' trader (not middlemen) who want to buy or sell is constant (not necessarily true, but assume it for a moment), then it stands to reason that the tighter the spread, the less the market makers are profiting. At a small enough spread, they make no profit, since the profit from the flow on both sides is cancelled out by the losses from adverse selection (trading when they couldn't cancel in time - being victim to a predator). But if they can increase their speed, and minimize their adverse selection losses, then they can quote tighter and still make a profit. And do it with machines, and cut overheads of hiring humans, and you can quote even tighter since you need less trading profit to make a net profit. So this is the value generated, automation + speed = the smallest amount of frictional costs being extracted from the market.
Ok, this makes sense. But it still leaves the question, how much are the frictional costs decreased by HFT, and is that benefit enough to offset the social costs?
Such as? All I heard was smart boys and girls go and do a quantitative finance degree instead of Solve The World's Problems degree.
And that's bullshit. The social cost of _not_ providing other, better alternatives to our youth (downsizing NASA, underfunding research and scientific endeavors), lack of patent reform, lack of bandwidth and unified cross-country 3G/4G has much worse costs.
No, they go and do a Solve The World's Problems degree and then end up in quantitative finance because they can't get rich solving the world's problems.
> The social cost of _not_ providing other, better alternatives to our youth (downsizing NASA, underfunding research and scientific endeavors), lack of patent reform, lack of bandwidth and unified cross-country 3G/4G has much worse costs
I think we're in violent agreement. This is exactly the social cost I was talking about. Our society is run by people who can't be bothered to compensate someone properly for curing cancer, but who will throw millions of dollars a year at someone for shaving a few microseconds off an HFT algorithm. That makes no sense to me.
Why are things this way? Lots of reasons, of course, but one of them is that, whenever anyone suggests that maybe it would be better if smart people were directed into more productive endeavors, the financial people have kittens and say it's going to wreck the world's economy if we don't shave those few microseconds off the HFT algorithm, because there won't be enough liquidity or something like that.
And that is bullshit. Sure, if we were at a point where all the big problems were solved, then yes, making tiny incremental improvements in the price of liquidity might be near the top of the priority list. But as things are, it should be somewhere around 56,732nd place. The problem to be solved is that there is huge, huge value to be captured by things like curing cancer, much, much more value than there is to be captured by shaving microseconds off HFT algorithms, but nobody knows how to capture it. How is making stock trading a fraction of a percent more efficient going to fix that?
This (false) assumption is propagated regularly by articles and books wherein the same pattern repeats itself:
1. limit orders are defined
2. "... but on that morning Joe Trader was shocked to see that his limit orders weren't working!!"
3. description of Joe Trader NOT using limit orders ... or using them and canceling them and chasing prices ... or whatever.
I've seen this pattern in a lot of media since the Lewis book. In reality, limit orders haven't changed and you can set a limit and go to sleep / go on vacation, just like you always could.
Because NOW is when we have the most actionable knowledge.
If you don't buy now, what if the price never goes back down? You missed a buying opportunity.
At any given point in time, the "true" (buy-side) investors are likely to be relatively far apart on prices. This is especially true in thinly-traded markets. For example: consider housing, which is the canonical example of an illiquid market.
I know the value of my house (it's around $300k). Say I want to sell it. The more liquidity I need, the worse price I'm going to get. Anyone with any intuition for the housing market knows that if I have to sell my house tomorrow, I am going to get a god-awful price for it; the "tomorrow" price for my house is many tens of thousands of dollars off its true value.
This creates huge problems for homeowners. The obvious standard advice for sellers is to wait patiently for the best price, counting on many weeks or months before a reasonable offer arrives. But I have to pay money every month I hold on to the house. If I need to move immediately, for instance for a job, or because my financial circumstances have changed, I might need the house to sell quickly, and because houses are illiquid I have to accept a crappy price. Or consider trends in the market: by forcing me to hold the house for months rather than days, I'm maximally exposed to swings in the market. So if Chicago housing prices crater while I'm trying to sell, the extra time it takes to unload the house takes that price swing out of my hide.
(You can easily see how the same thing happens in reverse in "hot" markets like Palo Alto, with the buyer now assuming the role of the hapless Chicago seller).
The exact same thing happens in the public markets; it's just not as intuitively obvious because we're working in smaller deltas of time and price.
But the public markets have a huge advantage that the real estate market doesn't: market makers.
Imagine if houses traded (were bought and sold) so frequently that a smart company could make good educated guesses about the current value of any given house. Imagine if that company would on any given day snap up a house offered for sale. That company wouldn't pay my asking price for my Chicago house, but it would pay something much closer to it than the "true" market would. After buying my house, the company would immediately offer my house for sale at any price greater than what it bought the house for. I wouldn't care, though: either I'd be thrilled for the opportunity to sell my house quickly and painlessly, or I just wouldn't accept their offer and instead do what I do now, which is to wait for the best offer.
Assuming the housing market makers were smart, the money they'd get by forcing me to accept a lower bid would be "free money" they get simply by being smart about valuing houses and having capital available to deploy. We'd all be a little irritated at them for scalping distressed buyers and sellers. But something else will inevitably happen: other smart firms will smell the free money, and they will compete for it. To capture a share of the premiums, all they have to do is offer a slightly more favorable price, so that's what they'll do. Over time, the money the housing market makers will get less and less "free", and the price penalty for immediate liquidity will get lower and lower.
In my fantasy real estate market, I now have the opportunity to avail myself of a reasonable "market" order, or to buy or sell on a "limit" price. I could do that if there were housing market makers, but in their absence I can't, because the price hit for selling tomorrow is too great; it might be a 25% discount on my asking price, or even something close to 50%. That's the cost of illiquidity, or, stated more directly, the value of liquidity.
It's probably also worth saying that liquidity is a presumption of the public electronic markets. Because it's taken for granted, there are whole trading strategies (hedging, for instance) that depend on its presence. These trading strategies face execution risk: if they can't bail out of a position within a specific window of time, they incur losses for the trader.
(It is because of comments like these that I have your comments page bookmarked the same way I would a blog.)
In particular: you can come away from Lewis' book believing that the markets prior to electronic trading were reasonable fair, when the reality is the opposite: the trendline of fairness and transparency in the market is sharply positive, and the inflection point of that trendline is the advent of fast electronic trading.
For those like me who don't understand, what does it mean? What are the benefits? What are the drawbacks?
Basically, with more participants, the bid-offer spread will begin to close, and trading will occur at more prices.
A very illiquid market, I want to buy for $9, somebody else wants to sell for $10, nothing else. And now? How does HFT help? How does a traditional market maker help? He buys for $10 and sells for $9 and then somehow recovers the loss?
(Further down tptacek posted a great analogy. https://news.ycombinator.com/item?id=7853500)
So you and the seller see better offers than if they didn't exist, even if those offers aren't good enough to get you to trade.
People are more likely to trade if they feel a lower risk of being stuck in a 'bad' position.
Many times, exchanges actually pay market makers to provide liquidity to try and kickstart a particular market.
The more liquid something is, the closer it is to cash.
Think about your car -- if there was a public price for your car and you could decide at any moment whether you wanted to sell for that price, or close to that price, that would be a benefit to you. But instead selling your car is quite a chore and you are never sure whether you are getting ripped off or not. A liquid market has a public price and a degree of fairness and confidence baked into that price.
In order to provide liquidity market makers agree to buy stocks from anyone who will sell them, and then take care of finding the buyer on the other end. The difference between what they buy it for and what they sell it for is called the 'spread' and it is how market makers traditionally have made money. The spread is also the fair fee they charge for the service they provide, which is liquidity, and the risk they take. If market makers did not know for sure that someone else would be willing to buy the stock right away, that they might have to sit on it, they would be taking on a very large risk. The more people that buy and sell, the less real that risk becomes.
HFT makes their money, generally, by front running. Front running is the practice of buying a stock because you know the California Pension Fund is about to buy it, and so you know the price is about to rise. You can immediately turn around and sell it when it goes up.
Front running by a bank is illegal, but it is widely suspected that many banks engaged in the practice for years. It is illegal because the bank is supposed to be acting on their client's behalf and in their interest, so when they are trusted to place the order, they first buy some of the stock for themselves or for the bank's inside funds. It is basically insider trading.
When HFT engages in front running, they have detected that an order is coming down the pike, and they jump in front at the last nanosecond. The reason they spend so much money trying to do this is because it is essentially fool proof when done correctly. You can argue that by not taking any market risk, and not holding the stocks, they aren't providing liquidity, they are parasites on real stock traders and they are essentially using inside information. But its not that cut and dried because there is almost no way to eliminate the problem. At least HFT reduce the time window during which someone's signal to purchase something can be exploited.
It used to be that humans managed all of this. Large brokerage houses would be responsible for executing these trades, and it was common practice for brokers to tip each other off when they received large orders. These tips were very valuable because they guaranteed a winning trade. If you ever wondered how meat-head frat boys could make millions of dollars on Wall Street despite not being especially intelligent, this was a big part of it. At least now the people who are scalping everyone's orders have to be brilliant.
This is not possible.
For example, you want to buy 10,000 shares of X.
Matching engine finds 100 at 9.98 - it’s the NBBO (best buy offer), it’s required to execute that, and carry on looking for ways to fulfill your order below or at your bid.
Meanwhile, the HFT offering those 100 shares sees this, decides its a parcel of a large buy, and if it has a quicker route / algo than the matching engine to other exchanges, goes there, executes immediate buys at say, 9.99, then immediately raises their price to make a profit on the spread.
What is actually happening is that the HFT is offering shares on all the exchanges on both sides of the bid/ask spread at the same time. When they get filled on an order at a price on 1 exchange it is used as one of many signals to indicate if their pricing is accurate. If their algorithm thinks that fill indicates a price movement (for instance if the entire level was removed) then they will update their prices on all the other exchanges accordingly. If the algorithm doesn't believe this indicates a long term price movement they won't (or they may even re-up at the same price on the exchange they got filled on).
No one is going out and buying ahead of someone else because that is too expensive (you have to pay the bid/ask spread to do it) and too risky (you might be wrong about your price determination).
Finally, just a detail, you can typically opt out of NBBO matching if you want.
Instead of competing to save you a few more slivers of a cent per share, they compete to fill your order a few microseconds faster (at the next-highest price permitted). Seems most traders would prefer the money.
Furthermore to me HFT looks like stealing because it is very inefficient relative to the need. If exchanges were to reorganize how they work slightly, we would get much better price stability, somewhat less liquidity, and have a much lower premium.
The necessary change is simply this. Exchanges would have to set a price at which orders are to execute. Whenever there are excess orders in one direction or the other, that price would drift up or down, slowly, until the order executes. The result is that trades might take a minute to execute, but margins would be far less. Sure, a ton of finance people would jump up and down screaming bloody murder about how many of their complex trades would be affected. But my opinion is that that economic activity takes a lot of brains and resources, and doesn't significantly add to the purported purpose of finance - to make money available to companies engaged in useful production.
How would this even work? Understanding Market micro structure is literally my day job and I'm having trouble parsing your proposal.
Consider the market where I want to buy at a limit of 9.90 and someone else wants to sell down to a limit of 10.00.
What price would the exchange set? People have already said the maximum the market will buy for is 9.9 and the minimum the market will sell for is 10.
if there is no overlap you just cant' trade no matter what the exchange does.
How does HFT change that?
Now the spread is 8 cents. Anyone who wanted to buy can now buy it at a penny cheaper than an instant ago and the opposite holds true for anyone wanting to sell.
Today the price is the price at which the last transaction happened.
I am proposing instead that the price is set and updated by the exchange, and transactions only happen when they can execute at the exchange price. And the exchange price moves at a preset rate - down if there are outstanding sell orders, up if there are outstanding buy orders, else still.
So in your situation the price would wind up somewhere between 9.9 and 10, then stay indefinitely until another order began moving the price.
Basically for someone placing an order you're losing the ability to trade instantly, and gaining the prospect of trading at a possibly better price. Therefore you don't need a pool of people trying to give you instant trades.
Isn't this only true if you measure price stability based on amount of changes in the price, but not magnitude of those changes? How can they lower the spread but simultaneously decrease stability?
See the flash crash for the best known example.
Paper on frequent batch auctions: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
"There are some subtleties involved [...] since it takes light roughly 4 milliseconds to travel between Chicago (where ES trades) and New York (where SPY trades), [...] We would also like to suggest that the fact that special relativity plays a role in these calculations is support for frequent batch auctions." (emph added)
If it is FIFO then you still have a latency race.
I always see this argument. Market makers provide liquidity, and yes, HFT often act as market makers. However, I am not at all convinced that the high frequency element adds anything useful to the equation. I don't care whether my trade executes in a microsecond or a second. In fact, I often leave limit orders open for DAYS in order to get the price I want. I am also not convinced that most HFT are truly acting as market makers.
I think the most charitable thing that can be said about HFT is that it narrows the spread. In order to make it worth their while, the HF traders then have to make a lot more trades. Whether this is adding any real value to the system is harder to say because while the spreads are smaller, the volatility is higher.
HBS is graduating lots of portfolio managers and analysts who contribute to this phenomenon. Not so many HBS grads in HFT, based on the folks I know in the industry.
HFTs certainly aren't driving major global corporations into the ground by pushing for stupid forms of management like share buybacks, "outsourcing" core competencies, and the like.
I give this guy credit for referencing these impacts. I don't think HFT is really part of that trend at all. HFT is actually pretty benign when you dig into the details. Sure, they scalp pennies here and there, and it adds up. But the competitive pressures on technology force efficiencies to a point, which is good for the market.
Maybe IBM will never go out of business even after creating little or no value. But instead it will find methods outside of creating value in order to create wealth. Much like high frequency trading.
From what I gathered from Lewis's book, the crux of the problem is that people were doing their trades in multiple exchanges and that HFTs were simply noting trades in the fastest exchanges to reach (like BATS) and using that information in exchanges that were farther away.
To take his analogy, it would be like using the results published in another newspaper the day before to determine your ballot choice.
The only real purpose of financial markets is risk management. Any human society (and non-human societies also) has to risk some capital to gain more capital/value. The purpose of stocks and financial markets in general is to diversify the risk, while still funding oportunities and it's the only proven mechanism by which a large portion of the population can participate in this wealth creation.
The value of risk management is hard to fathom. But if you are a programmer, you might want to try and simulate "gambler's ruin" and see what happens with high risk or low risk strategies. In this simple game, computing the right risk management is the difference between stagnation and exponential growth. In a complex economy, this seems also to be true.
High-frequency traders are somewhat similar, in that they don't care enough about the long-term health of the companies they buy and sell. Therefore they are bad.
The reason speculation based on short-term results is problematic is that it creates perverse incentive structures that discourage a company's management from thinking too much about the long-term success of the company. But that's only possible because speculators are able to create market demands that punish and reward specific kinds of behavior. But HFT doesn't really create demand; it only responds to it. So the analogy fits in every spot except the only one that matters for the sake of the argument.
Without 'Finance', you would not have: - Cheap mortgage rates - mainly due to securitization. - Cheap loan rates - yes, car leasers, I'm looking at you.... - Rational commodity markets. Think Starbucks, every car/computer made today. - You get the idea.
Certainly, some aspects of modern markets appear to add little value, but the bulk of 'finance' benefits not only companies but everyday consumers immensely. -
HFT versions of those market participants are dramatically more efficient and fair than the participants they are replacing. That efficiency is shared with every other participant.
If you believe in public commodities and equities markets, it is extremely hard to argue against HFT and in favor of pit traders...
That particular team had been highly profitable, but their last quarterly result were poor. It mattered not that overall they were still vastly ahead in terms of profit for the company. They had a bad quarter and the next was looking poor, so they had to go, in order for the managers to have something to say to their investors.
If the share price dropped a few cents or if the results weren't up to or exceeding market expectations, they would fire a few thousand people. I kid you not, this is how investment bank culture works. You are only as good as your last quarter.
Traders would take very risky positions with huge or unknown long term risks, in order to make short term gains - for which they are richly rewarded. No one cared that this wonderful basket of exotic options was actually full of potential toxic waste, as long as it was printing money right now. By the time it exploded, most of those traders and managers would have made their millions and be retired or working somewhere else.
It was and probably still is this intense focus on the short term that caused ML to very nearly go under and why ML is now called Bank of America ML. I doubt they have learned and the next black swan event is going to take them out, or the one after that.
Another possibility the author should consider: there are some CEOs who have the ability to convince human investors that their long-term visions are worthy and deserving of steadfast support.
What the author fails to address is the fact that not every chief executive has a great long-term vision, and not every chief executive is capable of persuading investors that he or she is capable of realizing such a vision. The implied notion that investors are somehow obligated to give the companies they invest in an arbitrarily long period of time in which to execute regardless of evidence of tangible progress is foolish.
http://www.ft.com/intl/cms/s/0/b1a73be4-a57b-11e1-a77b-00144...
http://www.oslobors.no/ob_eng/Oslo-Boers/About-us/Press-room...
Saying "HFT is bad, just look at electronic front-running" is like saying "Sorting algorithms are slow, just look at bubble sort".
"Sophisticated" trading strategies exist because the finance industry has focused itself on one goal to the exclusion of all others: Make as much money as possible. It doesn't matter if it's a zero sum game or if the most fit companies are being selected based on research in a darwinian process. A rational actor is going to chose 30% over 10% every time.
The bottom line is that predicting the success of a given company is hard. Predicting what the other actors in the market are going to do is also difficult, but is much easier by comparison. Asa participant in the market you want to be in Keynes' newspaper contest, not in the business of making broad predictions about the future. As a member of society we'd prefer that capital be allocated to the firms that will use it to our most benefit. I don't think anyone wants to use the law to change the current system so that it is more useful to society. A lot of people are also unhappy with a lot of finance being in its own bubble of "irrelevance". This is a decision that we're all going to have to make about how we want the financial markets to work. Things seem to be working out ok now, in a general sense, so change might be a long time coming.
I don't know if this is actually true if you average up all the successes and failures. Probably more like, for every successful [daytrader|money manager|HFT firm] there are 9 that lose money or go bankrupt and nobody ever hears about again or includes in average return calculations.
It's just that the rewards to the few successes are so tantalizing, there's never any shortage of folks jumping into the game trying their hand at it.
Not sure I'd agree. With companies you can at least throw some old school research and hard work at the problem. Difficult but at least nobody is out to eat your lunch there. Its just difficult as opposed to cut-throat.
>rational actor
Can you really count on that if 60% of the market is AI and a sizable chunk of the rest is trading on "superior"/inside info?
A) He implies (but never proves) that HFT market makers use low latency connections, to buy shares ahead of other market participants, to sell back to them risk free. This is not how HFT market making works. HFT market makers are pricing on all the exchanges at the same time. So what they are doing is not buying something and selling it back to you risk free, but changing the price of their own offering to reflect new demand.
B) That any of this is secret or requires insider information. It is all available on public websites, including governmental agency ones (for instance all "exotic" order types go through a public approval process). Further, in the book the only proof of ill gotten proprietary information being used for profit was from the supposed "heroes" of the story.
C) That HFT is a large force that abuses it's power to take advantage of buy side participants who have "Main streets" best interest at heart. In fact, HFT firms are the small guys bringing efficiency at the cost of the powerful entrenched buy side middle men.
Our society is in a state of secessionism. Much of the escalating economic inequality comes from that impulse. The rich of yesterday (1945-73) saw themselves as leaders of the society. The rich of the new Gilded Age (1974-2014+) have given up and just want to escape it. They want private schools, country clubs, and closed social networks. They don't want to lead the masses, they want to leave them.
Silicon Valley has been making its secessionist impulse visible of late, but HFT and "high finance" show a different secessionist tendency: the desire to get outside of any given industry or company and "float among" them as a financier. The brightest young people are being told not to join the regular economy with the proles, but to become part of an elite system of hedge funds, private equity shops, and overcapitalized "startups" whose products are meaningless other than advertising expenses to get middle-management positions, 10 years earlier than otherwise, for the founders.
The smartest people have given up, sadly, on leading. This makes "leave" the attractive option. If you become a management consultant or investment banker, you don't have to commit to one industry or business. You float around until you make friends who can place you at high levels.
After spending 8 years in a number of places but much of it in "the real economy" I can't say that I blame the leavers. The leavers are now ahead of me, career wise, as venture capitalists and the like; and there isn't much out here in "the rest of the economy" to lead. Perhaps paradoxically, the world ends up being run by leavers, because average people don't want to be led by the highest level of talent; they want leaders they can relate to.