This Is What One Half Second of High Speed Trading Looks Like
blogs.smithsonianmag.com
blogs.smithsonianmag.com
It made a great conversation piece with my coworkers. Non-technical folks (most of them) were astounded that so much text was being interpretted and executed every single time they loaded the homepage (in less than a second).
What was the point? It helped set a certain emotional tone to our conversations. Folks found it a lot harder to demand their project be done "right now, it's easy" when directly confronted with the complexity behind a single page load.
Likewise, stories that highlight the immense speed and complexity of high frequency trading help set an emotional tone of alienation and fear. It helps trigger the same reflex Frankenstein and The Terminator played off of: "maybe we're unleashing technological forces that no one can control."
But of course if you are well and truly versed in a technology, that feeling goes away. I knew that our crappy brochure website was not very complicated as websites go.
Into a text editor, I pasted the HTML source of a rendered page as presented to the end user, then went back and replaced every call to an external js or css with the actual contents of those files. Then at the top, above the doctype declaration, I pasted in the ASP bootstrap code that ran on every page load. I did not include any representation of the compiled code behind the site, like SQL Server, IIS, etc.
If were going to do this now, I would definitely include js libraries like jQuery. The point for me was to illustrate just how complex websites are, not to take personal credit. If you wanted to illustrate it more precisely, maybe you could use different font colors for your code vs. library code.
It was a huge, big, complex, ugly thing. We laughed. Good times.
There are a lot of valid criticisms you can make about modern financial markets, but claiming that we are using computers to do things better than we did 10 years ago is a bad thing isn't something I'd expect to see here on HN.
But they are not using those computer to do something better, they are using those computers to do something that was impossible 10 years ago.
The problem I have with HST is what purpose does it serve, other than offer another way to gamble on stock price movements.
Now there is nothing new about gambling on stock price movements but with HST it takes the control out of the market and puts it into the hands of a bunch of computer programs.
And to say that can't cause problems, you have to look no further than the hoax bomb at the white house tweet that wiped out $130 billion in market value in the matter of seconds.
Without those HST computers system all trying to out do each other, that event would not have been so dramatic.
[1] http://21stcenturywire.com/2013/04/24/white-house-attacked-o...
Hardly what you call progress.
Or maybe it's just another zero-sum scheme for moving money from the rich to the very rich. Heck I dunno.
Real investment decisions - as in, decisions of which factories to build, which products to develop, who to hire, and so on - are made on a time-scale of weeks or months. Having the stock market react on a sub-second time scale certainly has no effect at all on those decisions.
To gain some perspective, consider if trading operated on an hourly-auction type system. The stock price would react slower and in a different manner that it does today, and that could change the income distribution among traders. But honestly, why should anyone care?
When it comes down to it, there is no upside to trades going that fast to anyone outside of the myopic circle of professional traders. Even those professional traders only have an upside as long as they participate in a vicious arms race. However, there are downsides.
One downside that many people argue is that the resulting distribution of income is not just, because HST firms suck a revenue stream out of the economy without building anything real.
The more important downside (in my opinion) is that HST sucks intelligent people away from sectors of the economy where they would positively contribute by developing and improving directly useful technology.
Completely off-topic to the original discussion, but assuming that you intended to imply the quantifier that I think you implicitly intended to imply - i.e., "all people", or even "most people" - you must be living in a completely different world and/or a very cushy bubble. [1]
I expect many people on this site are lucky enough that they can work on whatever they want - either because their interests happen to align with what's currently in demand by the market (as in my case) or because they were lucky enough to be born to rich parents or win the startup lottery.
But take a step back and look outside of that world. Consider, for example, the many people working as cashiers or cleaners. Do you really think that all or even most of them work those jobs because that's what they genuinely want to do, given a choice? It seems much more likely that the main reason why most of them work in those jobs is that they have to make ends meet somehow, and those are the kinds of jobs available to them.
As for the more on-topic part...
Why do you assume that those intelligent people would not engage in some other little value add activity that can make them a lot of money.
I don't. There may be other talent-wasting sectors in the economy (and I guess you always get some geniuses spending their time on whatever the equivalent of card counting of the day is). I have simply pointed out that HFT is one of them (and I do believe that finance in general is the worst offender at this time).
[1] If you intended to imply the quantifier "some people", I apologize for my misinterpretation - though in that case, I would wonder what the point of the statement was. Certainly it doesn't help paint capitalism in a positive light, because every economic system allows some people to work on whatever they want, while the rest tend to have the same superficial choice of job that they do in capitalism. (E.g., you were not simply assigned a job in communism either.)
Do web apps provide a net positive to society?
I'm (I would have thought quite obviously) wondering aloud if everything we do needs to provide a net positive to society.
Why is that the only or most important question? How do you even define "a net positive to society"? Are we talking about utilitarianism, and if so, are we trying to maximize the total utility, or the average utility, or perhaps the minimum individual utility? Does my eating a candy bar provide a net positive to society? And even if something can somehow be shown to not provide a net positive to society by some metric, should it therefore be prohibited by the threat of violence (i.e. made illegal)?
So HFT should expect to be manipulated by false news.
I bet they won't be to happy when that happens.
I suspect they don't give too much of a toss.
Firstly, the people who run/own those markets make money on every trade (i.e. ever buy/sell) so they'll make money if the market is going up or down.
Secondly those manipulating the market will be in the know so they'll be in a position to take money away from those people/organisations that are on the outside.
You only have to go back to 2007 to the GFC to see how easy it is for impotent governments and greedy bankers to screw the world.
Do you think any of those corrupt GFC bankers are suffering now?
* Acting on public information, milliseconds before anybody else - which helps everybody else about as much as insider trading does.
* Acting on trading information gleaned from "bid stuffing" (making and canceling orders really quickly) - which helps everybody else about as much as front running does.
* Detecting the presence of large trades about to go through or going through and capitalizing on the price movements they cause - i.e. imposing a tax others making large trades in the stock market.
In return, they flood the market with liquidity when it is least needed and withdraw it suddenly when it is (e.g. when prices are diving).
There's literally nothing to like about HFT.
"They can do it faster, therefore it must be illegal."
Acting on trading information gleaned from "bid stuffing" (making and canceling orders really quickly) - which helps everybody else about as much as front running does.
The only people that notice this behavior are...other people capable of doing this behavior. Why should it matter to you, the long-term investor, the trading tactics computers employ at the millisecond scale?
Detecting the presence of large trades about to go through or going through and capitalizing on the price movements they cause - i.e. imposing a tax others making large trades in the stock market
This happens in any market at any timescale. Everyone capitalizes on trends.
question is, do they benefit the market as a whole? Traders do not have a right to perform HFT, and if it destabilise markets, we can take steps to reduce it (of course, there is then the issue of how to take steps without causing different problems)
Benefits of having HFT traders, is that at the cost of having them around you can have fair and level ground. So if you come to one exchange to do a transaction, you don't have to think much about exchange intrinsics, like liquidity available there, fee structure, etc. Because these differences are being arbitraged away by HFT traders.
And again, what is the alternative? Exchanges that are connected by human traders making phone calls?
In the case of stocks, for example, there has to be one ultimately authoritative copy of who owns which stocks anyway. So why not just do all the trading in the place where the authoritative copy is stored?
That would certainly seem to be more efficient from the perspective of minimizing the overall social cost.
And what if trading on that exchange will stop for a couple of days for technical reasons? What kind of problems can that create to stock of large company, that can be traded only on that exchange? Or what, if the whole exchange will go out of business? Or decide to charge arbitrary large trading fees?
Competition and diversity is generally a good thing. Tend to increase efficiency and minimize overall costs. That's valid for exchanges as well.
Also, what is the social cost you are referring to? There are a lot of people here saying that HFT doesn't provide "social value". Does it somehow provide less social value than traders screaming at each other in the pits? Similar types of trading always existed, except now it's done by computers. For what it's worth, HFT provides well-paying jobs for a lot developers and ops people and often is very supportive of the engineering community.
Given this notion, competition is a good reason for the existence of multiple exchanges, because it means companies can change which exchanges their shares are traded over. However, it is not a good justification for the fact that trading of the same stock happens on multiple exchanges.
As for the social cost, you do realize that HFT does not come for free to society. At a minimum, society must somehow pay for those well-paying jobs to developers. And what do those developers give back to society? Before you answer, please consider whether you should distinguish between algorithmic trading and HFT. I do believe that algorithmic trading is useful to society (because it can do the job that screaming traders used to do better and cheaper), but the subset of HFT is not useful.
The only things anyone ever seems to be able to answer for HFT two-fold: one, that stock prices change faster, and two, that the spread is smaller. But one point one, nobody in the real economy cares about that, and to point two, the decrease in spread also doesn't matter to anybody in the real economy, because what you really care about there is the fluctuation of the share price over a larger timescale such as one full day. I have not seen any evidence that this fluctuation is affected by HFT in any way.
I don't think there's any reason to think that exchanges' main purpose for existence is to provide service to publicly traded companies. Exchanges are just a store, and their purpose is to buy/sell stuff and make a profit. So, in that sense, having multiple exchanges selling the same stock is the same as having multiple bike stores selling the same bike...it's competition and there aren't many downsides to it.
As for the social cost, you do realize that HFT does not come for free to society. At a minimum, society must somehow pay for those well-paying jobs to developers. And what do those developers give back to society? Before you answer, please consider whether you should distinguish between algorithmic trading and HFT. I do believe that algorithmic trading is useful to society (because it can do the job that screaming traders used to do better and cheaper), but the subset of HFT is not useful.
Nothing comes for free to society. Funding a never-ending number of failing start-ups doesn't come free to society either. What is that giving back to society? Your implication is that HFT trading is somehow taking something from society that it doesn't deserve. What is it taking? Also, in terms of making distinctions: HFT is the superset and AT is the subset, not the other way around. All AT shops are HFT shops, but there are actually quite a few HFT shops that don't run algorithms and instead run based on inputs controlled by traders.
The only things anyone ever seems to be able to answer for HFT two-fold: one, that stock prices change faster, and two, that the spread is smaller. This is the crux of the whole thing. Why does anyone have to answer for HFT? This is implying that HFT is hurting people however you haven't not given any proof of this (besides that it doesn't add value). If anything, I think HFT's biggest crime (as someone else has mentioned) is that sucks some really smart people into working on problems of a rather limited scope.
Because it allows the exchanges to compete on features (latency, order types) and pricing (transaction fees, connectivity costs).
In the case of stocks, for example, there has to be one ultimately authoritative copy of who owns which stocks anyway. So why not just do all the trading in the place where the authoritative copy is stored?
For equities, trading and settlement are two separate steps. Trading occurs on the exchanges and, for U.S. equities, settlement is handled by the DTCC, the central counterparty for all U.S. equities trading. Handling all the specifics and details of what happens after a trade is executed is a massive industry.
For some products, trading does occur only on one exchange. Typically in futures trading, an exchange will develop a product (a contract) and that contract can only be traded on that exchange which also handles settlement.
This means that those who can respond faster can get into a price level sooner. Breaking this out to minute bounds doesn't mitigate their advantage in any way shape or form.
It is hard to know what the results of a change like this would be. I suspect it would make the markets much less stable as there would be less information to make accurate prices, causing huge variances on minute long ticks.
1987 was caused by portfolio insurance from major funds, not HFT (which didn't really exist then as it does now). HFT also didn't cause the mortgage crisis. That was long term consumer financing & Gov mortgage buyers & pension investors. HFT was definitely involved in the flash crash, but what were the long term implications of that? Zero. The most recent blip was caused my dissemination of bad news via a source that had been compromised. Again, not HFT.
Want to hate HFT (I do!)? Hate it because it isn't profitable (which is why I left in 2010) and because it still sucks up engineering talent like a black hole of despair.
These anti-HFT articles that occasionally come out are hilarious because the money that put HFT on the public radar simply isn't there anymore. Look at GETCO.
The question I have is why is the public so willing to accept disinformation? Residual banker hate?
Large trades move the market, full stop. That's a feature, not a bug. Moving a lot of size changes the market's estimate of the value of whatever you're moving. Splitting up large orders into smaller pieces is attempting to hide that information, why should that be a privileged operation?
If HFTs push the markets higher slightly faster because of new public info then while it might be bad for you if you are buying on that news, it's great for whoever already owns the stock and is the one selling to you.
The same applies to your 3rd point. It might be a tax on whoever initiated the large trade, but it's a benefit to whoever is on the other side of the trade. HFTs put new information into the market much faster than humans ever could making them more efficient. They're also doing it at a lower cost than human market makers ever did.
If anything, HFTs have made it cheaper to trade. Because of the way exchanges are setup, the man in the middle will always be making a cut of the trades. Now though that cut is measured in pennies rather than what used to be often 1/2 dollar spreads.
Sure some of the predatory algorithmic trading serves no benefit to anyone, but you will always have people gaming/attacking the system if there's arbitrage potential.
When markets went over to computerised trading, it was argued that the human element of open-outcry trading was vital in preserving a culture of honesty and integrity.
Critiquing HFT is myopic, because it's based on a meaningless definition of "high frequency". HFTs are doing what traders have always done. There's a legitimate argument against speculation, but it's pointless to single out speculators that deal in the very short term. Singling out the newest technology is just a cheap shot, there's no fundamental difference between trading on a millisecond horizon or a minute or hour horizon.
One trader can act faster than another, and a millisecond(s) can be shaved of with the right location of your server (co-located near the exchange).
I'm undecided about HFT's ability to provide liquidity in the market. There are certainly some drawbacks (e.g. you have to have high capitalization to afford HFT colocation), but there might be some tangible long-term benefits as well.
But I'm more concerned about how much of a waste it all is. Jeff Hammerbacher famously quipped that "the best minds of my generation are thinking about how to make people click ads," when referring to developers pounding at the doors of google's and facebook's recruiters. HFT is similarly unfortunate. Somewhat useful maybe, but overall a depressing use of brilliant minds.
As someone who works in HFT, you have hit the nail on the head about the real problem with it. It isn't front running, or instability, or unfair trading or any of that. It is simply that so many very smart people are working on it, instead of other more useful ventures. Of course, I feel the same way about data mining social networks.
It seemed fairly clear to me that the parent wasn't saying that at all. What he was pointing out was that discussion of, and concern about, faster access to information, decision making and execution causing volatility in the market isn't new. That's not a straw man at all.
Also, as an aside, it's worth nothing that Hammerbacher only said that after making millions working at Facebook. I'm pretty sure that makes him an asshole.
* - For fairness, you could execute trades within a generation in a pseudorandom order determined by a random key, which is cryptographically committed to prior to the opening of trading, then reveal the key after the market closes to prove that you executed them in the correct order.
But you're incented to do the same thing! So if you're going to put in an order for 200 maybe I should put in an order for 400? You can see where this is going...
People worry about HFT destabilizing the market. It seems to me that would be a much greater risk if you're incentivizing people to put in crazy orders that they don't actually want to do just to grab the percentage of demand that they desire.
This seems to disincentivize putting up for sale more than the seller wants to sell, and to also disincentivize "Sybil attacks" where the seller has an incentive to create false identities for himself. (Just a thought experiment, would be happy to know what I'm missing here :-)
1) Cool use of infographic :)
2) The little dots represent price changes, not trades.
FWIW, some more context around this would be-
All the major exchanges stream their best prices to all the other exchanges.
This is required because US exchanges must execute trades at the National Best Bid or Offer, or NBBO [1].
So how do you track the NBBO. In theory, by having all the exchanges send their best bids and offers to a central Securities Information Processor, or SIP (the 12:00 box) which then disseminates them (which you can see by the waves of stuff it sends out)
In practice, it's slow to go through the SIP on every price movement and the exchanges want to have the best possible information, so the exchanges mostly stream prices directly from the other exchanges (hence the fanout effect on every price movement)
Most of these prices movements don't result in trades, they just represent somebody entering or leaving the market at the BBO.
This feels like a distinction that could have been made better, rather than just saying "yeah, this stuff's complicated."
[1] http://en.wikipedia.org/wiki/Regulation_NMS
edit: formatting / citing
I'm still having a very hard time understanding what kinds of events would cause so many rapid price changes. So many dots in just half a second -- even when time-stretched to 5 minutes! Are those price changes meaningful at all?
The negativity seems to be a result of previous bias.
HFTs are not publishing a lot of useful results in AI or game theory. They are in a zero-sum game against each other as well as all legitimate investors, and all their profits come at the expense of people trying to better route society's resources.
Also, exchanges will revert back trades that happen over these flash crashes.
Also the incredible focus on these two events, considering all the other turmoil in the financial markets, has always struck me as a clear case of scapegoating with Luddite underpinnings. Pretty soon, we will all believe that the recession was caused by HFT's.
... or is that just an insane non-starter, since it affects too many rich people in the finance sector and their colleagues rotating through the current regulatory structure?
I think it's an interesting long-term problem faced by society to decide whether we are OK with investment markets becoming completely impossible for "normal", human investors to navigate. It's already rapidly going that way--the capability to profit is already wildly tilted toward firms that act at HFT volume. If a poker site became overrun by bots to the extent that it was no longer fun for humans to play, they would fight back against the bots (as most do to some degree). But with investments... we may be getting to the point where the average day trader needs a 3000U server farm to compete effectively, and then I question whether we've lost sight of why we subsidize and spend money regulating that market at all. It's like a chess league where the grandmasters get to use computers as aids, and the newcomers have to play blindfolded.
[1] Nothing catastrophic has happened yet, but there have been a few flash crashes that have necessitated trading to be shut down, and having to "roll back" trades is never perfectly equitable and only decreases confidence in a market.
[2] Some of those flash crashes were traced back to one overzealous seller that threw HFT's for a loop (read http://en.wikipedia.org/wiki/2010_Flash_Crash -- it is quite interesting). I shudder to think of a day when the market fluctuates wildly and the data is literally too complicated for regulators to trace it back to any identifiable cause. That day is closer than anybody in finance is likely to acknowledge.
I think HFT is ridiculous and the epitome of capitalism. If you outlaw'd it, a new form of lightning fast arbitrage would arrive.
My point was that if you call apple increasing over the course of 1 year, you might double your money if you're lucky. It's much more likely that the HFT operators will outperform you, as they have over the past decade.
Why do they need to? Winning isn't getting more trades in, winning is making money!
According to the Times[1], HFT firms' profit topped out at less than $5B in 2009. That is, not to put too fine a point on it, peanuts.
[1] http://www.nytimes.com/2012/10/15/business/with-profits-drop...
These results don't seem related at all. You buy stock at price A and sell and price B (B = A+X, hopefully). What does it matter to you that other parties made money (perhaps more than X) while the stock was going to B? Institutional investors have always been more savvy about the market than retail ones. I don't see how them making money around the edges changes the calculus for "the rest of us."
Yes. That's actually a bit of a problem. It is perfectly all right that day traders are being out-competed by HFT firms, by itself that's fine. Humans replaced by computer, nothing bad about that. What is not good, is that a successful entry into arbitrage market nowadays require a lot of infrastructure and technical skills.
That increased barrier reduces a number of players on the market. That reduces diversity. And that increases risk. That's bad.
A simple example how that increases risk. Imagine that a single HFT firm had managed to out-compete all others. And now owns perfect infrastructure with nanosecond level latencies, speed-of-light channels between the exchanges and trading algos without any bugs and teams of quants maintaining them. Obviously no day trader will be able to compete with teams of PhDs. And no new newcomer HFT firm will be able to compete. But as a result diversity of day traders will have decreased to exactly one. And any glitch in the operation of that firm (sorry, a box that was trading SPY just crashed) will be creating flash-crash artifacts without easy resolutions to them.
So yes, an increased entry barrier into day trading is bit of a problem.
Further, successful entry into arbitrage markets always required specialist infrastructure and skills. They used to be good old boy networks, brightly colored vests, and arcane hand signals. Now it is programming and networking.
In the previous world there was not "NASA" effect at all. Now at least the skills can be transfered to other industries.
Yes, but that tax would just be passed on to "regular investors" in the form of higher spreads. Market makers are providing a real service. That service has a cost. FWIW, that cost is now much smaller than it used to be due to the fact that computers are doing it now instead of real humans. Automation FTW!
"The following animated GIF chronicles the rise of the HFT Algo Machines from January 2007 through January 2012."
If they arranged the machines on the horizontal axis, time on the vertical axis, and showed 'packet sent' events it would look much simpler. Maybe simple enough to add in details about the trades, like dollar values, instead of pretty expanding explosions.
That's all "in principle" - whether the details work out right is a much deeper question.
It's not like the order book for Johnson and Johnson is completely empty and there is absolutely nobody out there willing to sell shares. If I understand it right, the HFT machines are poking numbers around the current price trying to step ahead of a regular order before it comes in.
These people are all being scraped, very slowly but very surely, by HFT. That's the point I'm making.
At least we all admit that the modern stock market has absolutely no connection to the economy or to the interests of long-term investing or capital building. It's just a very large and very legal casino.
Barriers to entry. Cost of business, etc... It's really not a concern to people who actually understand HFT and trade in their own way despite. They generally really don't mind the added liquidity.
These people are all being scraped, very slowly but very surely, by HFT. That's the point I'm making.
Buy in at $40. Sell at $80. Where does the computer scrape money from the transactions? It doesn't. Like I said, you're only losing if you're trying to trade like a computer. It's like trying to out-robot a robot on an assembly line.
At least we all admit that the modern stock market has absolutely no connection to the economy or to the interests of long-term investing or capital building. It's just a very large and very legal casino.
I disagree to the first part, and I think the second bit is an oversimplification.
It obviously matters, or else people wouldn't be upset by HFT or claim that they're losing money to it. We also have to consider the fact that pretty much all money comes through the bond and stock markets at exchanges. The exchanges are price arbitration centers for the economy. Paying attention to them matters because the exchanges and what happens there matters.
A casino is a place you go to lose money. It's all games and the odds are stacked against you. Everything we do in life is a gamble. An exchange is a place for people to make informed decisions about the prices of goods and financial instruments. Everyone loses and wins some.
I do relate to the long-term investing/capital building. The central banks keep printing money which funnels into the bond and stock markets. This in turn signals an abundance of actual economic capital, which there isn't. With the communication that there is plenty of excess capital for the long term, we (and especially people dealing in finance) act as if it is the case and start consuming capital. Eventually, facts catch up and we get some sort of terrible economic downturn. As a result of the long-term economic irregularity, actors act for the short-term because the long term is just too undependable.
It's the same sort of automation that we've seen in many many other places.
There may be some very minute portion of the industry that actually dedicates a lot of time to trying to predict specific actions of other market participants, but this seems like an awful idea to me. It would be much easier to try to predict the future price so you can avoid accumulating a large position in the wrong direction (as a market maker) or grab a big position in the right direction (as a speculator). It is also pretty easy to perform arbitrage, although this corner of the industry is harder to break into, since it requires more in the way of pricey hardware and data services and less in the way of strategy and predictive power.
Enough said.
They address a lot of the common criticisms of modern markets discussed here. It provides a good perspective of what people within the industry see as the current state and future of trading.
Would you say two half seconds rather than just a second?
See: http://www.amsterdamtrader.com/2012/10/duyfken-sails-into-qu...
As someone who hires and interviews in this space, HFT firms seem largely the same as old trading firms were. Boom bust cycles that comer and go as profitable trades shelf life expire.
Also, HFTs are most definitely taking profits from each other. In fact, in a lot of cases the faster, more successful HFTs kill off the new ones to the market. Not to mention there are a good number of predatory HFTs who exist solely to game other HFTs