Online Algorithms in High-Frequency Trading | Hacker News Reader

Online Algorithms in High-Frequency Trading

queue.acm.org

116 pointsstevewepay105 comments

The sad thing about any thread including the term "High-Frequency Trading" is that we won't be able to talk about the technology, and instead will need to relitigate automated trading.

I think we'd have been better off if the title of this thread had been "Online Algorithms in Automated Trading Systems". As far as I understand, that title has exactly the same meaning, but doesn't set off a crowd of HN commenters who have never placed a limit order jumping in to crow about _Flash Boys_.

I'm a little frustrated, since this is a technical topic that happens to be very relevant to me right now (I'm not a trader, FWIW).

So what is your opinion on the book Flash Boys?

tptacek already responded to your previous comment about Flash Boys below [0]

[0] https://news.ycombinator.com/item?id=9045150

The High-Frequency Trading Arms Race: Frequent Batch Auctions as a Market Design Response[1][2] is a must-read. The tldr; is that a continuous mechanism (sequential trade processing) only looks efficient in time-space, but is actually inefficient in volume-space (most of the volume trades at stale prices). The solution is discrete batch auctions (batch trade processing).

1. slides: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre... 2. paper: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...

This is really interesting, but doesn't it presume that there's only one trading venue for each instrument, and that instruments aren't correlated?

You can make time discrete for a single auction, but the time between auctions still has to be continuous unless everything's traded in a single exchange, which can't in reality happen even if we want it to (and we probably don't).

As I understand it, most of the most dramatic HFT stuff is already targeting multi-venue trades or correlations.

I think it is modeled in terms of endogenous vs exogenous entry. But I might be wrong, please correct me if I misunderstood.

It's obvious that they see the issue, since the examples in their slide deck are about correlation between different securities and between the same security trading on different exchanges. What I don't see is how batch auctions get around the distributed systems problem; we're still talking about an eventually-consistent system, right? There still must be opportunities for sniping, right?

(I'm not smart enough to know the answer, which is why I'm asking).

Decentralized assets[1] (technically possible already today, but not yet used for any high-volume instrument) have a single trading venue, namely the blockchain(s) on which the assets live. If multiple chains are involved[2][3], each individual trade is still atomic, but there would be a more complex ordering of trades: you could have trades which take longer than others to occur, and failed trades would also be publicly visible.

[1] https://en.bitcoin.it/wiki/Colored_Coins

[2] https://en.bitcoin.it/wiki/Atomic_cross-chain_trading

[3] https://github.com/TierNolan/bips/blob/bip4x/bip-atom.mediaw...

I'd share the same sentiment. I was really disappointed yesterday when the only comments (after 5 hours of nothing) was a seemingly snark question on the evil of HFT.

And to everyone who feel the need to voice their disapproval of HFT: if this was an article about improving the penetration of M1 Abrams, are you gonna be voicing your disapproval about ... something? Why/ why not?

Should we really be putting engineers to work designing more efficient techniques for punching plasma holes through sheet metal? They should instead work on improving sheet metal technology, as it is useful both for protecting both against natural disasters, and against those pesky hole-punchers.

It's interesting how you consider placing a limit order as a criterion for being eligible to comment on this topic.

The first section of this which relates HFT to one-pass algorithms is kind of useless. Online estimation algorithms are much more broadly applicable than that, and I'd expect that their relevance to HFT would be immediately obvious. I've never worked in HFT, but I've probably written the online variance algorithm at 3 of 4 companies I've worked. If you care about metrics such as "how quickly are each of my customers sending me requests?", then this stuff is relevant to you.

But more to the point, is there anyone in these fields who doesn't know this stuff? At a basic level, the things that you can do a really good job estimating in an online way are closely related to sufficient statistics, which every undergrad stats course should cover. And hopefully anyone actually involved in writing HFT systems has way more stats background than that -- right?

a) there is a pernicious and wide spread myth about the elite status of people working in HFT. My experience is that it is largely the same as any other technical profession. That is to say, it is no more or less likely for people in HFT to know about online estimation than any where else.

b) Given the reactionary view that many HFT participants have to information sharing, widely known/documented algorithms are often thought of as secret sauce and are frequently not talked about.

c) I agree with your sentiment specifically to one-pass algorithms being not terribly novel but am very happy that people with HFT backgrounds are publishing anything at all as it will hopefully help to remove the backwards thinking around secrecy so prevalent in the industry.

regarding a), not all technical professions require deep knowledge of massive data processing and time series modeling, so this statement is misleading

regarding b), the details of how exactly to use the algorithms are the secret sauce and those details matter.

I think people are reading your comment as "they need to know these things, therefore they are elite" - but I think you rather meant "it's not about being elite, just that they have to know XYZ so they are more likely to know W".

yes, exactly. But, the reality is that top HFT firms at least try to hire 10x programmers. So call them elite or not, but the calibre is high

Eh, I don't think that follows. Top firms in many sectors try to hire elite programmers. Do they succeed? Further, how much of the industry is made up of hires of top firms?

I have no particular position on the elite status of those working in HFT. Certainly there are some smart people there, but I don't know how to generalize. I don't have anything like a representative sample of either HFT or programming more broadly to compare, despite having worked in HFT and out of it.

Top firms in HFT try to hire elite programmers and they tend to succeed. Think of it this way, if you are hiring hundreds or thousands of people a year, then it would be extremely difficult to make sure most of them are elite. Plus, chances are your projects don't require it anyway. Many projects need good talent, but not elite. But if you are hiring 1-20 people per year, then making sure that at least some of them are elite is not terribly difficult

I can see the reasoning, I've just little confidence other dynamics don't dominate. I'm not saying I'm confident they do; and even if they do it's certainly possible for them to push the matter in either direction. But I'm saying that I don't know, that nothing you've said is anything I haven't considered, and that it's likely you also know less than you seem to think. And again, I'm saying that as someone who has worked in HFT.

I'm a systems programmer with a pretty primitive intuitive sense for maths. I "agree" with you, in the sense that the algorithms this article mentioned were intuitively obvious to me and thus seemed like they couldn't be sophisticated. What's the state of the art for online estimators?

I've been playing with simulations of market maker algorithms from the literature, and simple Bayesian stuff and MLEs are all I've really seen, which makes me think that the interesting stuff isn't in the literature.

There is a lot of literature, but it may not be easy to find out, and different domains use different names for it.

One keyword to look for is 'stochastic approximation', with a long (> 50 years) history in engineering and stats. Another one is 'stochastic gradient', which got a resurgence following the whole deep learning craze. There is also 'particle filtering' in signal processing.

One generic article I may recommend is http://leon.bottou.org/publications/pdf/online-1998.pdf, it does not require much math knowledge, but that may be less practical than what you want.

The 'onepass' keyword mentioned in the original article is another keyword to look for (see e.g. interesting work to do Monte-Carlo based linear algebra operations for very large matrices, this used to be a reference, not sure what's the state of the art is: http://cs-www.cs.yale.edu/homes/mmahoney/pubs/matrix1_SICOMP...)

> ... different domains use different names ...

I agree. In engineering, many of these algorithms can be coded using digital filters. Here's the exponential weighted variance:

  import scipy.signal as dsp
  import numpy as np
  
  b = np.array([alpha])
  a = np.array([1., -(1 - alpha)])
  
  # Compute exponential weighted average        
  signal_ = dsp.lfilter(b, a, signal)
  # Compute exponential weighted variance
  ewv = dsp.lfilter(b, a, (signal - signal_)**2)

There is another article in this same edition of the ACM Queue.

http://queue.acm.org/detail.cfm?id=2536492

It goes a bit further into the mechanics of market maker profitability. Does your simulator reflect what your queue position would be in a real electronic market? Does your market making algorithm have logic try to scratch out when it looks like the market is going to move against its position?

The particular emphasis on one-pass algorithms might seem self-evident, but the thread running through all of this is an engineering challenge that is very relevant to HFT. If your algorithm takes 60ms to evaluate your current position on each tick, by the time you are finished, the market has moved to the next zip code. Thus, to keep up with the market, you might find yourself rewriting the network stack for better response time.

Is there a mirror? Seems like too many connections at the moment.

google cache should do the trick if it's down

http://webcache.googleusercontent.com/search?q=cache:WqysSpS...

Needs a date stamp (2013).

Could somebody explain what the use is of HFT to society? Because I don't get it.

Links are also welcome.

Shortening the bid/ask spread and increasing liquidity among other things.

http://en.wikipedia.org/wiki/High-frequency_trading

Before HFT, there was only a few market makers, which meant the bid-ask was high (if you wanted to buy or sell right now, you'd pay more), liquidity was low (it'd be harder for you to get in or out of a large position and in hard times it would take longer for the market to recover - See 80s crash vs flash crash).

With HFT, there's a lot more competition over the bid ask spread. Trading firms are essentially in an increasingly expensive fight over pennies, which is good for everybody else, since it's cheaper prices, more liquidity, and a more efficient market.

The people it's bad for is the old-school style brokers (which are effectively gone now) and large investmant banks who have a fundamental view on the market (they need to be smarter about executing since the market is more efficient and reacts much more quickly to their buying and selling).

Which brings us to the next question: What is the social benefit of a slightly smaller bid-ask spread and a few milliseconds of faster liquidity? And are these benefits worth the massive amount of wealth extraction from the rest of us that HFTers are engaged in?

That's pretty simple to answer. The smaller the spread, the less going into the pocket of middlemen. Which is where we are now. Less going into the pockets of middlemen than ever before.

>And are these benefits worth the massive amount of wealth extraction

What "massive amount of wealth extraction"? Are you just making shit up?

> What "massive amount of wealth extraction"? Are you just making shit up?

I think he was probably just confused about how HFT works, since unless you're familiar with it it is very easy to assume it is screwing someone.

> And are these benefits worth the massive amount of wealth extraction from the rest of us that HFTers are engaged in?

Unless you were a specialist market maker or pit trader on an exchange there hasn't been any "wealth extraction". Quite the opposite, HFT has taken a system built on being tall, having good connections, and understanding an arcane set of hand signals (not to mention very high fees) and replaced it with a ruthlessly efficient automated system based on being good at writing software and planning network setups (and brought the fees down with it).

For the vast majority of market participants HFT has been a massive upgrade over the old system and dramatically lowered the cost of trading.

The book "Flash Boys" by Michael Lewis chronicles the rise of HFT, and basically argues that it doesn't really provide that much of a benefit.

http://www.amazon.com/Flash-Boys-Michael-Lewis/dp/0393244660...

Standard PSA about "Flash Boys". If you are interested in HFT you are better off quite literally not reading that book than reading it. It is very biased and largely inaccurate.

If you want to read narrative non-fiction about HFT, "Dark Pools" by Patterson, which also has its mistakes, is much better.

Are there any other resources you can recommend for learning more about HFT?

For blog post style reading 'yummyfajitas' series on HFT is good:

https://www.chrisstucchio.com/blog/2012/hft_apology.html

For technical books:

- Trading & Exchanges by Harris (a little out of date)

- Algorithmic Trading & DMA by Johnson.

On the narrative non-fiction side Dark Pools is basically it.

Great, thanks!

There are several books which speak about HFT at a high level. If you are curious about the actual mechanics of stock exchanges and (a bit about) how various parties make their money, I wrote a bit about it at http://falconair.github.io/2015/01/05/financial-exchange.htm...

I would also recommend reading "Flash Boys: Not so Fast" (http://www.amazon.com/dp/B00P0QI2M2).

Written by a high frequency trader. Are there any third parties that have deeply explored HFT and come out in defense of it?

Sure. Vanguard has the widely publicized view that HFT helps to bring down the cost to retail investors for instance

http://www.ft.com/cms/s/0/ff8c6486-cb37-11e3-ba95-00144feabd...

This is a really fun read. Thanks for pointing it out!

Suggest you read this rebuttal. Lewis' book is deeply flawed. http://www.amazon.com/Flash-Boys-Insiders-Perspective-High-F...

Middlemen have always existed in markets. The middlemen have changed, but as you rightly point out, they now take SIGNIFICANTLY LESS than they used to because competition has forced spreads down to next to nothing. You pick: pay a 0.30 spread on a NYSE listed stock or a 0.01. It's sort of a no-brainer.

The thing that most people seem to be upset about is that now market makers generally have few to no obligations to provide liquidity and they are drastically smarter than they used to be.

Ultimately, unless you are slinging enormous blocks, if you send a marketable order through a retail broker these days you are almost always going to get a better fill than the displayed market. Likewise, if you are a large trader you have to be more sophisticated about how you fill your orders, you can't just lift 100k shares even when 300k are displayed unless you do so with a semi-intelligent execution algo (note: these are everywhere these days, and not using one brings into question the trader/brokers abilities and whether they are even qualified to be in the market).

If HFT lets Vanguard lower their costs, the savings of which are passed on to their customers, that's more money in the retirement accounts of regular people.

I just got an email that the expense ratio of my Vanguard target date fund is going down, so costs were reduced somewhere.

> The people it's bad for is the old-school style brokers (which are effectively gone now) and large investmant banks who have a fundamental view on the market (they need to be smarter about executing since the market is more efficient and reacts much more quickly to their buying and selling).

What about index funds rebalancing their portfolios? (Honest question)

The Chief Investment Officer at Vanguard, of VFINX fame and widely considered to be among the most trustworthy fund companies, stated that HFT had improved things for firms like theirs.

Index funds, or really anyone with long horizon investment objectives, are major benefactors of HFT. Note, "HFT" really means nothing in modern context, but regardless, what most mean are "Electronic Market Makers". Long horizon investors benefit directly from being able to access cheap and plentiful liquidity pretty much whenever they want to. Spreads are tighter because there is now significant competition to provide that liquidity.

By "fundamental view on the market", I mean those who have done research and know that the price of something is under- or over-valued. These are generally known as informed investors. When they decide some security/commodity/etc is mispriced, they will immediately put a lot of money into that idea, ideally (for them) before the market adapts. With HFT, the market adapts a lot quicker than before, which is better for general investors (who are uninformed investors) as they get a price that's closer to the real value.

Index funds don't have a view on specific securities. They don't perform research to try to get an edge. Instead, they believe that the market is generally efficient, the and they try to lower risk by diversifying across a large portfolio of somewhat related securities, and get a return that closely matches the overall market. As such, HFT helps them by givings them better bid-ask spreads and also providing prices that are closer to the actual value of an asset.

Note that those performing fundamental analysis and buying millions of dollars worth of assets aren't SOL. They just have to be smarter about how they execute.

A nit: informed traders are the apex predators in the ecology that market makers live in. In the literature, market-making algorithms are designed around hiding from them. HFT helps informed investors even more than it helps index funds.

Unfortunately, there are a couple types of HFT. At it's core, it just means trading using a computer to execute more/faster trades. That seems justifiable in making the markets more efficient. Where most people including myself have issues is when they get a pass in certain markets to frontrun normal sellers and eat some of their margins simply because their latency was low enough. An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal.

> Where most people including myself have issues is when they get a pass in certain markets to frontrun normal sellers and eat some of their margins

Luckily that is not allowed (nor technically possible) on any credible exchange.

> An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal.

You probably meant: https://news.ycombinator.com/item?id=8977198

Spoofing, which is what you are describing, is not useful for countering HFT, it is used to change the market price of an instrument by exaggerating the demand for it. This has obvious detrimental impact on any one, HFT or otherwise, who is trying to price the instrument.

Is it necessary for things to "have a use to society" to be allowed?

Where is the commitee that decides on the allowed actions?

The committees are called "governments" and rule life in nation-states. The dominant ones have a capitalist morality and therefore make laws to create and enforce it.

Depends on the nation: http://en.wikipedia.org/wiki/Everything_which_is_not_forbidd...

This guy's blog is pretty good:

https://scottlocklin.wordpress.com/2011/03/01/the-arguments-...

https://scottlocklin.wordpress.com/2014/04/04/michael-lewis-...

We see crazy competition on latency because these firms can no longer compete on price. HFT has taken the spreads down from $.25 to $.01 (putting a lot of market-makers out of business, saving a lot of money for anyone who is actually trying to buy or sell shares). But since the price isn't allowed to go below $.01 (sub-penny rule) the firms are now all in this wasteful zero-sum race to get that $.01.

While you will hear many negatives here I believe there are few positive outcomes.

The needs of HFT are such that technology barriers are pushed to the limits. Some of the technologies developed in the process make it back to the general public.

https://www.chrisstucchio.com/blog/2012/hft_apology.html https://www.chrisstucchio.com/blog/2012/hft_apology2.html https://www.chrisstucchio.com/blog/2012/hft_whats_broken.htm...

The second link specifically talks about social utility.

There is no benefit. HFT does not lower spreads in reality, only nominally. They put in orders but back away when a matching order comes in. The myth is that HFT improves liquidity. It may look like that. But those orders aren't real and can't be fulfilled.

This is again how scientificism goes. We think machines improve everything. "If only we trade faster, that will help the markets!" It's not hard to see this is BS.

The trouble with this argument is that, on basically every exchange in the world, you do not get to find out if someone has matched with your passive order until after the trade has happened.

There is no possibility to "back away when a matching order comes in" because the first time you find out about the matching order is when you are informed about the trade that you just did!

What can happen is that a market maker will post liquidity on multiple exchanges A, B and C, and will remove their orders from exchanges B and C if they are filled on exchange A (assuming they are fast enough to do it!)

This is not shady or nefarious. In fact, it is the only way that a distributed market could possibly work.

The market maker has some finite capacity to provide liquidity (e.g. they have position limits, they need to post margin, they have certain risk tolerances). They want to maximise the chances of their orders being traded on (i.e. providing liquidity), so they post liquidity on all available markets.

When they match on one of those markets, their capacity to provide liquidity has diminished, until they manage to offset their position somewhere (either on the same market, or on a different one). Moreover, they have just received a new piece of information about the supply/demand imbalance for this particular stock. For both of these reasons, it is natural to assume that they would want to adjust the prices they are offering on other exchanges. The market cannot function any other way.

I like asking programmers to think about the CAP theorem when they consider this problem.

You have the wrong question. The question is: What is the optimal market structure? Trying to strike a balance between the benefits of competition and the troubles of market fragmentation. Fragmented markets = HFT + competition. Unified markets = monopoly power + liquidity. You choose...

Wait, why would monopolies on trading venues and market-making provide more liquidity? Shouldn't liquidity be on the other side of the comparison?

A single exchange brings all traders together reducing search and monitoring costs and thereby reducing trading costs. The single exchange as a monopoly can extract rents from members and has little incentive to improve stuff, e.g. infrastructure. So the tension is between the monopoly power and the liquidity externality arrising from bringing traders together to the same place. The market makers do not necessarily have a monopoly, they can compete against each other, but face monopoly level fees, potentially. (This is my amateur economic analysis, but I think basically this is what economists think.) A fragmented market increases competition on fees and infrastructure, but traders face search, monitoring costs and the cost of dealing with HFTs.

I find this question soooo frustrating. I mean, we live in a free ( relatively speaking ) capitalist society. Why does there have to be use to society for anything, unless the issue in question has negative externalities on others ?

HFT may have negative externalities on others. The money they make has to come from somewhere. For example: Buyers pay more or seller get less than they'd get without HFTs acting on the market.

The (quite common) liquidity argument isn't a very good one IMHO, given that HFT have no interest in holding positions for any long time, so they won't buy what they're not reasonably sure they can sell quickly (ie. where there already is liquidity).

The other argument here is that before HFT, traders wanted even larger fees. HFT can't be the cheapest possible option out there, if they can afford to lay their own fiber between continents to operate.

So what is it that makes HFT desirable enough that we (as a society) shouldn't find means to make trading happen without them?

The money they make comes from a premium the market's "officers" used to extract from the market's "customers". Market officers used to scalp dimes and quarters off the price of every share transacted by deliberately quoting wide spreads. Automated traders shrunk those spreads, so much so that they're pushing the boundary of a single penny.

The great thing about this is, automated traders don't capture the value they're creating by bidding the spread down: the buy-side does! Automated traders are getting paid to liberate money that used to go to fees for sell-side middle-men.

The sell-side is, as you can read subtextually from _Flash Boys_, pretty furious about this.

So what specific part of HFT operation makes them able to do that?

Is it their own (really public) instance of Core War (https://en.wikipedia.org/wiki/Core_War)? Is it the milliseconds advantage gained through custom transatlantic fiber?

If either of these aren't core to what enables them to dump prices, we could get them down even more by getting rid of these toys.

> So what specific part of HFT operation makes them able to do that?

Lower costs by having computer programs rather than human traders. Two or three programmers can do the trading that would previously have taken a hundred traders, and there's a corresponding saving in salary/bonuses/etc.

> If either of these aren't core to what enables them to dump prices, we could get them down even more by getting rid of these toys.

How would you do that? Having algorithms is core to it. Having custom fiber is only core to beating the other HFTs (they can't compete on price so they compete on latency), but what are you going to do, pass a law against installing optical fibres?

Prices are already as low as they're legally allowed to be (sub-penny rule). Which is what causes a lot of the weird millisecond-chasing.

"The money they make has to come from somewhere" - yes, it comes from willing participants engaging in voluntary transactions. "May impose exernalities". That's pretty much true of just about anything, but good evidence is necessary to consider this statement useful

Anyone who thinks HFT is something needed should go read Flash Boys. HFT is basically front running with some variations. There is a small amount that is necessary arbitration that will have to be done at some speed, but by and large it is not performing any sort of a service, it is taking advantage of the people trying to buy and trade stock by being a manipulative middle man.

There are three sentences in this comment. The first two are wrong, and the third is hard to understand, but if I understand it correctly, it too is false.

1. _Flash Boys_ has been panned by technical experts and makes very little sense to people with a basic understanding of how the markets work. I can link to technical reviews if you'd like to see them. I'd be interested in any technical reviews of the book that defend it; I haven't seen any. As a fan of Michael Lewis: this book was surprisingly bad.

2. Front-running is an agency problem. It's what happens when your broker knows you're about to move a lot of stock and, acting on that insider knowledge, jumps in front of your trade. Market-making isn't front running. To make a colorable argument that HFTs are frunt-runners, describe a hypothetical sequence of trades in which that happens ("BOB BUY 10 @ $10, CAROL SELL 10 @ $10.01, &c"). If HN history is any guide, that exercise will quickly deflate the notion that there's front-running (or even any unfairness) happening.

3. HFTs were preceded in the markets by the human specialist system, in which people deliberately increased the spreads to scalp money from trades. HFTs fight over pennies. Human specialists rigged the stock markets so that every share included a premium denominated in dimes and quarters paid directly to the specialists.

Could you please link to the technical reviews you mentioned? I would be very interested to read them because as an outsider, Flash Boys seemed overly simplistic, but I haven't been able to figure out what the real story is yet.

Here's a good free starting point:

http://www.amazon.com/review/R3PJO6KJGRMWUE/ref%3Dcm_cr_pr_v...

Again: a single positive technical review would be interesting. I'm not being hyperbolic: I haven't read a single one. The sense I get is that everyone who's ever formatted a FIX message thinks _Flash Boys_ is terrible.

The 2nd-to-last slide[1] compares the IEX "speed bump" to other anti-HFT measures: "Ingenious, eliminates sniping. But, only works while IEX is small relative to the rest of the continuous market (free-rides off price discovery elsewhere)."

1. http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...

I replied to this but now I'm uncertain about my response and am rereading. This is neat, thanks!

So if all this is so great, how come so many people are going to great lengths to eliminate HFT with the exception of the people that make money from it? If it was offering such a valuable service, wouldn't people want it to be there even if they weren't the ones profiting from it? Flash Boys has been panned by HFT people, not people trying to trade stock fairly.

Simple: pursuit of the status quo ante.

HFT ate the lunch of the people who benefited most from the old system: giant broker-dealers (investment banks) who made grotesque fees for the simple service of shopping large blocks on behalf of pension and mutual funds. Those funds now get better service without special fee-collecting agents.

_Flash Boys_ was not just panned by HFT people. Where are you sourcing that objection from?

It sounds like you are confusing straight electronic trading with HFT. Brokers aren't necessary and neither is having someone able to execute trades with sub millisecond latency.

Also, why do you keep putting underscores around Flash Boys?

You asked a straightforward question. I answered it. I don't understand your followup questions, and so will leave it at that.

Nothing in this comment contradicts or disproves that front running can and is happening. Flash Boys described a large order going to an exchange and HFT firms seeing that, buying stocks on other exchanges before the real purchaser, and selling them back to the real purchaser after the first exchange they get to can't fill their large order. HFT firms know there will be buy orders at the other exchanges and they run out in front of those buy orders to become a middle man. That is the very definition of front running and it was obviously happening. Do you have information that directly contradicts this?

First, that's not the definition of front running, and second, the expectation that a large order can trade atomically has never been valid, and, if you consider the computer science behind it, can't be valid.

If that isn't front running, then what is? And if you wouldn't call that front running, what would you call it? You haven't offered anything to back up any of what you've said.

All I'm saying is that sub-millisecond HFT is only useful to the people doing it and no one else. People defending HFT as being necessary and helpful is ridiculous. Even an exchange where orders go in over the course of a second, are matched up within the exchange, then traded, would not offer latency to a person, but would offer everyone a chance to sync with the exchange cycle and submit their orders with enough time to be matched. If all the exchanges someone was trying to use worked this way it would offer no disadvantage to a human trader, but would make the HFT arbitrage much less lucrative.

Like I said upthread: front-running happens when a broker takes an order for a client and, before submitting it to the market, trades against that order. Front-running is an agency problem.

An HFT firm doing that would get fooled by activity on one exchange which resembled part of a large order being filled, but didn't actually propagate to other exchanges. These firms are doing a sort of high-frequency gamble, whereas a broker acts on certain information to which only they are privy before its execution.

There is a huge gulf between "there exists bad behavior that can fairly be called HFT" and "all HFT is bad behavior." Even if we agree that the behavior you describe is bad (it's not "front running" - that has a particular other definition), much HFT does not resemble that.

Mistakes aside, the weirdest thing about Flash Boys, to my mind, is that it's cast as a David vs. Goliath narrative - with the role of "Goliath" filled by the awful HFT shops and "David" by Goldman Sachs and J.P. Morgan. When "David" is orders of magnitude larger than "Goliath" and tends to swing the industry by the tail, it's an odd framing.

Reply on news.ycombinator.com