U.S. stock markets are rigged, says author Michael Lewis | Hacker News Reader

U.S. stock markets are rigged, says author Michael Lewis

reuters.com

223 pointsschintan192 comments

The approach touted by the 60Minutes piece, the coop-like IEX market that aims to equalize participant lags, is one obvious countermeasure.

Another promising tactic could be the "Frequent Batch Auctions" as described in the paper referenced in this blog post:

http://johnhcochrane.blogspot.com/2014/02/budish-cramton-and...

Direct paper link: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...

Paper abstract:

We argue that the continuous limit order book is a flawed market design and propose that financial exchanges instead use frequent batch auctions: uniform-price sealed-bid double auctions conducted at frequent but discrete time intervals, e.g., every 1 second. Our argument has four parts. First, we use millisecond-level direct-feed data from exchanges to show that the continuous limit order book market design does not really “work” in continuous time: market correlations completely break down at high-frequency time horizons. Second, we show that this correlation breakdown creates frequent technical arbitrage opportunities, available to whomever is fastest, which in turn creates an arms race to exploit such opportunities. Third, we develop a simple new theory model motivated by these empirical facts. The model shows that the arms race is not only socially wasteful – a prisoner’s dilemma built directly into the market design – but moreover that its cost is ultimately borne by investors via wider spreads and thinner markets. Last, we show that frequent batch auctions eliminate the arms race, both because they reduce the value of tiny speed advantages and because they transform competition on speed into competition on price. Consequently, frequent batch auctions lead to narrower spreads, deeper markets, and increased social welfare.

In a batch auction like that you still have a latency game. There is no reason to submit your order until the last possible moment since you will want to incorporate as much information from other markets and news sources as possible. You would just end up with a crush of orders in the last 100 uS of the batch interval.

I haven't read the paper, but I would guess that the one-second-worth of orders in each batch are secret until the batch executes

Doesn't matter. There is still the incentive for low latency.

A much smaller and less exploitable incentive. A few milliseconds of info from other news sources is not abusive as far as I know. Unlike the ability to react to pending trades before they happen.

You are obviously not involved in the trading. A few milliseconds definitely matters. And how would you react to a "pending" trade before it happens?

I think you are conflating two types of information that can be taken advantage of:

1) Information about other orders coming into the system 2) Information about world events that may affect stock prices

The whole point of HFT is extracting maximum information from (1) and using it to make money (supposedly from slower market participants). That is what batch auctions would remove.

Nobody minds if people make money from the second type of information - but it is many orders of magnitude less frequent (ie how many times a day are there economic releases or earnings posted)?

Then why are people investing massive amounts of money to shave milliseconds in communication times between trading markets (London, Tokyo, NY, Chicago)?

They are all attempting to gain an edge on correlated inter market differences.

That only matters because both exchanges have the same problem. I think it is definitely true that if both exchanges ran on a synchronized 1 second schedule the advantage would go away, because the information from the previous round would be uniformly available early in the next.

It is less clear to me whether fixing just one exchange fixes both or neither.

The ONLY reason HFT companies invest money to shave milliseconds is to have quicker access to orders that come into the system (this article elaborates: http://blogs.wsj.com/marketbeat/2012/09/20/collocation-the-r...)

It absolutely has nothing to do with reacting to real-world events. So if all activity were to take place at 1 second intervals (kind of like microprocessor clock) - their main advantage would go away.

What about randomizing the auction time?

That sounds like an interesting wrinkle. The (odds of a) round closing could also be sensitive to the in-rush of sealed bids or even how different later orders are from earlier ones. That might help ensure an early-close either just before, or just after, a wave of orders that hints at information the earlier-bids didn't have. (That is, tick the unseal-matching process based on a signal of substantive change, rather than only wall-clock-time.)

But that way you could force the closing of a round by flooding with bids, effectively giving more control rather than less to anybody trying to abuse this system.

Not sure that'd be fatal, or worse than the current races.

The reset after a calculated flood-to-force-a-round-closure might even bring a compensating external benefit: a slight pause to digest the new information.

It all depends on the specific mechanism design, including choices like whether a flood can be guaranteed to land in a certain round... or might wind up as 'first in' to a subsequent round.

From the 60 Minutes piece: http://www.cbsnews.com/news/is-the-us-stock-market-rigged/

Ah, a band-aid on a band-aid.

What was wrong with open outcry? Sure, you had to physically be there, but nothing has changed - traders still cluster around exchanges to keep latency low, HFT kit still lives "on the floor", and what was once done with humans and bits of paper is now done in a far less efficient fashion with computers. Less, because the marginal benefit ends up going into the pockets of those who can afford exchange co-lo and mountains of hardware to parse FIX a tick faster than the other guy, meaning that the rest of the market ends up suffering, and not having a more efficient fill mechanism, as touted.

HFT is that asshole who takes his umbrella with him into the pit, and blocks your trades.

If anything, the solution here is to abolish algorithmic trading, and decrease tick resolution substantially, which if lengthened enough, would take C out of the question and make the entire market equally accessible to all.

I mean, seriously - does the value of Brent light sweet actually change thousands of times per second? According to the markets, yes, according to reality, where people buy futures with straddles/swaps so as to have a fixed, manageable cost, no.

The entire thing is un-necessary, and doesn't add value to humans.

The most obvious and simplest thing wrong with open outcry and human market makers is the number of specialists that it takes to handle a full universe of stocks. A small electronic trading team can provide/take liquidity in 8000 stocks or greater. If you need specialists you are going to have to provide a living salary for every single specialist who will only be able to handle 20 stocks at most. The cost of trading for the end consumer is higher.

> The cost of trading for the end consumer is higher.

What about when you factor in the artificially elevated buys and artificially depressed sells as a result of HFT?

What specifically are you referring to?

Did you read the article? :) Pre-emptive buys driving up the price across exchanges.

The cost of trading is still drastically lower now than it was with open outcry. There were a number of federal indictments related to NYSE specialists blatently ripping off retail customers.

http://www.nytimes.com/2005/04/13/business/13nyse.html

My favorite solution to the breakdown of correlation is to have order books for beta adjusted prices. That is, you don't bid for Google stocks, you bid for a Google stock minus an equivalent amount of S&P. The price of this pair is much more stable than the price of Google, which means spreads can be tighter.

The nuttier generalization would be to have an order book for each principal component of the covariance matrix of the market. It wouldn't work in practice for a variety of reason (least of which is that the covariance matrix is unstable, ill defined, etc) but it's a good thought experiment to understand the problem.

This is an interesting idea. For this to impact market correlation, would you need to insist that everyone place their orders in beta adjusted terms or would it be enough for the bid making protocol to allow for it?

Allowing it is enough, and you could populate it initially by deriving quote from the other books. The CME does it for futures.

For those of you who want to understand this topic better (i.e., a lot of the people spinning conspiracy theories), you might want to learn the basic mechanics of HFT. I wrote a tutorial on the topic a couple of years ago, might be helpful:

http://www.chrisstucchio.com/blog/2012/hft_apology.html

http://www.chrisstucchio.com/blog/2012/hft_apology2.html

I really like your idea of allowing sub-penny ticks in the highly liquid stocks as a way of tilting algo trading away from the latency arms race. Unfortunately, it probably runs too counter to the uninformed "no one should be trading to make $0.01 per transaction!" popular perspective for the SEC & exchanges to implement it.

As someone who actually makes a living in HFT, if you really wanted to get rid of me, sub-penny ticks would be the single worst thing I can think of. There is literally no transaction tax that would make trading unprofitable. 2 zero's worth of decimilazation though and I have to go find honest work.

I'd only get rid of you insofar as you are fighting over latency. If you can deliver price improvement or useful market depth there is still a place for you.

latency arb is still important to keep venues in sync.

I think decreasing the minimum bid/ask spread value would make it very difficult to continue doing traditional market making. I think that exchanges would then need to add some other incentives/perks to ensure liquidity providers stayed on their exchange, which starts to sound more and more like a specialist system which I'm generally against.

Of course, I don't think there is some natural right for market makers to exist. I do really like the idea of an extremely small tick size as it much more elegant than a transaction tax.

Nice posts. You see 0 value to the latency arms race, and I think you are probably right.

However, I think it's not hard to argue that some speed increases have value. For example, if the world's only stock market only allowed trades to happen n times a year, we would expect a pretty inefficient market and this mechanism for capital allocation would be pretty poor. And as we increase n, market efficiency would increase, and this has value.

So are we at a point right now where an increase in n has no value (say market information gets processed on the order of nanoseconds instead of the current microseconds)? What's the cutoff point?

This is a good question and one I don't have a good answer to. I don't actually know the optimal points for most of these values and I'm generally content to let the market decide.

This is why I think eliminating the subpenny rule is a good policy change - it eliminates a market distortion while not requiring you to perfectly tune a parameter (e.g., the size of a transaction tax or a minimal trade latency). All it does is allows price competition.

Of course as a real policy proposal it's dead in the water, since it doesn't provide much opportunity for moralizing or posturing.

> You see 0 value to the latency arms race, and I think you are probably right.

How do you draw the line between investments in reduced latency which are socially beneficial, and those which aren't? It's not like these are cheap investments that HFT firms are making. If they're truly pointless, we should expect them to lead to overcapacity and ultimately the exit of some players (unless, of course, the arms race is being unnaturally subsidized in some way, e.g. by the lack of a sub-penny rule).

When Paul Reuter was using carrier pigeons and telegraphs to move tradeable information between stock exchanges ahead of his competitors, I wonder if people considered this an arms race of negligible social value. After all, the world had gotten by just fine without telegraphs up until then.

Meaningful stuff could happen in the elapsed time, so motivating people to get info there sooner was more valuable. As someone who's working in HFT, I think that the surface defined by a given expenditure in (expenditure on price accuracy)x(expenditure on speed)x(profitability) space, the gradient points too far toward speed on too much of the surface.

Saying something that is such a zero-sum waste is great because as a side effect it brought us the telegraph is like saying government bureaucracy is great because firms have had to develop new technologies that happen to be useful in other domains in order to deal with it.

> If they're truly pointless

You've conflated "value to society" with "potential to make money". The two are unrelated. Dumping toxic waste in the river may make money, but has negative societal value.

Whatever societal value there was to faster trading was exceeded when trades could execute by the time your browser refreshed. Humans make financial decisions on the scale of minutes at fastest.

At this point HFTs are still pursuing "making money" (and there may still be money to be made) but certainly not providing any societal value whatsoever; since they are stealing from the general public their societal value is strongly negative.

> So are we at a point right now where an increase in n has no value (say market information gets processed on the order of nanoseconds instead of the current microseconds)? What's the cutoff point?

My intuitions want to say the point of diminishing returns is around the limits of human cognition. Stocks are ownership of some human activity, so once you start valuing them faster than human activity you're really redefining what stocks are. Now stocks become predictions on ownership of some human activity.

Since the first formalization of Sport (some might argue earlier), it's been a human tendency to go Faster, Stronger, Better. Moving from ownership of some human activity to predictions on ownership of some human activity is a manifestation of this Faster, Stronger, Better (since the prediction is itself a human activity... we're owning the human activity of predicting human activity).

All that is not to say any of this is inherently good or bad (I disagree at the loaded term 'rigged' being used by Lewis). There's no single cut-off point in additional value earned. It's more of a transition zone where you're not sure when exactly you left it, but at some point you realize on the other side is not the world you started in.

I don't see why the speed of human cognition should have anything to do both it. After all, the HFT algorithms are the result of human cognition. It's just computers "doing human cognition" much faster than human brains can, which is what all computers do.

This is a pretty weak defense of HFT:

>Although the latency competition provides little to no value to the end >consumers, all HFTs must play the game. If they don’t, a faster HFT >will beat them to market and their order flow will be reduced.

So yes, it's essentially useless to the retail or wholesale investor.

Actually, worse than useless, since the HFT has a built in advantage that brings no value to the rest of the market. It's an endless zero sum game where other investors (who do the real work) are losing.

>I’d also like to note that I don’t believe automated market making is useless. It isn’t. Providing liquidity and reducing the spread is valuable.

Which HFT is actually pretty poor at. Spreads were reduced significantly by having electronic trading systems (computers are cheaper than manual labor), NOT through high frequency trading systems (did spreads drop significantly in the last 5 years?).

Liquidity was already maxxed out in the equities market before HFT, too. HOUSES are illiquid -- it takes weeks to close on a sale on one of those at best. THAT market could do with additional liquidity. Would be nice to close on a sale in a day, right?

Being able to buy and sell your share a thousand times in a second is not providing useful liquidity to anybody except HFT traders.

At the end of the day they're levying a tax on the rest of the market - i.e. as the OP points out, the market IS rigged.

>In the Hacker News comments on part 1 of this series, there were several comments suggesting that HFT’s somehow steal pennies from ordinary investors. I don’t agree with this claim, but I’m very interested in hearing well reasoned disagreement.

I'd be interested in hearing this well reasoned disagreement with the OP. I'm not so interested in reading the same old tired defenses of HFT, though.

It's not about liquidity or transaction costs. It's about gaining an information advantage similar to what front running or insider trading gets you.

So yes, it's essentially useless to the retail or wholesale investor.

Having liquidity available to take is not useless. You should re-read the article - it is a zero sum game between liquidity sellers, and a positive sum game between liquidity sellers and liquidity consumers.

At the end of the day they're levying a tax on the rest of the market...

It's not a tax. If you choose not to take liquidity you don't pay for it. If you do choose to take liquidity you pay less than you would under other systems (e.g., specialists).

>Having liquidity available to take is not useless.

Um, I covered that point pretty thoroughly.

>It's not a tax. If you choose not to take liquidity you don't pay for it.

It isn't a choice. You cannot chose to not participate in an equities market without HFT. Not right now, anyway.

>If you do choose to take liquidity

You you even understand yourself what that phrase means?

If I post an ALO order and don't cross the spread, what "tax" do I pay to the evil HFT?

And why would I not be paying this to a human market maker, in the event that HFTs hadn't driven them all out of the market?

If it's not a tax, then why was the RBC guy able to save millions of dollars by using software to ensure that his orders arrived simultaneously at all the exchanges? If, as related in the 60 Minutes story, institutional investors are each being charged hundreds of millions of dollars per year, how can that be seen as anything other than a tax?

Maybe it's more of a toll or a rent than a tax

Institutional computer buyers pay hundreds of dollars to Dell for every computer they purchase. Is Dell "taxing" them? Of course not - if they don't want computers they don't need to buy them. If you read the article, you'd realize the RBC guy was buying liquidity. He wants to purchase what HFTs are selling.

There is a simple obvious way for anyone to avoid paying the HFT "tax". Use ALO orders. They come with execution risk, but you don't pay the spread and the matching engine will pay you rebates. Can you tell me why anyone anywhere would choose to purchase from HFTs rather than using ALO orders?

You'll notice that any accurate argument against HFT can be applied to any kind of trading.

There's nothing sinister about HFT except that it's a faster version of ordinary market making. If you think market making is evil, then you'll think HFT is evil. But market making is perfectly healthy, economically speaking. Information advantages and speed have been part of this world in the 400-odd years that centralized trading has been a thing.

HFT is a threat to other market makers, but you're smart enough to know that common people shouldn't be market making and can (provably) only benefit from reduced spreads and increased liquidity. Right?

Nonsense. HFT is a variety of front-running, which is provably only a detriment to "common people", and is illegal in most cases.

They even have an example in the linked article of a trader being forced to pay higher prices for his buy orders due to HFTs front-running his orders; you literally couldn't be bothered to even read the linked story.

HFT is a variety of front-running...

Please explain the mechanics of this.

...a trader being forced to pay higher prices for his buy orders due to HFTs front-running...

Clearly you didn't understand the article, which was about predatory trading increasing market efficiency. It was not about front running, which is when your broker trades in front of you in violation of his fiduciary duty to act in your best interest. (Note that your competitors do NOT have the duty to act in your best interest.)

Just because the exploit is illegal in one circumstance and not in another doesn't mean it's not an exploit.

When the exchanges are complicit in the deal - being paid for sending information early to particular parties - it's pretty easy to argue there is a conspiracy to defraud the general public, and so regulators have regularly argued:

http://www.gotgoldreport.com/2014/03/why-new-york-ag-wants-c...

I assume that when these practices are finally banned you will change your tune, right?

I don't understand. Using the multicast quote stream (which costs a few thousand/month) is an "exploit"? Or are traders somehow obligated to ignore the actions of other traders? Perhaps anyone using a computer is obligated to act as an agent for people without a computer, rather than a competitor?

I truly don't understand what you are arguing. Could you please explain the mechanics of what you believe should be illegal?

The article you linked calls it insider trading and not front running. In addition, the issue discussed in the article is about making company information public earlier to a select group of people (which is insider trading). However, most HFT (particularly market makers) do no care about this and rely only market data provided directly from the exchanges. This data is provided fairly to everyone.

>Please explain the mechanics of this.

You say that a lot.

I already explained the exact mechanics in this comment: https://news.ycombinator.com/item?id=7500223

...which you did not respond to.

I previously ignored your comment because in the context of the conversation, it was completely irrelevant. In the comment you replied to, I was asking how one can "cut in front of you". Your comment was just about rapidly trading and rapidly waving one's hands in order to somehow do something evil.

"Use pattern recognition software to outcompete other traders" is not a trading mechanic.

no, the higher price was due to arbitrage and not front-running. Large orders always move the market (this is simple supply/demand economics), HFT makes this move much faster than before. Large orders are usually cut-up in smaller pieces to minimize this price move and get a better price. Katsuyama is just butt hurt that his 1990's trading style doesn't work anymore.

It has absolutely nothing to do with front running. Front running has a very specific definition in market regulation.

Front running refers to a broker placing his orders (i.e. the BROKER'S orders) in front of his customer's orders. This is a violation of fiduciary duty, SEC regulations, FINRA regulations. When the broker gets a customer order, it must be given priority over the broker's proprietary trading. This has been in place for decades, well before trading was electronic.

If an HFT (or any other trader, regardless of speed) places an order into the limit order book of an exchange or other trading venue AHEAD of another customer of that exchange, that is not front-running. That is the normal and correct functioning of the markets, which operate by price-time priority.

tl;dr - trading faster or more frequently than other market participants (who are not your brokerage customers) is not front running.

University of Michigan researchers proposed a "centralized call market" to mitigate fragmentation (inconsistent pricing and order handling across different exchanges) and latency disadvantages:

http://www.ns.umich.edu/new/releases/21535-high-frequency-tr...

Haim Bodek discusses how certain exchanges provide special order types that give traders priority over others:

http://online.wsj.com/news/articles/SB1000087239639044398920...

Of course HFT is just a small part of the built-in asymmetries in the markets. Firms exploiting their own clients, conflicted investment advice, corrupt fund management, routinized insider trading, retail bucket shops, hidden fees, payment for order flow, specialist/MM privileges, etc. are all ways that investors and beneficiaries can be disadvantaged.

It's not just insider information that is a problem. Is the world really better off because UBS/hedge-funds hire out satellites to surveil Wal-mart parking lots in order to get an earnings prediction 1 day before Wal-mart files their public earnings disclosure?

Tell me you have a source for this.

http://www.cnbc.com/id/38722872

http://gawker.com/hedge-funds-are-now-poring-over-satellite-...

wasn't necessarily goldman-sachs, changed the post

Skybox is already launching satellites explicitly for this purpose (and others):

http://skybox.com/products#applications

Scroll through to the "Financial Trading Intelligence". If only they had offices in places besides California...

"They are able to identify your desire to buy shares in Microsoft and buy them in front of you and sell them back to you at a higher price," Lewis, whose book is available on Monday, said on the television program "60 Minutes" on Sunday.

I'm guessing the net effect of this is that you get trades for a few cents more than they would cost you otherwise. I guess "otherwise" means a perfect world where Goldman Sachs is a charitable organization run by Mother Teresa directing a band of angels.

This might be an issue for day-traders but if you're a buy and hold investor aiming for doubling or tripling of your holdings over years-decades, this small fractional cost at inception is likely negligible. (Which is not to say that the stock market is a safe place to be - there are many other cons to be aware of if you're a buy and hold person - insider trading, bad quality earnings, etc.).

It was not that long ago that investors were stuck with paying high commissions equivalent to several percentage points to acquire a stock. Now you can get 1000 shares of anything for $5. And maybe a few dollars more to line the pockets of HFT firms. Things seem better to me now on the whole.

I don't think the cost is necessarily negligible.

If the system is setup in such a way that there are those who benefit from every trade, skimming fractions of a percent per trade, those add up. They are effectively shrinking the market capitalization of that stock... and over a long period of time that could be significant.

If a stock's losses to this are in the range of 3% per year, over a 25 year period the stock will lose 50% of it's market cap from this activity.

That's fairly significant.

Dare I say it, but...

"To the tune of Ten of Billions" is that sooo material?

That amount's not unheard of for a single company, hedge fund, hell even individual (there was also that guy that loss 8 Billion).

If it's half of all trading volumne, but a tiny fraction of profit, then this is razor thin margins. In these areas, we've always been comfortable allowing the big boys in any industry (with their economies-of-scale) to execute efficiently, and hopefully compete each other out of ripping to rest of us off too bad.

The 'tens of billions' is also without context of how much money HFT save investors by basically keeping spreads at zero, and nearly always guaranteeing liquidity. Providing liquidity to the market isn't a public service... participants need to be compensated.

"Nearly always" guaranteeing liquidity isn't that useful a service - HFT's liquidity evaporates when it's most needed, like when the markets crash.

On the other hand, there's a very good reason why HFT liquidity has evaporated in previous crashes (such as the 2010 Flash Crash) -- the exchanges broke trades in an unpredictable fashion, so that any attempt at market-making would have opened the market maker to considerable risk.

Let's say you step in during a major market crash and buy AAPL when it's trading at $200/share. Let say it then rises some more to $300/share and you sell, only to see the price recover to $500/share. What happens if the exchange breaks your original buy at $200? Then you end up being short on the way from $300 up to $500, even though you were right about the direction of trading and contributed liquidity during market distress.

They've theoretically instituted a fix for this ( http://en.wikipedia.org/wiki/2010_Flash_Crash#Trading_curb ) which will make it clear in advance what trades will be broken, so we'll have to see how HFT and market making responds in the next crash. Bid-ask spreads tend to widen significantly during market distress, which should actually increase the profitability of being a market maker during a crash.

I think it's been raised here before, but why can't all trades be submitted, kept secret then only executed simultaneously at some discrete interval - say every 5 minutes?

If you have 3 people buying at price x and 2 people selling at price x, which 2 of the 3 buyers get their purchase?

The rules for this are the "matching algorithm". In most markets this is determined in a fifo fashion. Adding a discrete time delay wouldn't change this, there would still be a race to be first into the 5 minute period.

Then randomise rather than FIFO it. Or pro-rata it -- (an option submitted with the offer)

So A) there are lots of markets that do pro-rata matching (larger orders get priority). B) there is a very reasonable bias against non-repeatable matching algorithms in current markets.

If you did randomized matching and 1 market participant seemed to have very good "luck", how could you convince the other market participants that the fix wasn't in? You have to understand that modern markets assume the worst case when it comes to collusion, insider trading, etc. Randomized fills is the worst case scenario given those assumptions.

Perhaps publicise the random numbers used at the same time as the order is executed, together with the identifier for the particular trades in the queue.

Publishing the number would not in any way, alleviate the concern that the system was not fair. You are thinking about this problem from a mathematics stand point when the issue is a social one.

I plead guilty to approaching from a mathematics PoV. My sole intent is to overcome the tyranny of distance and exploitation of the time difference. The only people I hope to convince are other, mathematically sophisticated traders. By social do you mean the public-at-large's suspicion?

1, 1, 1, 1, 1... Can you verify if this sequence is random or not?

OK, I'm lets stipulate that I am not particularly mathematically sophisticated.

However, I thought that you had a list of numbers at least as large as the number of trades expected in the 5 minute period, then you could order the trades by shuffling the numbers. I think it would not be too taxing to determine whether the sequence over time is random enough.

That said, I'm starting to see whether that it would impossible to know whether any particular 5 minute (or maybe day's worth) of trading is random enough. Perhaps crypto can come to the rescue here. Publish lists of random numbers in advance pgp signed.

Anyway, enough of my ramblings :-)

>there is a very reasonable bias against non-repeatable matching algorithms in current markets.

What about psuedo-random that uses the trades themselves as a source of entropy?

Or add a few hundred milliseconds of random jitter to the transaction time. This would remove any advantage to low-latency traders.

Then forget the FIFO convention. How about a hardware random number generator?

However, this doesn't fix the problem with placing and canceling lots of orders to sway the bucket toward the end of the 5-minute window. I suppose if some could do this faster at the end of the window then others, they would still retain an advantage.

(Imagine somebody placing many orders, monitoring how many are likely to be fulfilled advantageously, and then cancelling the disadvantageous ones as close to the end of the window as possible.)

That could perhaps be solved with fees for cancelling bids or asks.

There are already fees attached with canceling large amounts of orders, and most exchanges provide less fees (or even rebates) for providing liquidity (being the order that is taken off the book) vs taking liquidity.

Because it wouldn't reward insiders.

"At an event hosted by the [Chicago Mercantile Exchange] in October, Leo Melamed, one of the pioneers of Chicago's financial futures markets, called Budish's proposal a "nonstarter." He said the markets are working pretty well and there's no need to fix something that isn't broken.

Otherwise, Melamed said jokingly, "I'm going to consider going to North Korea and show them how they can become the center of finance in about three weeks."

So yeah this has been discussed and no, the powers that be like things the way they are. Go figure. This will need stronger powers to force the hand of the traders.

http://www.chicagotribune.com/business/ct-high-speed-trading...

You could hardly expect the people profiting from this to do anything except come out against it.

Its amazing how similar this line of reasoning sounds to the development of democracy mode in Twitch Plays Pokemon.

This is available on exchanges today. It usually happens 2 - 5 times per day.

I don't understand how HFT can make me pay more than I want to for a stock. I set a limit order--the limit is how much I'm willing to pay. If some HFT firm pushes the price past my limit, then I don't buy.

This is true, but I believe if you don't set a limit order, then the market value will go up. Then if the price keeps going up, you'll have missed your chance to buy the rest.

It can make you pay more than you otherwise would, not more than your limit.

Just because you have a limit order doesn't mean you aren't getting fleeced, as you could've bought it for less otherwise (assuming some level of fixing as is suggested)

You still lose. In your scenario, you are blocked out of buying the stock. And without a limit order, you get the stock but pay more.

Imagine this happened when ordering food. Sure, you could put in a limit (no more than x french fries and y per fry) but your price would either be run up or you'd go hungry.

But I'm not buying food, I'm buying a stock--which, unlike food is a totally optional purchase.

My limit is the price at which I thought that stock was worth buying. If the price never falls that far, then I wouldn't want to buy it anyway.

HFT is manipulating the pricing so that it is no longer worth it for you to buy it, therefore fleecing you of the profits you would have made if they weren't there.

Of course, this is not illegal (it's like any other type of trading, pushing up the value so it goes beyond what you're willing to pay) - but it's achieved by using robots in such a way that normal humans can't hope to compete.

Of course, you can build your own robots (ie your own HFT software) to take advantage of the weaknesses of existing HFT programs, but if you can do this, you would be working at an ibank as a quant of some kind.

Can you actually describe this market manipulation?

Yes, but what the parent poster described is moronic for an HFT to do. If they see a limit order come in, there is no incentive to push the price beyond the limit order. Why would that help them in any way? There is no implication that the person issuing the limit order will suddenly decide to pay more.

Additionally, to push the price up in the first place, it means the HFTs have to buy all of the stock being sold below the target price, which is taking on major risk. There is no guarantee that there isn't someone simultaneously dumping big chunks of shares at given price targets.

> HFT is manipulating the pricing so that it is no longer worth it for you to buy it, therefore fleecing you of the profits you would have made if they weren't there.

This makes no sense to me. Are you claiming that HFT purposefully drives up the value of every single stock? How would they even do that? And why?

HFT pushes incremental buy orders, which your sell order accepts (price discovery) but are canceled before they are filled. Knowing your limit, they then fill your order at your limit.

It's illegal for anyone to issue an order they don't intend to complete, but the HFTs get a pass for some reason or other that must end with quite a few zeros.

HFT pushes incremental buy orders, which your sell order accepts (price discovery) but are canceled before they are filled.

This is incorrect. If you place a BUY@$100 and someone has a SELL@$100, the exchange instantaneously matches the orders and a trade is issued. You can't cancel an order which was matched.

The "price discovery" happens before the trade is placed. The instant the exchange receives the BUY@$100 order it is published to the world via the exchange's multicast quote stream.

"Issue an order they don't intend to complete" is an anachronism related to pit trading - in the pit, it's possible to throw the "I accept" hand signal at the other guy and then change your mind after trading hours during reconciliation. The FIX and OUCH protocols do not have a similar "oops I changed my mind now I'll be a jerk after hours" message.

It's called quote stuffing:

http://www.zerohedge.com/article/its-not-market-its-hft-crop...

Nanex has discovered some wiggles in a graph. So what?

What they discovered (as have others) are innumerable canceled quotes, which are the mechanism for price discovery. A bit more than wiggles on a graph.

I don't understand. The order was valid and could be filled until it was canceled. Can you please explain the specific mechanics of what you think is happening, who it harms and how?

The top post in this thread did so clearly.

The top post is based on a completely incorrect understanding of how a matching engine works.

You don't understand the concept of quote stuffing?

What I understand.

1) Rapidly post/cancel trades, risking a fill. 2) ??? 3) Profit!

The ??? represent the part I don't understand.

(If you are actually interested, "quote stuffing" is typically the result of algorithms interacting in unexpected ways. It's not intentional or beneficial. I've spent weeks trying to reduce the amount of it.)

IIRC quote stuffing is an attempt to trip triggers to the benefit of the HFT. Basically you have a trigger that says when the market hits $X, do Y action (say sell). Since your trigger is slower than their processing, assuming a small enough market, they can "move" the market to $X but reverse before your trigger finishes.

This only works because you don't actually have a "Sell at $X".

In the simplest of cases, let's say a stock you are following is at 50.50. You want to buy it at anything under $50. It drops to $49.99, and someone cuts in front of you in line, and buys it. You miss the purchase, and the stock goes back up.

Multiple this millions of times.

This isn't to say they are all completely evil. They do provide liquidity as banks have walked away from equity. But there needs to be more transparency on what is going on. (For example - should it be legal to send false buy or sell signals which will be intentionally cancelled?)

The false buy/sell signsls are a legitimate issue. Given that you can't easily make it illegal to cancel a limit order (after all people give up after a while) you could require that the order has to be valid for 60 seconds before it can be cancelled and that would put a huge crimp in the 'faking it' market.

A rule like this would cause every market marker to set their bid/ask spread wide enough to cover the risk of the typical adverse price move which could occur over 60 seconds.

You've hit on the challenge. Any hard and fast room either has unintended consequences, or can be worked around.

There is no "cut in front of you" message in FIX or OUCH. Most exchanges use price/time priority - if someone "cuts in front of you" it means they offered $50.01.

Or maybe "cuts in front of you" means you wanted to buy the stock at $50, but you didn't actually place the order until the last minute because you wanted to play a speed game (and lost).

How dare you use facts to back up your case in a HFT forum discussion?

This comment is a substanceless one-liner. We don't want any of these on Hacker News. Such comments are especially bad when they're sarcastic.

I'm not picking on one commenter here. Many HNers post substanceless one-liners. Please don't do that. Re-read what you've posted and, if it adds little of substance to the discussion, delete it. Fewer substanceless comments will mean a higher signal/noise ratio for all of us.

Edit: someone pointed out to me that kasey_junk is one of the few people who actually knows what they're talking about in this thread, meaning that he/she has contributed much more signal than noise. That's probably true! I don't have time to read all the threads, even the interesting ones—especially the interesting ones. I'm sure kasey_junk couldn't care less, but I've restored the karma that his/her account lost to downvotes here.

Keep in mind that HN is going through a period of experimentation while we figure out how to address some of the longstanding problems that PG never had time to take care of. We're going to get a lot of things wrong on our path to getting things right. We're also very interested in your feedback on these experiments, so don't think the feedback is one-way. The best channel for it is hn@ycombinator.com.

One way HFTs do this is to place thousands of tiny orders while trying to detect the presence of a large trade being put through (known as a 'whale'). They use pattern recognition software to detect a large, price-insensitive seller or buyer and bid up (or down) accordingly.

Another is to flood the exchange with orders in a bid to create congestion and slow down other market participants who might intervene.

This is why HFT in fact causes higher transaction costs on market participants. It provides the perfect way to leech.

"One way HFTs do this is to place thousands of tiny orders while trying to detect the presence of a large trade being put through (known as a 'whale'). They use pattern recognition software to detect a large, price-insensitive seller or buyer and bid up (or down) accordingly."

A) On every exchange that I know of, the only way that using tiny orders gives you any information advantage is that on certain exchanges fill information is processed faster than market updates. Nothing in that fill information would allow you to determine if there is a large price insensitive market participant in the market vs lots of small price sensitive participants.

B) Nearly every exchange out there allows large participants a variety of mechanisms to hide their order flow. The simplest is iceberg orders.

C) It is a feature, not a bug that prices change in the face of large order flow. Every market in the world, electronic or otherwise works on this basic assumption.

Finally, your example about spamming quotes to create congestion on matching engines is a problem that has already been resolved. The exchanges themselves had a very high incentive to fix this. Their solution was to introduce fill ratios and to fine folks aggressively for violating them. I don't know of any main line exchanges where quote spamming can provide a legitimate latency advantage still.

Regarding A), I think it's about finding icebergs and pinging dark pools.

> It drops to $49.99, and someone cuts in front of you in line

Um, how do they do that, exactly? The major equity exchanges are all FIFO on every tick price. If you put a limit order in for $49.99 before somebody else, and an offer crosses your bid, you'll get filled before they do, period.

Let's say you want 100 shares paying no more than $49.99. Your screen shows the following sell orders:

Exchange A: 25 @ 49.99 Exchange B: 20 @ 49.98 Exchange C: 60 @ 49.97

You expect your order to get filled with all shares from C and B and just 20 from A at the highest price, the problem is Exchange A is closest to you, B is 50 miles away, and C is 800 miles away in Chicago.

The HFT bot sees your order arriving at Exchange A, since they have a fastest link to B and C, they buy the orders there and put them on sell to you at $49.99 just a few milliseconds before your order arrives.

This is just a problem with my broker lacking the technical sophistication to synchronize the transmission of my orders in such a way as to ensure the best execution. Whether it's worth it for me to switch to a broker with that capability probably depends on how much volume I trade.

Either way, this problem can be solved much more simply by the application of some simple order timing algorithms than by regulations attempting, in effect, to repeal the speed of light.

Yes, but it's kind of stupid that they could get away with it in the first place since most people do not have collocation with the exchanges.

If you put in an order for $49.99, that is the price you said you are willing to pay. It is not fleecing you if someone meets the demand you express.

If you really wanted to pay $49.97 then you should have put in that order instead.

It doesn't.

If you watched 60 minutes tonight, or read the book, the problem was that a large trader (the guy they profiled ran the trading desk for RBC) would make an order for a large block of stock, say 400 shares of X, and then send that out to all of the exchanges.

What would happen is it would get to the first exchange and be partly filled and when it was the HFT guys would send trades on ahead to the other exchanges faster than RBC could and buy out the stock before the RBC order got there, leaving the order only partially filled. (or filled part at one price and filled the rest at a higher price).

In the story they mention that by restructuring how they sent their trades out so that they arrived at every exchange simultaneously eliminated the possibility of the HFT guys front running them.

In order to make it possible for everyone to do that they have created their own exchange (iEX) which locks out the HFT folks (well it has them going through an extra 60km of fiber to slow down their access).

His issue is with market fragmentation then, not HFT. If all the liquidity providers were at the same market, his large order would hit all of the liquidity at the same time. This is a valid complaint to me, but blaming the HFTs for "front-running" (if you could even call this that) is a bit like UPS complaining that FedEx is using airplanes to quickly move packages around. Keep up with the technology or stop competing.

No, his issue is with HFT companies using fragmentation to trade risk free and skim obscene profits off the top from institutions that make these large trades. From the 60 minutes piece, one hedge fund estimated that this kind of thing was costing them $300m/yr.

Yet, I for the life of me cannot describe how they do this. Can you write out the basic trade that HFT companies use to get all this risk free profit?

exchange A has a bid at 99, offer at 100 exchange B has a bid at 99, offer at 100

mutual fund X places two market orders to buy and take out the current offers, one on A and one on B. Its order on A executes at $100, taking out the liquidity and making the new best offer there $100.01

HFT firm Y sees this execution and cancels their offer of $100 on exchange B and replaces it with one at $100.01 all before the order that was placed on B arrives.

This is a market fragmentation issue, Reg NMS tries to address this but doesn't really work at these timescales, because you only have to comply based on the current quotes you are getting from the exchanges, which could be many milliseconds behind (and if your system is slow to process the quotes and makes more money as a result, you won't be getting a task from your boss to speed it up).

http://en.wikipedia.org/wiki/Regulation_NMS

There are many other types of useless latency arbitrage that don't depend on market fragmentation, like trading on the difference in price between ETFs (basically a basket of stocks usually trying to match something like the makeup of the entire S&P) and the underlying individual stocks. Each time the underlyings change, the ETF price changes (within reason, risk hedging activity etc. could make the ETF temporarily get out of sync, but that is just another latency arbitrage oppurtunity). Whoever is faster wins.

It's pointless, there are datacenters full of servers running wait loops on incoming data via DMA so that they don't have to pay the latency price of a system call and context switch to the kernel. Basically burning up part of that mutual fund's money into waste heat.

What you describe as "useless" latency arbitrage sounds like the mechanisms for a very efficient market to me.

As a buyer of an ETF or as a market participant looking for trading venues I needn't spend lots of resources ensuring that the prices are in sync. Those costs have been born by the specialists in those arbitrage opportunities and it is a very cut throat efficient system.

Take out the arbitragers and all the other participants as a whole keep more money. Spreads may increase, but one party to the transactions always earns the spread and the other loses it, netting out to zero, rather than compensating latency arbitragers (and paying to generate waste heat in their datacenters) which nets out to less than zero.

There is some value to keeping things liquid, but that value isn't tied at all to the amount of money that arbitragers get compensated. Consider that whether the latency is 1ms or 5ms, the fastest arbitrager wins. But to bring things down to 1ms from 5ms may require 10X more expense and waste, lowering the compensation to the arbitrager. In a competitive enough environment they simply burn up all of the spreads into waste heat and redundant fiber infrastructure.

I can think of at least a couple of ways.

In an earlier thread someone used the term "legalized front running" to describe HFT, and in thread the final pieces fell into place.

Fact 1: we know that HFT companies have special hardware in place. They have effectively built FPGA's (or even ASICs) that can process incoming orders while the message is still partially on the wire. This allows them to react to new orders and executions with a latency that is below the transmission delay of the protocol packet.

Fact 2: HFT companies have built directional radio links to get their messages faster between exchanges. Speed of light is still a limiting factor, and the distance traveled between exchange A to exchange B is somewhat longer along the fiber.

Fact 3: The basic rule of trading is "buy low, sell high".

So, if you can process order information faster than than it takes to receive the entire exchange message, you can use that as an advantage against the rest of the market. The HFT systems know the current price for a stock in the system (the lowest standing price), and when they receive an execution, they know that someone else just traded at that price. They can send out a message to other exchanges to quickly buy the said stock at the said price and put out offers at a slightly higher price.

The net effect? "I just bought all of the stocks under your nose. Here's what I'm willing to sell them for you."

As long as the price they're selling the stock out is within the limit, they'll get an immediate risk-free trade. The HFT systems already had a buyer ready before making their orders.

[1]: http://ra.ziti.uni-heidelberg.de/cms/images/pdfs/High-freque...

This is called (spacial) arbitrage and this strategy has been around for a long time, well before HFT.

> this strategy has been around for a long time, well before HFT.

I have no doubt about that. Very few things are new.

But if you're deploying special hardware in an attempt to cheat the speed of light and gain an advantage of mere microseconds, the system is FUBAR (or recognition).

Watch the 60 minutes piece, they describe it in layman's terms very well. The basic gist is that for any large order, no single exchange will be able to fill it entirely. So exchanges will partially fill the order and let other exchanges fill the rest. What HFT firms are doing is registering the partial fill on one exchange and then using their direct fiber lines to the other exchanges to buy shares before the original order gets there to fill the remaining portion.

It's risk-free arbitrage...exploiting the difference in price between two exchanges that will exist for only a fraction of a second knowing that the rest of the order will arrive fractions of a second after they trade.

> skim obscene profits off the top from institutions that make these large trades

It's not like in the pre-HFT days market makers would just obligingly fill institutions for whatever volume trades they wanted to execute, regardless of how ham-handedly they broadcast their intent into the market. Any estimate of "costs" to institutions from HFT needs to be measured not against some Platonic ideal scenario where market makers earn no profits, but rather against the old specialist system that it replaced. In which case I suspect the costs would come out quite a bit on the negative side of $0.

There are some practices in HFT like paying for privileged newswire access that could use some reform, but the best solution to this fragmentation issue is for large institutions to stop sending out their orders like it's the year 1995.

It certainly sounds like there's a large dollar value that these 'large institutions' could capture by getting better order routing for their orders.

Basically, certain big buyers/sellers of stock want to be the ones to benefit from the information "Someone really wants into or out of this stock", and they really, really dislike when the HFTs democratize that information advantage faster than the larger players can gain extra returns from using it.

Trading communicates information to the market. Proprietary access to information is valuable. HFTs make certain information non-proprietary faster than the state of the art prior to HFTs. That changes the allocation of profits associated with that information. This is the entire ballgame. Retail traders swim placidly on the sea above totally ignorant of the krakens vs. mutant-sharks-with-lasers bloodbath happening 20000 leagues below.

Actually limit orders are a perfect way for HF Traders to fleece regular investors...for instance if you set a stop loss at a certain level and a HFT causes the price of a stock to drop precipitously, you'll end up selling likely at a loss for an artificial reason.

This scenario used to play out a lot during the 2008 financial crisis. I remember loosing money because of such tricks.

If you do not intend to sell a stock for $40 then it is, in general, not advisable to tell your broker "Sell this stock at $40."

Not sure what you mean, but you do a stop loss to make sure you don't lose your shirt when a stock moves against you.

No one intends to sell at a loss but that is part of the game. The problem is when market manipulators artificially cause the price to move against you thus triggering your stop loss.

If you don't want to sell your stock at a price below $X (for any value of $X), you should not tell your broker to sell below $X.

If you want a mechanism to retain upside while limiting downside, it exists. It's a call option, not a stop loss order. If you don't understand the difference you should not trade.

If you could explain the specific mechanics of what you believe is "market manipulation", it would be very helpful.

A stop loss is not a limit order, at least in finance terms.

I am using that to mean the various order rules for capping trading losses/gains...I know the finance jargon meaning is slightly different.

How it makes you pay more for a stock than your limit order works as follows. You see XYZ is trading for $49.99 and put in an order to buy 10,000 limit $50/share. Your trading platform replies you have bought 4000 at $50, the price has moved to $50.50 you have a 6000 share order waiting to be filled. You then say darn it I'll wait for it to go back to $50 and it goes up more and so on. As has happened to me many a time. Though as a trader you can play the game too and offer to sell shares at a high price if it looks like someone needs to buy. This stuff is one reason why paper trading gives a false sense of how easy trading is as historical prices don't actively try to figure what you're up to and screw you.

I thought that what happened was that the HFT players were able to find your limit and so front run your order.

My understanding is that if the price was initially lower than your limit, they can buy it at that price before you can, and then sell it at your limit.

To give you an idea on how HFT has blown up since 2007: http://www.technologyreview.com/sites/default/files/legacy/d...

Context: http://www.technologyreview.com/view/428756/watch-high-speed...

I have always felt the markets are rigged, but by insider trading. High frequency traders just add to the problem. I thought the Internet, and websites like Stocktwits might level out the playing field, but the I've Never seen anything close to insider information on any of these sites. I truely believe we are on a bubble right now--especially tech stocks(Tesla might be the exception, even with the owner telling investors the stock is overpriced.) Facebook throwing around billions for for that disappearing picture company floored me. Oh, and Google--the minute Duckduckgo gets a little bit better--bye--bye, but I'll use your free Api's as long as they stay somewhat free. My deceased father spent years dabbling in the stock macket when he retired; the only real money he made was on a tip he got from a drunk, bolsterious father in law who was a CEO. Oh yea, this brings me to Jim Cramer. I have watched him for years. He has never timed a down turn right. He just might be the best ironic contrarian stock adviser ever? Right now, he believes the stock market will just continue to rise. I am waiting for the drop. We are in a weird time though; gold at unheard of levels. Stock market rising. The only real inflation I see is food. I have a funny feeling the economy is much worse than it seems though. Companies know we cannot go without eating--so let's raise the price. I'm a nobody, but I hate to see middle class families loose their life savings when the market takes a dive. I hope these lucky tech billionaires help the American poor when things get really bad? There are very few countries that would have allowed them to Mae there wad as easily as they had it in America. Could you imagine having a billion dollars in Mexico? You couldn't leave your home. That's the one thing they take for granted, while griping about tax rates? Sorry for rambling.

FYI, all Michael Lewis books are must-read for those interested in Finance and especially useful to understand some important Wall Street psychological features and possibly explain why some bubbles have gone bust in the past decade. These books are also technically pretty sound. In order of personal preference: The Big Short, Liar's Poker and Boomerang. Lowenstein on LTCM is also good in the same verge with a similar technical level as Flash Boy seems to be but for 1990s Arbitrage Trading.

There are a many of news articles and youtube videos which present HFT as a bad thing.

Since large orders are filled by different markets, anyone who has studied the mechanisms via which these orders are full-filled can theoretically arrive at a strategy that might be profitable. Overtime, more and more people will learn these things and it will no longer be a profitable strategy to implement - this is sort of like the "January Effect" described in the book named intelligent investor.

I don't understand why HFT has to be blamed for a difference in prices across multiple markets. I read somewhere that REGNMS caused market fragmentation and HFT traders spend time studying the current markets and identify inefficiencies that can be used to make profitable trades.

This Reuters article is like an advertisement for the IEX trading platform. IEX figured out a certain way to beat the other algorithms and they told Reuters about it. Now that everyone knows it, I think, pretty soon other algorithm developers will modify their implementations.

Overall, I think that HFT is basically algorithmic trading, but which needs more investment (for renting/buying server/connectivity) and more time/interest (for studying and understanding the order placement/fulfilment mechanisms involved).

It's not that HFT is to blame for differences between markets, but that differences between markets plus the market structure are to blame for (the socially wasteful bits of) HFT.

>I don't understand why HFT has to be blamed for a difference in prices across multiple markets.

Because it isn't.

It's good this is getting more attention, however there were many good books before this one. "Quants" and "Dark Pools" were my favorites and a good place to start.

The Physics of Wall Street: A Brief History of Predicting the Unpredictable

Automate This

The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed It

Dark Pools: High-Speed Traders, A.I. Bandits, and the Threat to the Global Financial System

Hedge Fund Market Wizards

To me this does not sound "rigged". It sounds like one participant in the market feels entitled to the profit another market participant makes.

One issue with most criticism of financial markets is that people tend to view only the price, and may be the transaction cost. What they completely fail to see is risk management and liquidity. Both have intrinsic value, and people are selling and buying liquidity and risk management. This concept is hard to grasp. But the HFT traders are providing a value to the market.

Restricting HFT will undoubtedly lower liquidity, and maybe even increase risk. That's why regulators are so anxious about it.

The HFT traders are indeed taking on some amount of risk, which otherwise would be with the stock exchange or the other participants. The salient question is whether or not the competition among HFTs is hard enough for the Bertrand paradox to kick in.

OK, then why would investors actually switch to doing business with iEX if HFT brings any value at all? Looks to me like HFT is a like 'free' games that 'offer' to buy in-game items which are needed to play the game properly. In this case, investing is the game and if you don`t have a billion dollar fiber optic running, you play at a net disadvantage.

"a billion dollar fiber optic" has to come from someplace. Money costs money, which is the basic tenet of finance. If an HFT company invests a few hundred million Dollars into the trading system, the profit better beat the market. That's why I think that there are probably enough competitors in the HFT sector to squeeze the profits.

The investors who "switch to iEX" are money managers who have to place orders to diversify a portfolio. Without the new strategy they essentially pay some extra money to intermediary traders. The value of intermediary traders is hard to grasp. They do carry the risk inherent in their positions, and they provide liquidity by bridging the time-gap between the intention to buy and the intention to sell.

iEX basically seems to offer their clients the chance to bridge this gap themselves by cleverly dispersing orders over different markets and obfuscating their intentions.

Finance is a fascinating technology in itself...

Group the requests by second and randomize their order. Boom.

Bogle has a great book on how horrible hft is for our economy (well, he has several technically) called The Clash of Cultures... Reading it now and am fascinated. Could be that this is my first foray into learning about finance, but either way, I highly recommend checking it out.

Thanks for the recommendation. (Why did someone down vote you??)

The story states that IEX runs HFT trades through an additional 60km of cable. Does anyone know or have a source for how this specific number was chosen? Who decides whose orders have to go through those extra 60km and whose don't?

As Mark Cuban might ask, "Can you spot the sucker in the stock market?"

This caught my eye "travelled along fibre optic lines and hit the closest exchange first, where high frequency traders would get a glimpse, and then use their speed advantage to beat him to the other 12 US public exchanges and 45 private trading venues."

Why not simple stagger the network dispatch to those exchanges so the order hits all exchanges at exactly the same time and with that protect you from HFT snipping tactics. Seem simple solution and somewhat obvious so what am I missing?

Someone's trying to sell a book.

Interesting article - I'd only seen high speed trading discussed rather abstractly before. Katsuyama's "Essentially, our fill rates went to 100 percent. We couldn't believe it when we actually figured it out," is interesting experimental data on how it's actually working. I wish him luck with the IEX thing.

I never buy/sell at market price, always at limit.

Am I still affected by HFT in a meaningful way?

If not, perhaps the every small-time investor should be defaulted to a limit buy/sell unless they choose otherwise.

Yes if you've ever had an order not get completely filled... happens to me a lot, it's kind of annoying and usually not worth chasing the price up.

Every small investor should use limit orders. Market orders are rigged at the Market Maker levels via a process called Slippage. "Slippage occurs when a market maker changes the spread to his advantage on market orders." basically they stretch the spread out forcing you to pay a higher or get a lower price for the stock to their advantage since they fill the quantity from their own volume.

That is the entire game of market making. You are literally complaining about market makers doing their jobs.

As a market maker, I might sit out there offering to buy 1000 shares of Microsoft at $40.20 and sell 1000 shares at $40.30. Then someone comes along and buys 1000 shares from me at $40.30, so I need to put a new quote into the market. Do I put it in at $40.30 again?

Hell no! There are two main reasons why not -

1. I know that there are buyers out there, and most likely the reason there are buyers at $40.30 is because they believe the stock is worth more than that. I need to revise my prices upward to reflect that new information.

2. I'm now short 1000 shares of MSFT. If the price goes up further, I will lose out. I need to persuade someone to sell me 1000 shares to cover that short position, so I need to offer more money - therefore I need to raise my quotes.

Most likely is that I'd now put in a new offer at $40.32 and raise my bid from $40.20 to $40.22.

This is the mechanism by which market prices reflect new information. Market makers change their prices to reflect information in the order flow. This is the entire reason that liquid, efficient markets are able to exist.

All time stock market highs, inflated stock prices due to HFT bidding up prices and thus siphoning off money by the billions, companies operating at a loss or minimal profits getting billion dollar valuations, it's been a while since the last major stock market drop hasn't it?.. Is it just me, or are we playing with fire here? It seems like if one thing goes wrong the whole house of cards will come crumbling down.

I'm worried about my retirement after reading more about HFT. Where should I put my 401k to avoid losing a big chunk of it as soon as the market takes its next drop?

I see much of propositions for solving problem with HFT. Mine? "Random delay for signal (between 100-200ms) on network devices", problem solved. Book "Dark Pools" describes how HFT works 'behind the scenes', great pice of journalism what's more: people rarly point out, how big drain on resource HFT is, bilions of dollars spended on fiber lines between continents, to gain a few miliseconds, this is madness.

great. here is the new exchange they are talking about: http://www.iextrading.com/

next step: build an investor exchange opensource based on bitcoin. take over the world. if you're into this kind of thing let me know.

> build an investor exchange opensource based on bitcoin

Can you elaborate?

All stock markets are so far removed from actual useful endeavour that it's hard to see what their function might be.

I think we need some human-friendly reform in this area.

The HFTs are doing something which reminds me of the mutant attack with Bitcoin. And the 'fix' is also very similar to how you can try to defeat mutants, if not by making mutation impossible in the protocol (which we are trying to do), but also making it more difficult to beat the propagation of the original transaction with mutants; by transmitting it to more nodes concurrently.

Unlike the mutant attack, in this case there are possible positive effects that can arise.

But what I don't understand is this. I place a human order to buy 1,000 shares of X for price P through a public exchange. The HFT algorithm can get those shares for (P - a) on a private order book I don't see, before my order even arrives there.

But from the perspective of the seller(s) on the private book, they had open sell orders with weighted average (P - a) for those 1,000 shares. That's open orders, a.k.a willing sellers. As long as there are real shares being bought by the HFT before the sale to the original buyer, then how can you complain?

But what I really want is to be able to place these cross-exchange orders with something like a 2 phase commit. Get a 'lock' on the public buy and the corresponding private sell(s) before having to commit. And then clear the committed transactions concurrently on both exchanges.

That's a risk-free trade, and the result you might expect is the spread is shared between buyer and seller. Maybe some of it goes to transaction fees.

And that's how I look at the HFT algos. They are basically a hidden transaction fee on a very fast routing of your order. There super-order-routers charge a transaction fee which is applied by tweaking the listed price. Over time the amount of the tweak decreases until it exactly equals your stated price, which means the super-order-router gets no fee.

If you had [paid for] access to that other order book, you could have spent less / earned more on the trade. But there's the added cost of having a direct connection to that order book. So basically you buy it yourself, or someone else pays for it and acts like an automatic passive router. In the end you get exactly the price that you asked for.

I wonder how often the HFT algorithm is selling shares that it doesn't really own. No one should have been surprised that MtGox wasn't running full reserve. Who's publishing the crypto-proof that the public exchanges are running full reserve? Haha, actually we know that they don't - they just call it naked shorting. So I guess I'm asking, how much naked shorting (even over brief time periods) do the HFT algos get away with?

What order book are you talking about?

And any broker dealer can sell shares they don't own...Because they are a broker dealer. Call your congressman.

So this amounts to some amount of free float for the broker dealer. How much are we talking in terms of $-days?

If they can sell and cover whenever they want, then yes it does start to sound like a 'print money' button.

You can get a report on this everyday from the various exchanges.

WHAT????!?!!?!?!?! Oh my god...

The criticism I've heard of this is that HFT only extracts at most 2-4 billion from the economy each year. Is that the case? If so, I find the arguments against to be kind of trivial.

Hmm... it's almost as if brokers break up large orders into smaller orders for exactly this reason.

Stock markets aren't rigged, they're fundamentally flawed. HFT is only a small modern flaw that's emerged.

In theory a company can float itself in order to raise capital and improve performance. The business will invest the capital, and in turn return some of the improved profits to the share holders.

In a modern context it simply can't work this way. In a world with fiat currencies, any organisation with securities can directly invest in an stocks/bonds higher with a return higher than the Bank Rate to an extent far greater than individuals, with less risk.

This leaves individuals with little to no capacity to get a decent return without brokerage. While possible to make money, it's a roll of the die that would be better of done in a casino.

The bigger story of HFT is that it enables the big banks to manipulate the stock market. Because they know all market orders before they are executed, they are able to cancel all bids and offers before it is possible for anyone to hit or take them. They can also put up huge orders on the bid or offer to make a stock look weak or strong in the short run, knowing there is no risk of being executed. If they see another order come in they can just cancel their order. The most criminal advantage HFT gives them is that they can front run orders. Let's say for example that Fidelity calls Goldman Sachs and says they want to buy 1 million shares of Tesla. This information is priceless as Tesla stock is guaranteed to fly higher on this trade. Acting on this information is the most blatant form of insider trading that exists. However, somehow the big banks can create programs that do exactly this. Whenever they see a huge influx of buy or sell volume they can quickly jump in front and close out the transaction a few seconds later for a huge profit.

What you describe above would be highly illegal.

That's the whole point...Michael Lewis said he can't believe it hasn't been made illegal....5 years ago, during the financial crisis there was a good story on it in the New York Time and one of the Senators said they were outraged and he was going to change it, etc, and of course nothing ever happened.

Are you saying that Goldman Sachs's brokerage division is transmitting information about its customers' orders to the GS prop trading desks before submitting them to the market?

No, I'm saying Goldman Sachs has computers located at the stock exchange that are programmed to trade based on order flow information.

So just like dozens of other firms?

Why wouldn't they be? They have an obligation to make money and the profits would be far greater than the fines even if they weren't running a revolving door with their regulators.

How does the HFT know Fidelity called Goldman Sachs? Are they tapping the phones?

Whenever they see a huge influx of buy or sell volume they can quickly jump in front...

Could you tell me what FIX/OUCH command is used to do this?

I was drawing a real life comparison to how HFT works...it gives the big banks order information before it is executed.

As for actual commands, sorry I am not a programmer, just a former trader.

So you were involved in illegal activity? Maybe you should call the SEC and pickup the whistleblower percentage?

>Whenever they see a huge influx of buy or sell volume they can quickly jump in front... > >Could you tell me what FIX/OUCH command is used to do this?

Uh, buy and sell.

Could you tell me which part of FIX/OUCH makes quote stuffing impossible?

Buy != "jump in front".

Can you point me to the order type or flag that allows me to jump? How can they front run? Do you even know what front running is?

They can front run because they receive all of the order flow decimals of a second before the orders are executed. Let's say for example you saw (or your computer told you) that X market orders for Y shares to buy a stock just entered the market...your program tells you this is a lot of volume for this stock and it should push up the stock. Before any of the orders are executed your algorithm jumps in front and buys the stock and sells it a second later for a nice profit. This is a very simplistic example.

shakes head If you send a martketable order the first any colocated participant (HFT or not) will see is your order crossing a number of order on the other side of the order book. If you send a market order direct to NASDAQ for GOOG then you will cross with the best price with best time priority shares of GOOG. You are probably referring to internalization which is a function of your broker.

Reply on news.ycombinator.com