Someone got the natural gas report 400 ms early

this isn't the same as being 70% of price movement. HFT is comprised mostly of market makers, who have books that, over the course of the day, are close to net zero. they do a lot of buys, but they also do a lot of sells.

most price moves over the course of a day are actually driven by people who take positional views, and buy or sell large positions. so actually, we're not really that far from kansas

This is only looking at the volume on public exchanges. There are a lot of regulations around public exchanges which prevent them from operating efficiently. For instance, unless the stock has a very small price, you cannot offer sub-penny prices on this exchange.

Most retail trades actually never see the exchange, they are sold in bulk by brokers to places like Knight or Getco who internalize the order flow. They cross customer orders and take on some orders. Since this does not happen on a public exchange, they can give sub-penny price improvements.

The same happen with dark pools where many hedge funds will send their trade to obtain better executions.

When the market makers who handle these order flow start carrying to much risk on their book, or if they don't want to take the opposite side of your trade, they send it to the stock exchange.

This means that the stock exchange is mostly a place where high-frequency traders meet to offset their exposures to one another. In this respect, it is not surprising, nor problematic that 70% of the volume come from HFT.

The ones that say Apple has the vision and skill to enter and exploit many more massive untapped markets as we've seen them do repeatedly compared to GM's fairly static markets?

Oh, wait...

Their P/E ratios are very similar. Why does that surprise you? Apple is a very mature company. They're currently making huge earnings (which naturally lowers the P/E ratio if those earnings are not expected to continue at that level), and they're a big long-term risk because nobody knows exactly what's going to happen with Android vs. iOS. It's extremely plausible that margins in the smart phone and tablet markets will take a dive in the medium term as a result of vigorous competition, which is where Apple derives the bulk of their profits. Expecting Apple to be doing five years from now as well as they've done for the past five years is to expect them to come out with something new which is as revolutionary as the original iPhone. Maybe they will, but the market obviously isn't betting on that happening.

On the other hand, GM is not doing great earnings wise, but there is no obvious reason to expect that their existing customers are going to evaporate, or that their margins are going to change significantly from what they already are.

Couple of things:

1. Reference to Apple PE ratio is likely a market driven phenomenon that has occurred before to the other technology company that grew very large very fast (MSFT) ie. sometime around 2000, most mutual/institutional investment funds literally owned more of MSFT (and now they likely do of AAPL) than they were legally allowed to own. At this cap, given that these buyers are the largest "long term" drivers of a stock's directionality, the stock must change direction. The second part of this effect is that now a bunch of them are underwater, and the psychology of holding a bad trade will affect whether they decide to book the loss (likely they wont for a while). TLDR: Apple is simply too large, relative to the tech sector, for its stock price growth to match its business fundamentals.

2. Re: markets functioning properly - we should remember that the "markets" are literally a construct. For all the logical arguments made about how HFT reduce the bid/offer spreads, I'm philosophically opposed to them. Mark Cuban has articulated why better than I can: http://blogmaverick.com/2010/05/09/what-business-is-wall-str...

In a nutshell, if we constructed the markets fundamentally to make it easier for businesses in the real economy to raise and price capital, and HFT starts to account for a multiple of that, then the purpose of the "market" construct has been hijacked.

This leads to all sorts of gnarly questions about how to decide what the right volume is etc, so I recognize its'a thorny area - just pointing out that HFT is not so benign, and oftentime comes with consequences that far outweigh the benefit, and happen too often to ignore.

This is a data structure problem at its core - the fact that someone entered one billion BUY orders at $15.51 should not prevent me from seeing that there's an outstanding order at $15.52. The stock market employs the queue only for legacy reasons.

Let Px(t) be the price at time t. Px(t+δ) given Px(t), i.e. P[Px(t+δ)|Px(t)] is D = Gaussian(Px[t], vol) in an efficient market. We know relevant information is going to be added to the market at t+δ, but we do not precisely know its content until t+δ. The market can, however, make an educated guess. This information is a kernel with a central tendency* and an entropy, i.e. second moment, H. The price at t+δ is now D' = Gaussian(Px[t], vol + Hω), where ω is the weight of the information. Since H and ω are both positive we can see that D' is platykurtic with regards to D. D'-D, the impact of knowing that salient information will hit the market at a certain time, looks like a short butterfly P&L.

Thanks for the book recommendation; I enjoy Kahneman's work.

*If ω were very low or H very high, e.g. a unitary distributed kernel would have infinite Shannon's entropy, you are correct in assuming that it would have no impact on D', i.e. D'-D would be zero.

The ad industry's efforts have led to online privacy being all but non-existent, but Wall Street has turned our economies into shit and saddled the world with massive amounts of debt that will never be repaid.

But you can consider the ad industry far more "innocent" than Wall Street. The latter has known exactly what it's doing and causing all along.

Doing so doesn't give them a sensationalist headline that finds its way to the top of HN.

This would cut the advantage those hardwired trading machines have, that do tons of trades in few seconds or even milliseconds.

This should be one of the reforms against such things, in the interest of common folk investors.

Absolutely. I'm just saying there isn't much justification to assume it was a mere mistake.

Particularly given the trades that show up just after the report was released, but before a person could conceivably process them and initiate the relevant strategy.

That alone suggests gamesmanship, even if the "early" trader went earlier than he meant to.

However I can't remember where I read that and could be totally wrong.

Surely everyone knew that the report would be released at 10:30, and they had strategies (or hedged positions) that were not as likely to be influenced by the contents of the report as by the market forces surrounding their orders?

I think Apple's a weird case. They had a "down" quarter, which caused the tech analysts to go all crazy and claim that Apple's a sell. So that caused people to sell, which drove down the price, and dropping below 500 probably triggered a lot of people's stop loss strategies, which drove down the price even more...

I'm not so sure. The financial system is a nonlinear dynamical system. The hallmark of such systems is that small local perturbations can lead to very large changes in system-wide state. High-frequency trading vastly increases the number of small perturbations, and while most remain local, there is a finite probability that some will percolate upward in scale. So micro-scale trading may increase our exposure to catastrophe.

And this gets at a bunch of people, including yourself, claiming that HFT somehow defeats the "purpose of the market(tm)." But if I look this up on Wikipedia, I get the notion that it exists to help corporations raise money and secondarily to provide an indicator of the general economic mood. I fail to see how HFT goes against this. That said, I'd support everything Mark Cuban suggested short of the trading tax (because I believe (without proof) that it would have unintended negative consequences far beyond HFT).

Mark Cuban aside, I mostly see various people who, to quote an Amazon review of "Broken Markets: How High Frequency Trading and Predatory Practices on Wall Street are Destroying Investor Confidence and Your Portfolio", sound like the reviewer's grandparents explaining why they don't use computers. No hard proof - just lots and lots of fear-mongering.

So much so that I'm starting to believe that the underlying issue here is resentment and jealousy that they didn't pull these hacks off themselves and walk away with the big bucks.

Finally, how is HFT worse than the "good old days(tm)" where major brokerages all had seats on trading room floors from which they could manipulate the bid/ask spread with impunity? I'd counter that it's actually better now that anyone capable of getting the cash and talent upfront can get into the same game that's been played since the very beginning, except that the pace is continually accelerating and those left out in the cold are succumbing to future shock. Buffett knows how they'll deal with the events preceding an actual Singularity(c) if one occurs.

But that's just my opinion, feel free to disagree.

As for the 10 minute idea, I'm not entirely sure. I'll ask some friends who are in the industry and see what they think.

I am just generally wary of people in one industry proposing regulations for those in another industry.

Any form of lottery is vulnerable to this. If you clamp all the timestamps to ten minute intervals, then you need some arbitrary rule to decide which of the valid trades in some time window to execute. If you randomly choose a few, you encourage people to put in more orders than they can execute. If you satisfy them partially, you encourage people to put in larger orders than they can execute.

As a result, the only way to do this that encourages proper behavior is to have some arbitrary competitive rule in place. The easiest and most obvious such arbitrary rule is the fastest order wins. This might seem anti-competitive at first glance, because it keeps out the tiny investors who can't get enough capital to compete in this area. But really, it's not that bad because it only takes a few million dollars if that to get top-notch hardware in a Manhattan or Chicago datacenter. In addition, market-making is by-and-large a commodity service: the entire HFT community can only make money when the market is inefficient, and market-maker competition makes the market more efficient. The HFT world makes money off of market inefficiencies, which are capped by the size of the market and the amount of outside investment. It's a hyper-competitive world of arbitrary rules, in place to encourage a more efficient market for investors. So you don't want to do it yourself, and you don't want to regulate them because you want HFT to be a insane low-margin nightmare for the benefit of everyone else.

Because it gives you a better price. There are two sides to any trade; people go to the HFTs because their prices are better than anyone else's. If they weren't providing value, no-one else would trade with them.

> that makes a tiny proportion of the populous vastly wealthy only because those people already are wealthy.

HFT has greatly democratized market-making; in the old days stockbroking was an old-boy's network, virtually impossible for new participants to enter. Nowadays, three or four blokes with computers and one investor can start a new trading firm, and many of the biggest HFT players started that way.

>OK, can you give a reason why that happens and why it leaves more with middlemen

Because there's a higher risk. If I offer to buy microsoft for $50 but that offer has to stay out there for an hour, and news of a lawsuit comes out 20 minutes later, I'm going to lose lots of money. So the middlemen need to be able to bear that risk, they need much bigger capital reserves, and they can't afford to offer the penny spreads we see nowadays (because they need to make a greater profit to offer the same return on their bigger capital reserve).

> At the moment a change in price so transient as to pass in milliseconds gets exploited to extract value from the system.

Where's the value being "extracted" from? Certainly not from a fundamentals trader, who's getting the best possible price with the smallest possible spread (sadly, regulations require shares to be sold in increments of $0.01 and no smaller, so there's always $0.005/share to be made on every trade, which by modern standards is absolutely huge - and is why the HFT guys are willing to spend so much on low latency to maximize their chances of getting that $0.005/share. But the fundamentals trader always pays exactly $0.005/share; the HFTs are just fighting among themselves for who gets it).