This is because of the shift to HFT.
It is all about the shift to algorithmic trading, of which HFT is only a small sub-category.
Past the high infrastructural barrier to entry, and assuming they can stay ahead of the competition, the consistency of returns does seem lucrative, but the actual magnitudes of the opportunities are hard-limited by the nature of the time scales. You can't move any significant amount of capital in milliseconds and expect to profit. Compared to the average hedge fund, they are playing with pennies.
You are right that the actual execution algorithms aren't doing super complex analysis in real-time. Usually that is done offline through simulation to generate rule sets, decision trees or factor weightings. That work is definitely quantitative. Aside from pure arbitrage, most HFT strategies have a statistical edge, not a mechanical one.
> To "flip the spread" you need to predict where the market will go after your order gets elected.
Lets talk abstractly for a moment, that sentence is exactly wrong. The whole point of flipping the spread is that you can immediately sell what you just bought and make the spread risk free. That's what market making is, providing liquidity for the spread as profit. Now, pragmatically, of course the market doesn't sit still for long and the price may change between your buy and sell so of course it's good to predict the market direction to reduce the risk of being on the wrong side of a move. But HFT is trading at the microsecond scale, prices aren't moving much at that scale in that short a period of time and you only need to know where the market is going for say the next second to be in an out of a slew of trades.
> The market is pretty efficient.
Agreed.
> The average limit order loses the spread and rebates after execution.
I don't believe that. That would mean that slippage exceeds the spread and more on most orders and unless you're trading a very illiquid market there's no way that can be true unless you're trading very large, large enough to move the market alone.
> You are right that the actual execution algorithms aren't doing super complex analysis in real-time.
That's my main point.
> Usually that is done offline through simulation to generate rule sets, decision trees or factor weightings. That work is definitely quantitative.
I agree with that as well, but that's not HFT trading, that's market analysis and mostly I believe, though I admit I could be totally wrong here, that analysis is mostly used for slower algorithmic trading, not HFT. In HFT, it's better to be faster than smarter, especially with the sub-penny rule imposed by Dodd-Frank.
FWIW, every successful HFT trader I know does this type of "market analysis" driven trading backed by very fast execution. Speed alone is no longer an edge in most markets. Most are holding risk for more than microseconds or single seconds. Trade direction is auto-correlated in short timescales so even being front of the line you are unlikely to buy and sell to earn the spread so quickly.
> In a competitive equilibrium, this has to be the case.
I don't agree, the spread exists for a reason, it is quite literally the minimum cost of doing business for selling liquidity. If selling liquidity were not profitable, no one would do it.
> If the average limit order lost less than the spread and rebate, it would be profitable to mechanically copy every limit order with an infinitesimally small limit order placed right behind it.
That doesn't follow. You have no idea how much liquidity is sitting at that price so even posting at the same price wouldn't guarantee execution let alone posting behind it. Limit orders are profitable because they are inherently offering a service that the market is willing to pay for, liquidity, that's it.
> If the average limit order lost more, it would be profitable to copy every marketable order trading to remove any residual liquidity on the book at that price.
If the average limit order lost, it'd be called a market order because market orders lose the spread and limit orders make the spread, that's how it works. That is the equilibrium, market orders buy liquidity from limit orders for the price of the spread.
> Of course HFTs limit orders perform better than average or they'd be out of business (and many do go out of business).
Yes, exactly, and they perform better due to latency putting them first in the queue, that's rather the whole point of HFT, being first in line.
> Speed alone is no longer an edge in most markets.
Incorrect, speed is always an edge for the fastest trader; it's simply no longer easy to compete on speed for most firms. Dodd Frank has devastated HFT with the sub-penny rule making latency even more important and driving most out of business.
> Most are holding risk for more than microseconds or single seconds.
Yes, because they can't be the fastest, they have to start speculating a bit. This is because the hay day of HFT is over; they're being forced into speculation by the inability to compete on price due to the sub-penny rule and the inability to compete on speed due to the massive costs of doing so. But once you start speculating for longer than a few seconds, you're no longer a HFT trader, you're just an algorithmic trader. HFT is market making, if you're speculating on direction and holding for expected moves, you're not a HFT, you're an algo trader.