HFT Quote Stuffing is a software bug
chrisstucchio.com
chrisstucchio.com
I've seen quote stuffing used to mean what Chris mentions here. The implication being that HFTs were using large volumes of low quantity orders for nefarious purposes such as DoS the exchange & SIP or confusing your competitors by making the order flow to heavy to process. These kinds of games did occur, but it was very briefly a long time ago because the exchanges themselves had a vested interest in stopping it and an easy mechanism which to do so (fill ratios). I prefer the term quote spamming for this, and I agree with Chris that most of the time when this happens it's just a bad order management bug.
Quote stuffing can also refer to the practice of putting a few large orders into the order book with no intention of trading them in order to make the volume on the book look larger than it is, and then cancelling them quickly to take advantage of the "real" volume. This is a predatory activity designed explicitly to take advantage of market making strategies and is already illegal. Your mileage may vary when it comes to enforcement...
I also don't think it can make that much. It may work for a little while, but no algo is going to sit there bleeding money to you forever without hitting a risk limit or being reviewed/adjusted. If your business model depends on somebody else doing something dumb, it's not going to work for the long-haul.
This kind of behaviour that undermines the integrity of markets should not get a pass based on who it does or does not hurt, tho. If nothing else its bad PR that keeps people out of the markets or creates other "absences" or dark patterns from people not showing up to trade, fro whatever reason.
It's one of my pet peeves that when it comes to this industry there is very little standardization of the language.
"The practice of spoofing, or sending in fake orders in order to gain information about what other investors, traders and other algorithms are doing, is the corner stone of most High Frequency Trading strategies (HFT)."
This is crudely done with volume of traffic, but more elegantly done with frequency and amplitude (and so on) given some set of (possibly wrong) assumptions about how competing algorithms will respond to this rapid fire order placement.
I know almost nothing (beyond ZH and chapters 1-5, so far, of _Dark Pools_) about HFT, but I can trivially think of ways you could be evil with quote stuffing as it is described in the OP.
Most cases of this or similar behavior like stop running have involved human traders at loosely regulated click trading shops or smaller electronic firms that don't play by the rules:
https://www.finra.org/Newsroom/NewsReleases/2012/P178687
http://www.cftc.gov/PressRoom/PressReleases/pr6649-13
Aside from the regulatory risk, it doesn't fit the business model for most "HFT" firms, who generally are making many low-risk, low-reward trades with a small statistical edge based on proprietary models. Spoofing usually requires showing large false interest which entails considerable risk if you actually get elected. If you are fast enough to make markets or arb crude oil, you can make way more doing that than manipulating the DOM hoping to push algos into doing something stupid until they figure your tricks out.
Predicting markets is hard. Predicting how thousands of heterogenous agents will react to your order is even harder, if not impossible.
My very small amount of knowledge on this topic leads me to believe that the idea that one algo would specifically target another algo is not at all farfetched. Who cares how the market responds to your weird trading if you're trying to create distortions for one, or a few, actors ?
Liquid markets with a diverse set of actors are more resilient to manipulation attempts. It's hard to do unless you have more capital or are willing to take risk that other actors in aggregate are not.
Betting you know more than the guy moving large size rarely a good move, though...
But there are other ways to DoS - what if I observe your trading activity and I discern something about your underlying algorithms and how expensive those algorithms are in terms of memory or CPU or latency, and I make a bet that a certain trade pattern will dramatically overwhelm your resources via these second and third order effects...
It's just speculation, but the point of the OP was that there was no way this could be useful to an HFT algo and I am pointing out that in a few minutes we can come up with all kinds of ways that it could be.
Since the traffic is anonymous, how do you propose to do this?
[Edit] one HFT directly overwhelms another HFT
No, the dos needs to go through the exchange, thus is is not "directly" but indirectly through the exchange.
Traffic other than your own is anonymous, but that doesn't mean you can't spot patterns that are almost certainly the behaviour of one other agent. On some exchanges you can track the history of resting orders, too, which gives you some more info (though still doesn't tie all of any player's orders together, anywhere).
This blog post details similar issues that can lead to rapidly canceled orders. Generally, firms that are smart avoid moving their prices around "too much" since:
a.) You lose your spot in the queue. If I am near the front of the price/time queue buying at 100.01 and move to 100.00 and back up again, now I'm behind everyone else and will either lose opportunity or face more negative selection (I'll only trade when someone wants to sell all the contracts bid at 100.01, whereas before I would get some trades where someone only wants to sell a few).
b.) Sending orders is more computationally expensive than doing nothing. If you send enough orders you will slow down your own system.
c.) Depending on market rules and behavior the messages you send may cause your gateway to queue up or block you from sending more messages which hurts you.
I don't think quote flicker can provide any "evil" benefit to the person doing it. It doesn't make sense to do it to slow down competitors, since most feeds are multicasted to everyone and you still have to process all the orders you send, both from the gateway and over the data feed. If you are faster at handling this situation than your competition then you are going to beat them anyway so what's the benefit in slowing them down?
I do think it can be disruptive in the sense that it places extra load on the exchange and can annoy participants who have trouble taking prices on their screen. Some markets have quote/trade rules to encourage more efficient messaging which is their right and not a bad idea if the limits are set reasonably. Rules like this make more sense than a minimum resting time, which just penalizes makers at the expense of fast takers. Imagine I am putting quotes out in SPY, an S&P 500 ETF, but they need to rest for half a second. Someone sees the S&P 500 futures or the individual stocks move and then my quote becomes a free option to trade at an unfair price. In a market like this, market makers would be forced to put up wider prices to compensate for these losses.
A high quote/trade ratio in itself is not disruptive. A lot of the Nanex charts use thinly traded derivative products like ETFs as their whipping boy. The underlying value of these products will change whenever another stock or commodity changes, so market makers need to update their quotes quite rapidly to avoid trading at unfair prices, even if no trades have occurred in the product itself.
Suppose your fund has invested a lot in some hefty compute hardware but your trading strategies are on the weak side, whereas you know your competitors have better strategies but their computers are slower. Isn't it then in your interest to stuff the order feed and make everyone more dependent on computer power?
I can't speak for everyone but most "HFT" models are built around providing some kind of service to the market, generally either making markets to try to make the bid-offer spread or arbitraging away inefficiencies between related products.
Even if you are evil, building a model that solely seeks to take advantage of another trader doesn't make sense in the long run. Eventually they will alter their behavior or exit the market. It's not sustainable.
- walking others into the book,
- creating false midpoints and
- trying to cause stale pricing and slowing others down.
Flashing orders with the intent to manipulate is wrong and is illegal already. It doesn't matter if you do this by hand or with a computer. Putting up a tighter best price, even if it doesn't stay put very long, can only be beneficial. At least some of the time, other traders will get filled at a more advantageous price than they would otherwise.
It's odd that he touches on the obvious solution of rate limiting, but doesn't actually discuss it as a way to stop this problem. If quote stuffing is a bug that harms the people carrying it out, then why don't HFT programs have a rate limiter on them that make them stop? He even goes so far as to say that you can get shut down for doing this, so why isn't there a tripwire to make sure you can never, ever do it?
They do. But a rate limiter is just another signal - you still need to set the rate limit high enough that you can correctly respond to market conditions, but low enough that you don't go wild.
"Never, ever do it" is like "zero tolerance for bugs". It's a goal to strive for, but it's not achievable.
You don't want to do that, because it doesn't let you make as much money as you otherwise could? I totally understand. But in that case, quote stuffing is not a bug, it's a deliberate tradeoff.
But I suppose that "we let this happen on purpose because it allows higher profits, even though this particular behavior doesn't help us" doesn't sound as nice.
And given that quote stuffing doesn't actually hurt anyone, it's not like stopping it is a high priority.
It does hurt people in some way. It increases the amount of data we need to handle. More storage space and more processing required.
And you've been a bit disingenuous in your article - there is a theory behind the "2.???" part. The theory is that slower participants are overwhelmed by the quote traffic and end up lagging, and are then taken advantage of. I don't personally believe it, and I'm sure 99% of these things are just feedback loops, but it is theoretically possible. The faster exchanges are cycling in say 20-30 micros these days, so any participant who can't match that rate will easily fall behind. Who knows, weaker participants may still be taking 1-2 millis to handle an update. So just before some known vol event, say an economic figure, you flood the exchange and lag the other participants by say 50 millis, then pick them off after the new information arrives.
How does it help you to slow yourself and C down? That just means A will beat both of you.
The thing is, you know to ignore the burst on the public feed, since you know it is coming (you created it). You still have to process the order book updates, but you can skip your decision logic. So any participant who has an internal latency (packet decode + orderbook update + decision logic) greater than 1 exchange cycle, and who doesn't have an "out" clause to stop making decisions if it is falling behind, will be vulnerable.
Like I said, I don't buy it, but it is possible in theory.
Also, "know to ignore the burst" doesn't mean you don't have to process it. It merely means you need to subtract the result of the burst from your signals.
I have no strong opinion about HFT. I don't have a lot of interest in the stock market in general. I do, however, have a low tolerance for weaselly language and misdirection.
Similarly, if you build a system on MySQL and later see dates like `0000-00-00` (yes, really), that's not a bug. That's a "side effect, but one that's allowed on purpose". Pretty much all bugs fall into that category - after all, you could have spent more time on formal verification/testing or just wrote the whole thing in Coq.
I'm sorry you found the language "weaselly". My sole intent here was explaining the mechanics behind quote stuffing. How would you prefer I phrased it?
As for wording, don't call it a bug. Say that it's a necessary side effect of making money with HFT. It's an inherent design limitation. The word "bug" implies that it can be fixed, that you can theoretically do HFT at the same speed as you are now without any quote stuffing if only you had perfect code.
Let's take a vaguely similar example of streaming audio. I worked on a near-realtime streaming audio product for a while so it's something I'm familiar with. Latency and reliability trade off pretty directly. The product I worked on defaulted to a latency of two seconds. This made it pretty robust against network hiccups like wifi interference, but the substantial lag made it annoying or just plain unsuitable for some applications. It was possible to adjust the latency (although this was an advanced technique to the extent that we hid the setting pretty thoroughly) and you could get it down to a fraction of a second and still have the system work overall, but it would be much more prone to dropouts due to network problems.
Now, there are some things that could be done to improve this problem absolutely, like adding forward error correction to the protocol, but nothing is going to eliminate the tradeoff entirely. Because of that, I'd never describe it as a bug. If someone complained about the latency, I told them that it was an unfortunate but necessary consequence of ensuring reliability. The latency can be eliminated easily, but at the cost of less reliability.
Likewise here, quote stuffing is an unfortunate but necessary consequence of HFTs. It could be lessened or eliminated by making HFTs less HF, but you don't want to do that, because you feel that the Hest possible F is more important. I call the writing weaselly because it portrays quote stuffing as completely unavoidable, but the real scenario is that quote stuffing is completely avoidable, it's just that you prefer to make more money instead of stopping it.
Here's a benchmark: human market makers rigged the public markets so that tradable instruments were quoted in fractions of a dollar, rather than in pennies, so they could surreptitiously increase spreads. It's hard to get more brazen than that.
Anonymous limit order books are far better. Nobody's price is better than anyone else's and machines have no problem competing spreads down to near-zero.
I don't think this is right. Erh, obviously U.S. equities traded in quarters and eights (and later sixteenths before decimalization), but to say that they were using a weird fractional system because they wanted to rip people off isn't documented anywhere in the literature. They could have moved to decimals and still kept the minimum price variation above a penny†, or they could have just added more fractions (32nds, 64ths, 128ths) that would have moved the minimum price variation into roughly the range you get by quoting in decimals. If you think quoting in 128ths sounds silly I won't argue with you, but it's something they do actually do for the US treasury futures contracts on the CME††.
I suspect that they just used what worked. In a market of human traders, it's much easier to deal with large round numbers. I think that's why odd lots were treated specially too. E.g. It's a lot easier for humans to keep track of 5000 shares at 50 1/2 than in 100 small orders from 200 shares to 5 shares all priced between 50.40 and 50.60. As markets adopted technology, they naturally moved to a smaller minimum price variation (and eliminated the special treatment of odd lots) because computers don't have any problems dealing with an aggregate of small orders vs one big order.
† They are actually talking about doing this for some small cap stocks later this year. http://online.wsj.com/news/articles/SB1000142405270230427530...
†† ctrl-f "Quotation Practices" -- http://www.cmegroup.com/education/files/understanding-treasu...
Search "odd-eighth".
Decimalization drastically reduced spreads. Scott Patterson suggested in _Dark Pools_ that it literally put many human market makers out of business; in other words: those human market makers were kept afloat by the gross inefficiency of quoting prices in 8ths and 16ths, an inefficiency that was taken out of the hides of investors.
† http://blog.pmarca.com/2013/03/26/unshackle-the-middle-class...
Can you artificially inflate spreads without making inflated spreads the explicit law of the market by quoting in 1/8ths? Sure. Is that what happens? No.
You implied that maybe humans had been using fractions because they were easier to deal with. No. They were using the fractions they were using in order to skim extra money off trades, and they fought tooth and nail to prevent competition from reducing spreads below 1/8ths --- in fact, if you read the link I showed you, it wasn't even 1/8ths; it was 1/4s!
I'm having some trouble with the contention that this is just such a rare thing. I do a lot of audio signal processing so I'm used to dealing with noisy signals at Khz rates. Threshold fluctiations are a fact of life if you're trying to do things like pitch detection, envelope following, or sidechain compression (all of which have obvious analogs in the trading context).
Obviously you don't want to impose too brutal of a low-pass filter on your input because you would miss important signals, but what prevents you from running the input signal to an LPF in parallel and keying a noise gate when the amplitude of the low-frequency (trend) signal drops? This is such a common problem in dealing with audio that we use specific devices called de-essers for recording vocals, because sibilant sounds otherwise tend to throw off other components of the signal chain: http://en.wikipedia.org/wiki/De-esser
Ironically, you mention using XKCD-type graphs because 'nobody reads' the explanations and caveats, but just looks at pictures. I wonder if you aren't missing a trick here by over-engineering your HFT systems for speedy response at the inevitable price of positional confusion. Is it possible that you're using Infinite Impulse Response filtering and running into denormalization problems as a result? If so, can you not rectify the problem by adding a small amount of noise?
I get your point about the negative aspects (for the HFT trader) of quote stuffing, how it can easily lead to being on the wrong side of a trade or punishment by the broker for flooding the output channel with unintended orders. But, you want to make money from automated signal analysis, that's your problem in much the same way that a sound engineer is responsible for preventing transient pops and feedback howls in an audio network. Sure, these problematic signal conditions are statistically rare, but when you're dealing with a high volume of samples you'r going to run into statistically rare conditions on a regular basis - and indeed you mention that this is the sort thing you might expect to happen once a day.
http://www.decal.org/file/2945
"Alpha is often nothing more than taking commonly available data and mathematically encoding it in a signal correctly. Correct often means something as simple as using an rate of change instead of a difference, normalizing a value, smoothing a chaotic signal with an EMA, using a heteroscedastic weighted linear regression instead of a simple regression, or handling all numerical errors or edge cases."
It's not always easy to get this completely correct in every situation and traders face a reward function that doesn't necessarily reward correctness so much as avoiding Type II errors.
it seems like this is working backwards from the assumption that quote stuffing must be benign activity, and then trying to rationalize it with signal/noise theories about finding the fair price level.
it is totally inaccurate to say that there is more load on the sender than the general public, and also misses the mark on what the DOS theory alleges.
the exchange bears considerably more load than the sender of orders. the exchange must (1) ack the order, or reject if any flags are populated out-of-spec, (2) add the order to the order book, (3) check for potential matches based on price of order, if so, cross and update feeds, (4) if non-marketable, publish out to (a) depth-of-book feed, (b) SIP feed, etc.
the DOS theory is that excessive activity in a single partition can slow down the publishing of data in that symbol range, which would mess up the quoted price for the exchange in a symbol, which would affect orders priced on the exchange, or anyone using the price in a derived manner (at a dark pool etc).
Interesting - the DOS attack is on the exchange rather than other market participants. But could you be more specific on how this helps you make money?
Other games involved spamming open auctions, settlement trades etc.
This gaming was a very brief phenomenon (because it is stupid, doesn't make as much money as traditional trading, and trivial to foil). The exchanges got rid of it almost immediately but it still gets passed around as folklore for one of the "predatory" activities that HFT does.
a.) Each trader has one or more gateway processes that do the order checking/rejecting.
b.) Each gateway maintains a queue of messages for the matching engine.
c.) The matching engine/order book for a product pulls one message at a time from each gateway that has a message waiting on its queue and processes it before spinning back to the first gateway again.
So a trader blasting the exchange with orders isn't going to impact other traders who also place orders around the same time. She'll only be slowing herself down.
Chris: You've also missed the most common cause of these issues in my experience - feedback from identifying your own orders in the public stream. HFTs need to filter their own orders out of the orderbook they are trading on, or a feedback loop can begin. This can be difficult, since some exchanges don't provide a clear way to identify your own order, or it may be the case that you receive the public UDP packet before the private TCP packet that contains the ID required to match the two.
For example (a basic one), say I have an algo that calculates the midpoint based on the best bid and ask volumes. And can only insert within x distance of the midpoint. I add a bid to the top of the book for a 1 lot, get the UDP public depth update, don't know whether this is my order or someone elses. So I re-calculate the midpoint, which pushes the midpoint upwards (greater bid volume), which means I then pull my bid, which means I recalculate the mid again, then re-insert, etc. etc....
I guess I assumed everyone's reconciliation system was as good as the one I worked on. But you make a very good point, as well as a meta point - this stuff isn't easy.