A detailed data-centric exposé on how the market is rigged (2014)
nanex.net
nanex.net
Everyone puts their instructions in, then the turn 'executes' and an algorithm (known to all parties) resolves the actions. Run one turn every second. If you want to buy all 20k shares on offer and nobody else is making the same bid, you get them. Seems reasonably straightforward - what am I missing?
You're free to setup a market that works in this way, and if consumers prefer it you'll make millions.
I'd hazard a guess because the point of markets is to disseminate information and the point of diplomacy is to keep information secret.
Seriously, look at it like this, this buyer is a big smart human with a big brain that talks to other humans and likely knows 'secret' information about the company, lets say it's Wal-Mart and he's ordered satelite photos, etc to gauge traffic and knows they are doing well. He has all that information and all the algo knows is 'BUY 20000 @ $4.53' the algo disseminates that information to all the other traders by cancelling its quotes.
The buyer pays a premium to gain 83% of the available stock on the market. It's like Mark Cuban and his Yahoo stock, if he dumped it all the day he got it the market would tank because there's more supply than anyone wants, so he wisely instructed his broker to sell over a longer period of time. The buyer could have chose to do this, or he could attempt to buy 83% of the available stock all at once in hopes no one noticed.
If you break ties based on FIFO, you've just created a clunky FIFO match, which is used everywhere.
If you break ties randomly, you've just created a clunky pro-rata match, which is used everywhere.
In general orders are serialized first and then processed according to the order that was serialized, it's impossible to reason about an order book with out serializing entries to it.
"My ethernet cable is 5m shorter than your ethernet cable" is does not sound like a reasonable input to the fairness function.
Imagine UPS complained to the Feds that FedEx had an unfair advantage because they bought faster planes...
And actually it isn't very easy because nothing travels faster than light so you need some ridiculous method to determine with nanosecond accuracy the time that each order was 'created' at, which will invariably lead to people 'creating' orders and then not sending them.
Also the entire reason why LMAX is such a great method is because it IS serialized and IS FIFO doing those two things allows 6 million tps.
It's like seeing a car on the lot announcing I will pay $18,000 for this car and then wondering why the dealer won't sell for less than $18,000.
There's no law that says you have to trade on exchanges with HFT, you can trade on exchanges that don't allow HFT, have fun trying to find a counterparty though. Markets aren't fair they are efficient.
With enough money you can destroy practically any company. Don't like the cut of someone's jib? Buy any debt linked to stock price, short the fuck out of it then call it, leak a story about the outstanding debt and then profit on your short, buy more debt and force bankruptcy as the senior lender.
I'm not sure what fractional allocation is, perhaps that's what I'm calling pro-rata? At any rate, the allocation method that is chosen is hugely important. It will have a large impact on market microstructure.
I find that the discrete mini-auctions are strictly inferior to already existing matching algorithms, as they are really just more complex (read: more gameable by HFT) , yet have the same characteristics of other, more widespread systems.
In the follow-up they admit that by not being stupid you get the fill you want. Hence not rigged or stolen, just that naive execution is a poor choice: same as in unfragmented markets, same as before HFT.
A good reference is http://www.amazon.com/Flash-Boys-Insiders-Perspective-High-F....
This is not what changing someone's mind looks like.
If someone sends a buy order to the exchange that quote is listed on for the same amount of shares, it will fill the entire order. What is happening here is that there are multiple exchanges and when the algo sees an order come in on one exchange it cancels orders on other exchanges quickly disseminating price information as liquidity evaporates at the current price.
The point of price is to disseminate information about the supply and demand of a good. When demand goes up price increases. The buyer is buying because he thinks the price will be higher in the future.
Both are motivated by profit, at a certain point the buyer will no longer be interested in buying at that price. The buyer obviously has a piece of information the algo doesn't and is equally trying to capitalize on the algos lack of knowledge of that information.
In the case in point the order was ~60% filled. The price the order was submitted at reflects the 'current' demand, not the demand after the order is submitted. It's pretty good to get a 60% fill when you are buying 83% of the currently available stock.
Those orders were resting there for a long time, it was not flashing.
Like you selling two adjacent houses and see a buyer purchasing one and wanting to purchase another. You understand that he/she wants the house really badly and increase the price for the second house.
Someone wakes up at 6 AM reads the paper and phones every real estate agent in town telling them they will pay the asking price and a little bit more, most agents sell to him and he makes a nice tidy profit off of you who now have to buy from him.
Is he front running you or are you just dumb? Especially given that you could take news paper ads out all week long and then let the guy who bought 800 of the houses take the loss on them. Or you could execute a bunch of small 50 trade orders and buy all the houses you want at the asking price.
I need to sit down to calculate this, but that is guaranteed to mean that they are reading from RAM and not disk. Could the entire stock market fit in a single hashmap in memory?
That's just some awesome hardware and super efficient software.
Edit:typos
I don't see why not. You also don't have to put the entire market onto one machine - you can shard the market across multiple servers, with one server handling each symbol.
Sure, but now you've added a router, consistent hash or the like, to the picture, which is either another hop or a custom physical rack instead of a traditional ToR. It's not impossible at all, it's just impressive that they do all of this in 1ms!
not necessarily. If the HFT companies are using some sort of secret internal API (very possible as they like to shave milliseconds off their execution time), it's possible that they are talking directly to relevant server, bypassing any loadbalancers or routers.
The order book itself is tiny and there are only ~20,000 stocks, and you only have to fit one stock per computer.
Imagine you are a market maker trying to make the spread and provide liquidity back in the old days. There is only a single stock exchange. Your firm obviously has a limit on how much risk it can take. How much capital/money it has available. So let's say you are allowed to post 1000 shares on the offer and 1000 shares on the bid. You are not allowed to hold a position of more than 1000 shares long or short. Life is simple.
Then things get complicated. A competing exchange is now set up and begins to thrive. Yay for competition! Yay for the free market! But you still have the same limits on your capital and risk. What do you do? Do you now post 500 shares on the first exchange and 500 shares on the second? You could do that. Then you are guaranteed that you will never be long or short greater than 1000 shares. You'll never break your limit. But what happens when there are 10 different stock exchanges? Do you post only 100 shares at each one? Should the available liquidity in the market be fragmented into 10 pieces? Of course not.
So what you do is post your bids and your offers for your full amount on each exchange. And then if you get filled on one exchange, you cancel your orders on the other ones. This isn't crazy, this isn't radical, this isn't some nefarious plot. It happens every day in all sorts of markets. I might put my house for sale on three different real estate websites. When it sells on one, I cancel the listing on the other two. No big deal. Or if I am a fisherman in India, I might send my sons to three different markets to sell my catch. When one finds a buyer, he phones the other two and tells them. (http://qje.oxfordjournals.org/content/122/3/879.abstract). This is simple stuff.
In more technical terms, the offers and bids are all bona fide ab initio. They are all bona fide when initiated. When posted, you do actually want to sell or buy at that price. When you get filled at one exchange, circumstances (your inventory) have now changed, and you are entitled to cancel.
Note in my explanation, the market maker isn't front-running, or even back-running, they are just responding to their own personal changes in inventory level (position) that means they can no longer offer those shares at the other exchanges. Electronic market makers may well have sophisticated models that predicts the short term price move causing them to pull orders at other exchanges, but we need not even consider those models to explain this behaviour. Changes in inventory is ample explanation.
Of course, by posting their full available inventory on each exchange, market makers expose themselves to a new risk. Someone might come along at the exact same instant and fill them on every exchange! Then they will be over their limit and massively exposed. Luckily, they have very fast systems and low latency lines to help protect them from this. If they didn't, they would be forced to offer 1/n times their current liquidity, where n is the the number of exchanges. That would suck for all liquidity consumers.
Which brings me to the most amusing part of this whole story, is that when the 'hero' of Flash Boys Brad Katsuyama figured out he could precisely fire off orders so that they hit all exchanges at the same instant in time with his 'thor' program, the SEC came knocking. Because behaviour like that, ambushing market makers on multiple exchanges, could in fact force market makers to offer much less liquidity. I don't have a problem with it, but it amuses me that the supposed hero was the one probably endangering market structure.
>"ambushing market makers on multiple exchanges, could in fact force market makers to offer much less liquidity"
The article (as well as your first paragraphs) demonstrate that this "liquidity" isn't real in the first place, and gets withdrawn if anyone tries to trade on it. So if the HFT are not offering it, nothing of value was lost! (Indeed it is better from a macroeconomic and social standpoint, since the wasted brainpower of HFT quants goes to some other economically useful endeavour.)
> So what you do is post your bids and your offers for your full amount on each exchange
No, financial markets order books don't work like this. High-latency markets where your counterparty is a spiv who goes back on their word work like that (OTC securities, real estate or indeed dealing with people at Indian fish markets). But in an order book, orders are supposed to be firm and executable. If they aren't, the market is rigged. As it very clearly is.
Where? I have explained that it is real. The confusion stems from thinking you can sum up the orders at each exchange. It would be like seeing the same house on three different real estate websites and then thinking you can buy three houses. They are the same house. Listed three times. The liquidity listed at the 13 different US equity exchanges is listed by the same market makers, so you can't sum it. Do you understand?
>But in an order book, orders are supposed to be firm and executable. If they aren't, the market is rigged. As it very clearly is.
How aren't they executable? No one can see your order coming. If you send a single order to NASDAQ or any other exchange it will execute. They are all executable and firm at the exchange level. Nanex seems to think you should be able to execute everything visible at all exchanges at once. You can't.
No it's not, since with the houses it would be easy to verify that there's no 3 houses available. The trader in the Nanex article had no way to know how many of the 24800 shares were actually available. He's trading at an information disadvantage.
To your analogy, it would be similar to a real estate agent flooding the market with multiple listings of each house, but with different pictures and different addresses, giving the impression of a buyers market to a buyer (phantom liquidity).
You are correct, public markets are anonymous by design, so there is no way to tell that the same liquidity is being offered in more than one place. That doesn't change the reality that it is. As I explained, it must be, or the transition from 1 exchange to 13 exchanges would have seen offered liquidity by market makers cut to 1/13th.
So like I said, anyone with a working mental model of the market knows this. You can't just aggregate displayed liquidity at each venue and think that number is valid. It doesn't matter what your 'screen' is telling you if the screen is doing something that makes no sense. No one ever pretended that you can magically execute on different exchanges as if they were one exchange. No one ever made that guarantee. The only people who think that are wingnuts like Eric.
And since we are talking about an issue that only effects people sending very large orders to multiple exchanges, the likes of Katsuyama have no excuses for their ignorance. They are paid to be on top of these details. You can't have a 'trader' in the modern age who doesn't know what a millisecond is, or isn't prepared or capable to dig below one abstraction layer. "dUh, my screen say 24800 shares are there and i pressed buy" is not what you pay an execution trader for. It is his fucking job to know what happens when his order goes down the wire.
>To your analogy, it would be similar to a real estate agent flooding the market with multiple listings of each house, but with different pictures and different addresses, giving the impression of a buyers market to a buyer (phantom liquidity).
They are not "flooding the market" or trying to present "phantom liquidity". They are doing exactly what you do when you put the house on 3 different websites. Trying to increase your chances of finding a customer. They are forced to put bids and offers on 13 different exchanges, because that is US equity market structure. It's no fun. Life would be a lot simpler if there were just one exchange. As I explained, it's like playing russian roulette for them... they are now forced to take the chance that they may get filled simultaneously on multiple exchanges and take on more risk than they are comfortable holding. But they are forced to do that since their customers are now spread across 13 exchanges. Market makers would like nothing better than to have 1 exchange. Arbitrageurs are the ones who love having all this fragmentation.