Hacker News Reader

Comment by darklajid | Hacker News Reader

I have the utmost respect for Patrick and the team - and just read the same thing via mail.

But - as someone that _loved_ Microcorruption and followed StarFighter from day one, stalking the people on Twitter as to not miss the launch: That's not for me.

The game right now - as this article, this mail shows - requires you to learn about finance systems before you even start playing the game. It's a game where the rule book is - from this pov - too long and already uninteresting.

Good luck, I'll sit this one out and would love to see a future expansion in different directions. Or another Microcorruption, Stripe CTF etc.

View on HN

I respect your opinion on this, but you might enjoy the tech tree which I keep describing as "low-level programming", which is what Erin and Thomas have been killing themselves over for the last several weeks. I haven't written about it much because, frankly, that's not my bag just like finance isn't your bag. (Granted, finance isn't my bag either, but I find myself warming to it after scratch-coding a stock exchange.)

Thomas describes that tech tree as "MicroCorruption the way we would have done it if we had dialed it to 11."

Sneak peak: https://www.evernote.com/l/AaedYkVFsB1Oa4eDakZay59-YF7Lu0mcJ...

(I don't have a good screenshot of it at present because they're currently doing UI surgery on the React 0.14 upgrade.)

Rolls-Royce manufactures awesome cars that I will never buy. Let me tell you something they are not worried at all.

It's ok not to be for everybody :D

Keep it up Patrick.-

Hi Patrick,

Will part of the game be writing the components of the exchange (order book, matching engine, etc.)?

A previous post suggested that some challenges would be reasonably solved by replicating a local implementation of the order book.

Interesting. I'm the opposite. I was sorta curious, so I started reading Flash Boys. It became immediately obvious that A: the author is fucking clueless, and B: I didn't know enough about how this system worked. The book called "Dark Pools" was a bit better (mainly as a light history of electronic trading). "Flash Boys: Not So Fast" was really fun, mainly because it systematically trashes the author of Flash Boys.

It was interesting enough that I've picked up a few books on market microstructure, that really get into the details. It's fascinating, not just technically, but from a standpoint of looking at humans. It's odd to see how such strange systems persisted for so long (like Patrick mentions in this post, the crazy 1/8th spreads that were gamed). Or in "Dark Pools", they discuss how some of these traders would call up Nasdaq wondering why a certain trade did or didn't work how it did. And they'd find out all sorts of strange rules. Which makes me wonder, wtf? Are Nasdaq's rules hidden? Or were people doing this as a living just too lazy to read the specs? [1] I can only guess how much easy money there was to be made 20 years ago.

I'm so looking forward to Stockfighter, enough that I started looking at FIX and OUCH (OUCH looks far faster since there's ~no silly parsing - just copy into a struct) just in case that's what they offered. It's too late for me to actually work in finance (far smarter people with more experience in math and compsci), but playing the game is still an option. (I've actually been thinking of how we might run some fake exchanges, federated, as a game.)

Anyways, I guess to each his own. But I'm guessing I'm not the only one that will find this scenario intriguing.

1: Even basic stuff, like market orders, are weird. Who the hell would ever submit an order that's literally saying "Sell at any price, including $0"? (Well I suppose now it's "current price minus circuit breaker limits".) Or "buy at any price up to $MAX"? Sure, most of the time it works out fine, but if you can't be bothered to come up with a limit, perhaps you shouldn't be buying or selling?

> Or were people doing this as a living just too lazy to read the specs?

Pretty much every domain I've worked in has stunned me with the number of people who do things for a living without reading the specs (often literally, or understanding the basics in areas where there aren't literal specs.) Both conscious cargo culting (where people know they are doing what they've learned to do from others without understanding the basics, and know what they don't know) and false knowledge (where people do things based on an understanding of the underlying process that is completely wrong, and so don't know what they don't know) are distressingly common.

And just to be clear, I don't hold myself to be superior in anyway. I avoid conscious cargo culting except when there is no practical choice; but I've certainly ran into places where I discovered I had been operating for a long time on false knowledge when good sources existed to correct that, but I didn't know that I had a gap that needed correcting.

When you've been treading water for so long just trying not to drown, it can be hard to focus on better swimming techniques when you get a moment of respite. Eventually, you can get into such a rhythm of almost drowning and recovering, that it can be hard to imagine anything else until you see it. Finally catching a glimpse of your neighbor who happens to just be merrily floating on his back can be revelatory.

> Even basic stuff, like market orders, are weird. Who the hell would ever submit an order that's literally saying "Sell at any price, including $0"? (Well I suppose now it's "current price minus circuit breaker limits".) Or "buy at any price up to $MAX"?

In MMORPG games, those kinds of orders on the auction houses are awesome because they mean someone wasn't paying attention. Profit for the person who finds them! :)

If enough people look for them and compete, the poor sucker won't be out of so much money.

Thank you for the book recommendations!

How was Michael Lewis clueless? He gave a step by step description of how regulation in reaction to previous crises and the money gave rise to HFT and specifically that Banks get paid by HFTs to submit their orders to specific exchanges. HFT is a game of speed that arbitrages between dark pools and the various public exchanges that ultimately provides no value.

Among other things, there are times in the book where it is very unclear whether he understands execution of an order against the orderbook is atomic. For example, he posits that someone can, on a single exchange, hit a sentinel order placed by a market maker with a larger order and then have the market maker adjust the prices of orders behind the sentinel, in response to the execution of the sentinel order, prior to the original order hitting the orders behind the sentinel.

That can't happen. It might sound plausible, understandably, to someone who has never done concurrent programming and who doesn't understand that e.g. locks are a thing which exists in the world. I am attempting to restrain myself from being too critical here, because that's a higher level of technical knowledge than I would generally expect from a journalist writing for a general audience, but then again this journalist wrote a book-length treatise on how distributed systems and concurrent programming are screwing the American public without successfully demonstrating that he has a handle on either of these topics.

For the better-informed and book-length version of this comment, see http://www.amazon.com/Flash-Boys-Insiders-Perspective-High-F... , which is some of the most devastating technical writing I've ever read.

> he posits that someone can, on a single exchange, hit a sentinel order placed by a market maker with a larger order and then have the market maker adjust the prices of orders behind the sentinel, in response to the execution [..] prior to the original order hitting the orders behind the sentinel.

"That can't happen." True, technically. But what you described earlier is close what is called the "Last Look" ability.[0] It's highly unethical, because it provides market makers and large operators a privileged data feed and ability to effectively go back in time.

Im simplified terms: if an exchange provides LL, executions against standing orders from privileged parties would be flagged as "pending" for a short window and the privileged party is sent a message of the pending execution. During the time window, they are allowed to respond with a cancel, which will be performed in front of the execution. And if a market maker knows that an execution would result in a bad trade, it will then start issuing cancels for other standing orders too.

There's also a kernel of truth in the other part I quoted. I firmly believe that market makers can't adjust prices for their standing orders, but in some exchanges there are order types called "reductions", where the market maker is allowed to adjust the available quantity of a standing order. So instead of issuing a cancel + order, the market maker can simply drop the available amount in their standing order while still keeping the order itself in the book.

0: http://www.bloomberg.com/news/articles/2015-03-03/barclays-l...

Last Look is a an issue on ForEx markets, but ForEx markets are already extremely dodgy. In many cases, your counterparty in the ForEx markets owns the exchange you are trading on. I've never encountered an equities or futures exchange that allows last look.

As for price reductions, some exchanges allow reductions, others model them as cancel + order but allow the new order to take up the priority of the old order (effectively a reduction) and still others model them as reductions, but you still lose priority (effectively a cancel and new order). That is largely immaterial though, because even if you allow reductions, if a large order hits a level it clears fully through that level before anyone can make any reductions. In fact, reductions aren't a problem, increases are because it would allow someone to sit in front of the priority queue until a level is fully cleared.

Something that did happen on some exchanges, which is closer to what you are getting at, is that on some exchanges the fill feed is less latent than the market feed. That is you would get notified that an order executed before that order was disseminated on the order book. This would allow you faster insight into the market, so people did use sentinel orders for that, but if a single order was hitting a level enough to clear it those sentinel orders wouldn't help you get out of the way on existing orders at that level, they would get filled as well. But it could be a leading indicator of multi-level sweeps. Largely that wasn't desired functionality on the exchanges side, just a by product of their architectures, but people whose job it is to understand how the exchange behaves, clearly had advantages there over those whose job is not that.

Can you guys explain what you mean by a sentinel order?

In this case, a sentinel order exploits the speed differences between different data feeds. So if an exchange sends data on executed trades out slowly, but notifies traders of their own trades in a fast way, then you place a small order so you can see market activity faster than the general activity feed.

The software engineer has a habit of imagining bow he would design a system, and then assuming the real system is built without any of the flaws he could design against.

His examples are just flat-out wonky. For instance, his key opener is a trader that is shocked to find out that when he buys tens of thousands of shares, the price, gasp, moves.

Or another one, where the price moves before the guy hits enter. So either he's saying HFT's have trojan'd traders' workstations or ...? Instead of investigating this, he just writes more "ooh spooky" lines.

At one point, he spins regulations requiring customer orders use the best price as a bad thing. Basically, you get the feeling from his book that he'd feel better if we went back to specialists on the floor with quotes in 1/8th, doing stuff you can understand as a human. That somehow, all this tech is bad. Hell he even makes fun of colocation - not adding multiple third-party providers of fibre to a connection to make it faster and less likely to break is obviously evil because... well he says it so ominously it must be true!

He doesn't say HFT provides no value. He says everything's a huge conspiracy (involving, apparently, everyone) and HFT is stealing money. He fails to realise there's risk. Instead he makes it out like HFTs can just buy so fast that money falls out (that if they buy at one price, the only possible outcome is a sale at a higher price, that the market can never move in the other direction). He even gives them magical capabilities, implying that they can determine total order sizes from a single trade.

That part about Brad Katsuyama in the beginning made me so angry. Here's a 'hero' trader making literally millions of dollars per year, and he has no idea how the stock market actually works.

It would be like UPS getting pissed off at Fedex for using this crazy contraption called the airplane to get freight overseas faster.

> where the price moves before the guy hits enter.

Can you give a page # reference for this? In the scene I recall, the price is stable on all exchanges for minutes--until they hit enter on very small order to BATS, at which point the prices increase on the exchanges further away.

“I hadn’t even hit Execute,” says the hedge fund president. “I hadn’t done anything but put in a ticker symbol and a quantity to buy. And the market popped.”

From location 142/505. And this is supposed to be the guy in charge of a $9B fund. The next page continues to state that all the big players (like Vanguard) have no clue how markets work. As if these evil HFT wizards live in another dimension where you can't just, I dunno, interview them. Or hire them. Or buy their companies.

The lulz never stop. Next paragraph has an amazing quote about colocation "What the fuck??!! That's got to be illegal!"

I guess if you act incredulous, insulted, upset, and bewildered, any mundane thing can turn into a sinister conspiracy. The sad part is that people are taking this book seriously, and politicians even seem to be "addressing" it as a campaigning point? Sigh.

The reason the price moved is because his trades were being front run. Otherwise the market had said there were 10k shares at X but when he tried to buy them he could not because an HFT bought those shares before he did.

No, no, no. The reason the price moved is because market makers saw the trades on one exchange, then updated their quotes on others. This is normal, regular, to-be-expected, good behaviours. Calling it front running just shows a lack of basic understanding. Or in Brad/Lewis case, probably an ulterior motive (selling Thor or promoting IEX).

In fact, the book even shows it's not front running because Thor just delays orders so they hit all market makers simultaneously, so they cannot react.

There's no "market", but a group of exchanges. The orders to a single exchange execute immediately; there's no way to do what you're saying. And across exchanges, this behavior is exactly like it should be. Here's how to verify: increase the times.

Suppose the trader moving a big block sent an order to one market to buy up 10K shares. Then he sits for an hour. In that scenario, do we expect the price to stay the same? Of course not! Everyone sees the one order on that one market and reacts (canceling their outstanding quotes on other markets, for instance). That's all that's going on, except faster than humans can react. Why is it suddenly bad when a computer does it?

Front running is well defined. Front running is not just acting like a normal intelligent person and updating your quotes (or even trading) after a large order executes.

HFT (the automation of human market makers) has provided enormous value by reducing bid/ask spreads by at least a factor of 10x.

Was it HFT or the move to electronic trading that reduced the bid/ask spread by 10x?

Certainly electronic trading. I've found this to be an excellent article deflating HFT benefit myths: http://www.nanex.net/aqck2/4594.html

The article is a bit suspicious. It talks about Thor, which timed orders to different exchanges so they'd arrive simultaneously. This is because a market maker would notice a large order on one exchange, and raise his price. This is totally and utterly expected - why should a market maker take undue risk? Increase the latency - suppose a user submitted a large buy order to Exchange1, waited an hour, then sent it to Exchange2. Is it not reasonable that the price would go up? So why is it suddenly so controversial when it happens on shorter timeframes? Why should RBC be able to make big trades and not have to deal with price impact? Why should market makers be left holding the bag?

Now whether or not HFT adds more value seems debatable. But it seems to be a fairly harmless result that naturally arises.

HFT has been around for a while longer than Reg NMS: http://www.bizjournals.com/kansascity/stories/2005/01/17/sma... describes an major player doing something that sounds similar to modern HFT in Jan 2005, presumably they'd been trading that way for a while.

HFT. It's basic automation. Taking (slow, expensive) human decision makers out of the loop drastically reduced costs.

You cannot really separate the two. If you support electronic trading, no matter what artificial restrictions you put on speed, the first person to get his order in will win, they will always compete on latency.

If you allowed arbitrary precision (or even 8 decimals) then there'd be a bit less emphasis on latency, right? As is, a penny is worth quite a bit so everyone has to queue up at the same price making speed win.

No there's a minimum spread that will exist because market makers won't make markets so tight that they are losing money. If they can't make money on the spread, then what's the point of posting that spread?

These discussions always happen whenever HFT is brought up. I've thought a LOT about how to remove a latency advantage from this game, and I just can't find a way to do it. I'd love to be proven wrong though. (And actually, so would the HFT shops - they hate the speed race).

What stops an exchange from quantizing to, say, a second or so? Use proportional matching, or something even more clever like POSIT does. I'm sure there's a reason that this simple idea doesn't work but I haven't figured it out yet.

As far as more decimals, it doesn't sound plausible that all the market makers are gonna come up with the same price to 8 decimals. So the time priority will give way a bit to price priority.

> What stops an exchange from quantizing to, say, a second or so?

That wouldn't be fair, an order book is a line, such a rule would allow people to cut line and get shares at better prices before you allowing them to drive the price up or down even though you were in line before them.

> As far as more decimals, it doesn't sound plausible that all the market makers are gonna come up with the same price to 8 decimals. So the time priority will give way a bit to price priority.

That's how it used to be, compete on price first, but a minimum decimal put in place by law, the sub penny rule, to try and dampen HFT, it's only made latency more important since HFT traders can no longer compete on price. And he's right, there's always a minimum price below which there is no profit, so whoever posts that price first wins. Orders compete on time and price because that what is fair. Latency is inherently part of fair trading, you can't remove it and keep the market fair.

> That wouldn't be fair, an order book is a line, such a rule would allow people to cut line and get shares at better prices before you allowing them to drive the price up or down even though you were in line before them.

Well, I think the problem is less that the book is a line and it's not fair (the definition of fair would change in this system, it would be fair that you had the same chance as everyone else in that time frame), but more that you open yourself up to different problems, such as the larger players being able to split their orders up in to many smaller orders, to increase they chance a large percentage of their orders will go through before any supply has dried up.

There's actually another industry that has to deal with these problems, and especially with pecieved fairness. Online event ticketing. Ticketmaster runs a queue. AXS runs a randomized pool. Both are gamed. Interestingly, I think some brokers actually do colocate close to TM servers, which makes the parallel fairly obvious.

Online event ticketing could get rid of scalping completely, by auctioning off the tickets, if there's excess demand.

That wouldn't eliminate the problem. Demand changes over time, and supply is fixed. There are quite a few ways to invest in tickets. One is to buy all the tickets on an event that is expected to sellout immediately. Another is to find an event that you expect to sellout at some time before the event that will allow enough time for demand to build, and you buy tickets before it sells out. In the end, ticket prices change over time, therefore there will be a market. Even if the primary seller (e.g. TicketMaster) raises prices, if you bought at a time when they were cheaper, you can resell at a profit.

Event ticketing is a market, and as such there will be people that want to invest (even if it's just fans who intended to go and then decided they couldn't, or the money they could get for the tickets is too much to pass up). Scalpers (brokers) provide a service, liquidity. Often they go too far, and use technology to get tickets in unethical ways (ignoring site policies, bypassing captchas, using software to tie up thousands of tickets they have no intention of buying, etc). But the fact that they continue to exist is because there is demand (and now TicketMaster is in on the game as well, their TM+ service allows you to resell tickets on their site along-side their own, unsold tickets).

This isn't true. I've been reading up on this, and there are markets that sorta do exactly this. "Call markets" instead of continous markets, I think? (Or a hybrid with "crossing" markets?) One example is POSIT where you submit orders and the matching rules are not a time line, but use other criteria (price, size of order). Even some continous markets give size priority. And to the extent you can "cut in line", that's the whole point of a quantized trading period. Instead of rushing to the best price, anyone that orders at the same price will get filled equally. This kills the need for speed. (The trick is probably in making cancels take place in the next quantized period.)

As far as sub penny, presumably different firms would have different costs, so the level of profitability would be different. While dominated by exchange fees (or rebates), at 8 decimals there's still room to compete. (Perhaps with new order types that allow displaying your quote rounded, so as to not trigger another feedback loop where firms keep lowering by 0.00000001.)

You can only use other criteria, when the others differ. For the same size and same price order, what possible argument can be made to fill the last arriving order first and still claim a fair market? At some point, latency always matters. If you randomize so latency doesn't matter, you just create a system where those with more capital get more orders. However you want to "fix" it, you're still going to be making a system that isn't fair to someone. Most people would agree that waiting in line is the most fair option IMHO, and that's proven out by normal human behavior in the day to day world.

What possible argument? Simple: This exchange works on a 5-second quanta. All orders arriving within a quanta are treated equally. How is that unfair at all?

Like I said, there are systems that do exactly this.

It doesn't work though. Lets say there are 500 lots bid at a price and 700 lots offered at the same price. Which of those 700 lots on the offer gets filled once the 5-second call occurs?

Like any other pro-rata system? How is this difficult to make work? "Call markets" do exactly this. POSIT does this. Even NYSE does this in some cases to give priority to specialists.

> This exchange works on a 5-second quanta. All orders arriving within a quanta are treated equally. How is that unfair at all?

It's unfair to the trader who put his order in first just as it'd be unfair to walk up to a sales desk and cut in front of people who there before you. All this does is slow down trading, it doesn't make it any more fair and now you can play the latency game for putting your orders in at the last second while allowing more time to analyse the data. 5 seconds is a long time to a computer.

That such systems exist is beside the point, they are less fair than simply allowing the fastest trader to trade first.

You have a weird notion of fairness. Why reward investment in low latency above all other investment?

In any case, quantising to eg 1s, allowing arbitrary sub-penny bids, and fulfilling by prize and using time as a tiebreaker would be `most fair' in some sense:

The counterparty gets the best price. And people are only able to `cut in line' if they make a better price.

(I don't think the fairness of waiting in line is a fundamental thing. It's just one way for humans to allocate scarce resources, and it's not very economically efficient---it just works well with human psyche and social norms.)

> You have a weird notion of fairness. Why reward investment in low latency above all other investment?

I didn't make that claim, best price should be ahead of place in line. But place in line is the next most fair thing.

> In any case, quantising to eg 1s, allowing arbitrary sub-penny bids, and fulfilling by prize and using time as a tiebreaker would be `most fair' in some sense:

Remove quantising to 1s and I agree. Quantising makes it in no way more fair and removes the ability to use time as a tiebreaker. If time is the tie breaker, you don't need to quantisize as it adds nothing.

> (I don't think the fairness of waiting in line is a fundamental thing. It's just one way for humans to allocate scarce resources, and it's not very economically efficient---it just works well with human psyche and social norms.)

It works well with the human psyche and social norms precisely because it fits our innate ideas of fairness, and thus fairness is a fundamental thing. It's so fundamental in fact that it's evolved into us as experiments with monkeys and fairness show it to be innate.

The minimum spread isn't a fixed quantity, different traders will have different minimums. Right now there is a price floor, below which the price cannot fall, which is bad for all those who want to transact in the stock market.

As for the speed race, why not limit transactions so that they can only be executed on the dot, once every $TIME_STEP? You won't be able to see what orders others have been put out, so you don't gain that much of an advantage by sitting on a very low latency connection.

This is correct. In ultra-liquid stocks, with a finer tick increment, natural buyers and sellers could undercut the market-makers' spreads since they aren't concerned about making short-term profits. Market-makers could also compete with more diverse pricing.

It is not necessarily a bad thing to have a price floor. Removing the floor is a good thing for the parties that meet at the tighter spread, but there is an externality where somebody who wants the shares nearly as much has their order go unexecuted despite providing price discovery to the market by showing their order. Look at GOOGL for example. It is $700 with a $0.01 price increment. Somebody can jump in front of your order by paying just 0.0014% more, there is little incentive to show limit orders on the market, which this makes the order book very thin. There is probably a better balance where the tick is large enough to avoid people jumping your order, but small enough that traders can compete: https://www.sec.gov/comments/4-657/4657-40.pdf has more comments on these effects.

You can trade in a single price auction in almost every market worldwide. A large amount of volume in US equities trades in this way, so people already have this option, but many choose not to use it.

> As for the speed race, why not limit transactions so that they can only be executed on the dot, once every $TIME_STEP? You won't be able to see what orders others have been put out, so you don't gain that much of an advantage by sitting on a very low latency connection.

Sure you do, you're still first in whatever line there is; latency still matters no matter how much you try and slow execution speed down. Latency isn't about the speed of trading, it's about being first in line regardless (speed to the sales counter) of what speed the trades are limited to.

I should have been more clear here: all orders that come in in the time interval are resolved at the same time (or if that isn't possible, the order is randomized). It now doesn't matter if your order came in a minute or a second before the deadline.

Even if you randomize the order, you just open yourself up to gaming the system in other ways. Larger players with more resources can split large orders into many small orders and they be assured a certain percentage will happen earlier, and certain percentage will happen later, but if you have reason to believe that there isn't enough supply at the desired cost, you can use this to your advantage.

Randomization usually doesn't work in solving such issues. But making trades execute simultaneously doesn't need to randomize. You can just give everyone a pro rated amount. A firm that submits large order gets more shares filled. This isn't really game able is it? If a firm quotes a price for 10x the shares, the run the risk of actually trading all the shares at that price!

(There's also more complicated rules possibly, like trades only execute when there's a certain demand, or other liquidity based measures. Not sure they're good but it's an option.)

I'm not sure how well a pro-rated amount would work. Just because I'm willing to pay X for Y quantity of Z, doesn't mean my order makes sense if the quantity is scaled up or down, so forcing this on all market participants might have some interesting effects. Do you have other markets where this is the case, I would be interested in looking at this in a slightly less theoretical context before I felt comfortable with how the behavior would settle (not that I'm all that knowledgeable about financial markets in general).

> (There's also more complicated rules possibly, like trades only execute when there's a certain demand, or other liquidity based measures. Not sure they're good but it's an option.)

Well, I'm not sure any set of rules isn't able to be gamed in some manner. Larger players will have an advantage as long as rules don't specifically penalize them, as they have all the capabilities of smaller players plus extra capabilities provided by economies of scale and dedicated resources. Even when you specifically penalize specific participants to reduce "unfair" advantages, I think what generally happens is that some participants move into the niche just outside those penalty definitions, to eke some of the prior advantage in the new grey area. This seems to be the norm in almost all regulation, IMO. Not that the regulation is always useless, but you can't game a system without rules, and regulation by it's nature provides more rules to look for specific ways to exploit.

Just one example: http://www.itg.com/liquidity/posit/#POSIT

Or even the NYSE: https://www.nyse.com/markets/amex-options/order-processing

Cliff Asness on how HFT has benefited retail investors.

How HFT has changed the allocation of the pie between various market professionals is hard to say. But there has been one unambiguous winner, the retail investors who trade for themselves. Their small orders are a perfect match for today's narrow bid-offer spread, small average-trade-size market. For the first time in history, Main Street might have it rigged against Wall Street.

http://www.wsj.com/news/articles/SB1000142405270230397830457...

Matt Levine's followup post. http://www.bloombergview.com/articles/2014-04-02/high-freque...

Except that most of Main Street owns stocks through mutual funds, which make large trades on their behalf...

Main Street mostly owns shares in index funds or other funds without alpha. They're much better served with faster price discovery.

Meanwhile, nobody on here complaining about those mutual fund fees that are orders of magnitude bigger than HFT haircuts for the savers themselves :)

The only reason retail orders actually get matched up well is because they're flagged as retail, are known to contain less information, and are a statistically good bet to transact against. Fundamentally, this was around before HFT and still is. The funny thing is that even if it's helping them in this small way, the whole idea of retail investing is some of the best marketing in the face of contradictory facts I've ever seen.

Please elaborate. If an order makes it to an exchange and that exchange has a bunch of small orders at all times, how is the retail-ness of the order relevant at all?

I understand that at other parts of the order chain this might matter. But inside the matching engine there isn't "retail priority".

Inside the matching engine, there's a sitting limit order on the limit order book flagged as a retail order, just off the bid/ask. Whereas you may not have taken the other side of that resting order before, you might now because you know it has extra informational value attached. Dealers will pay exchanges for the right to access this information, either outright or in reduced liquidity rebates they would normally otherwise receive (normally outright to my understanding).

How does that provide value?

I have no personal experience in this, but I've heard that the order book is now very "shallow" due to HFT, such that it's hard to sell/buy a meaningful quantity of stocks at the market price. Does that make sense?

The bid ask spread is a measure of how far behind you are the moment you buy a stock. If the ask is $10 but the bid is $9.75 you've lost 0.25 per share the second you make a trade. If the spread is only $0.01 (as is now common) you've only lost a penny. So smaller spreads are great.

It's true that markets now respond faster to large sales so if you're a hedge fund trying to unload $BIGNUM shares of MSFT you'll have a harder time. But that's good for everyone else. If that hedge fund has proprietary information that MSFT is undervalued you want that information getting out to the market as fast as possible leading to more efficient prices.

The bid ask spread is a measure of how far behind you are the moment you buy a stock. If the ask is $10 but the bid is $9.75 you've lost 0.25 per share the second you make a trade.

That makes some sense, but who have you lost it to?

I was thinking about it the other day, and it seems that the "liquidity is good" argument can make some sense if people are not willing to make a transaction immediately. If I want to sell a bag of apple for 1$ _right now_, and someone else is willing to buy that bag for up to 1.50$ in a week, then I can appreciate that there's a middleman (providing liquidity) willing to hold the bag for me during that time and that he makes some profit from the spread. However, if the buyer and seller are both willing to make the trade at roughly the same time, then it's much better if they do it themselves (splitting the 50 cents of value amongst themselves) than if a middleman gets into the picture.

More or less the argument made here: http://smbc-comics.com/index.php?id=3890

You've lost it to a market making who is charing you for providing a service. Whenever you trade with a market maker you are buying liquidity (which used to be expensive but is now a lot cheaper). You're saying "I want to take the price you are offering right now and am not willing to take the risk that someone else will never come along to trade with me at a slightly better price."

If you don't want to trade with a market maker you can avoid purchasing their liquidity by placing a limit order that doesn't cross the spread. You'll then wait until someone else comes along to trade with you. Of course, you're taking a risk that the stock price will move away from you and you'll never trade. Normally this is the risk the market maker takes for you (which is why you pay him) but you can always do it yourself if you want.

In that comic the first caveman is getting a price that is right in front of him. He's not willing to wander around hoping to find a better price in another cave. That's a service that the 2nd caveman is performing for him. That's why the 2nd caveman gets paid.

The key thing to realize is that market makers don't just "get into the picture" by muscling their way in when other people don't want them there. Think of them off to the side selling liquidity for anyone who wants to walk up to them and buy it.

And you haven't really "lost" anything since you never had the ability to trade at the theoretical future fair price to begin with--nobody does. As you mention, your best alternative is to work non-marketable limit orders. This, too, has a real price: adverse selection, risk of missing your fill, connectivity costs, time spent monitoring multiple markets and updating your order. If you're a large investment bank or fund, your cost to do this may be lower than crossing the spread. For the average investor, no way.

Indeed, I agree that "lost" isn't the best word choice here. Instead "spent" would be better. You've spent money paying for a service provided by the market maker. And due to the wonders of automation, the cost of that service has been drastically reduced.

Thank you for the detailed explanation, I get the point about liquidity and traditional market makers, that makes sense.

But isn't the point about HFT (at least, the part that is described in the comic) being "insider information" reasonable? Some people are willing to sell now for a low price, and some people are willing to wait longer for a better price, but I don't think anyone is unwilling to wait 1 millisecond for a significantly better price, no? Does liquidity on the millisecond scale make sense?

> He's not willing to wander around hoping to find a better price in another cave. That's a service that the 2nd caveman is performing for him.

I can see that, and in some contexts it's a very valuable service.

But suppose I'm caveman 1. I know that CaveBob is a middleman. Everytime he comes to see me, it's because he knows something I don't, and I end up regretting my trade with him because a few hours later, I invariably realize that I could have made a much better trade. Then I'll stop trading with CaveBob, of course, and I'll naturally bump into those needing my meat a few hours later.

In a stock exchange, I can't stop trading with CaveBob's, as far as I know. If a stock market without market makers and HFT rises by 5% every year, then by participating, I can hope to make as much if I'm average at trading. In a market with HFT, they will be taking part of the 5% with very low risk (through technological advantage), so it's not clear what's left, and the decision to participate is not so clear.

> but I don't think anyone is unwilling to wait 1 millisecond for a significantly better price

Two things here:

1) Is 1 penny better when trading a stock that costs $50 bucks really significant?

2) The issues is that you aren't guaranteed to be able to trade at a better price 1 millisecond from now. Maybe someone comes along and takes your price or maybe the stock moves away from you and you can never ever get that price again ever (or even the price you could have gotten from the market maker). It's a risk. And it's a risk you are free to take, or it's a risk you can pay the market maker to take for you. Your choice.

> In a stock exchange, I can't stop trading with CaveBob's, as far as I know.

You can absolutely stop trading with CaveBob! You can do this, as I said, by "placing a limit order that doesn't cross the spread." So if Bob is selling $STOCK for $10, and buying for $9.95 and you want to buy you can place a limit order to buy for $9.95. CaveBob won't sell to you at that price but maybe someone else will come along later to take it from you.

But again, it's a risk.

It's also a risk with an adverse selection problem if you think about it.

> You can absolutely stop trading with CaveBob! You can do this, as I said, by "placing a limit order that doesn't cross the spread." So if Bob is selling $STOCK for $10, and buying for $9.95 and you want to buy you can place a limit order to buy for $9.95. CaveBob won't sell to you at that price but maybe someone else will come along later to take it from you.

Does that solve the issue? Suppose that I'm usually selling meat at 10$/kilo, but the buying side is at 9.50. Then when there's a catastrophe (some other vendor burned down), I could sell them at a higher price, 20$/kilo. If the news reach the HFT 1 millisecond before it reaches me, they can buy it at 10$ from me, then sell it at 20$/kilo for a big profit, but I would much rather have sold it at 20$/kilo myself. Is there any limit order that prevents that?

No. If you put in an order you're willing to sell at $10 (unless you notice the news about the catastrophe and cancel your order fast enough)[1], you're selling at $10. But if, instead, you just take the market price (as provided by the market maker) whenever you want to sell then as soon as the information enters the market you'll end up selling at $20 (or $19.95 or whatever).

This is one of the other services that you are paying the market maker for: price discovery. If you want you can do this job on your own. You can expend a lot of effort to keep up with all the latest news so you know what price you should be selling at. Or you can let market makers all compete to buy from you at the best price taking into account all of the latest information. In your meat example, keep in mind that there isn't just one trader who will buy from you for $10 to sell at $20. They're all competing to buy from you for the best possible price so they will very very quickly bid the price up to something very very close to $20.

This is why most people love market makers! If you trade with them you can be assured that you are (more or less) getting the best possible price taking into account all of the latest information.

Going back to your comic, this is what's happening in frame two. The ship arriving (for some unclear reason) is changing the price of neckerchiefs. It's generally to everyone's benefit if this new information enters the market as fast as possible. Rather than thinking something shady is going on we should think this is great, and feel good about paying them a few pennies to keep the information flowing quickly.

[1] Which is the exact same risk liquidity sellers normally take! This is why, in flash boys, Brad Katsuyama sees the price move when he tries to make a large trade. All the people selling liquidity are worried that there is some new piece of information they don't know about yet so they move their prices so they don't get run over.

This is why you sometimes see complaints that so many orders from HFT traders are cancelled. They're constantly monitoring all kinds of news sources so that they can keep their prices accurate. The fact that they're cancelling orders isn't a sign of shadiness, it's just a sign that prices are moving.

> Or you can let market makers all compete to buy from you at the best price taking into account all of the latest information. In your meat example, keep in mind that there isn't just one trader who will buy from you for $10 to sell at $20. They're all competing to buy from you for the best possible price so they will very very quickly bid the price up to something very very close to $20.

If I put a market order to sell meat at the market price of 10$ at 10AM (because I just produced a bunch more), then the first HFT to get the news of the catastrophe at 10AM + 1 microsecond will buy them from me at that price, but if I had gotten the news at the same time as they did, I would have withdrawn my market order and refused to trade at 10$.

Maybe HFTs/market makers serve many different purposes and we're talking past each other. I agree with the liquidity argument, to the extent that I understand it. Is getting the news slightly faster really providing value?

> You can expend a lot of effort to keep up with all the latest news.

The news that my competitors burned down doesn't require a lot of effort to keep up with in the age of the Internet, but it does require large amounts of capital to get it 1 microsecond before everybody else.

If trades were only allowed every 60 seconds like the comic suggests, to allow everyone time to process the news, what value would be lost? Should society rejoice when an HFT gets 10% closer to the speed of light?

> If I put a market order to sell meat at the market price of 10$ at 10AM

Yes, at 10AM (before news happens) you can take the $10 price. And, unfortunately for you, the world might change 1 microsecond later. And it sounds bad when you say "1 microsecond" but what if you say "5 minutes" or "1 hour" or "1 day"? You're probably going to feel just as bad about missing out on a better price even if the timescale is a human one instead of an electronic one.

There is no way to avoid this. The world can change after your sell your meat. It might change 1 microsecond later or it might change a week later. You chose to sell when you chose to sell. You can't say oopsies and complain if the world changes after you made your decision. The fact that the world can change after you make your trade has absolutely nothing to do with high speed trading.

Very high speed price discovery actually works in your favor here! At least the price jumps to $20 really really fast after the news now. Without computers it might take 5 or 10 minutes (or whatever). The sooner the news is incorporated the smaller the time window in which you can accidentally screw yourself.

While thinking about this, keep in mind also that the price might drop (maybe there is a bumper crop of cows). If you sold now for $10 but then the price dropped to $5 you'd feel great. When you sell now you're taking the risk of a price change (in either direction) off the table. You can't on one hand, avoid a price drop, but on the other hand think that you should get the benefits of a price rise.

> the first HFT to get the news of the catastrophe at 10AM + 1 microsecond will buy them from me at that price,

This is actually wrong and not what happens. The trade you're talking about happened at 10AM (before the news). You put in a market order which means "take whatever price is being offered right now." not "put my meat out for sale and wait for someone else to trade with me."

So no trade is happening at 10AM + 1 microsecond (after the news).

> If trades were only allowed every 60 seconds like the comic suggests

Ah yes. Everyone eventually hits on this idea, but it doesn't work. What happens if, at a given price, the # of buyers and sellers doesn't match up at the time you want to execute all the trades?

1. Execute the trades that put in their orders first. Well then you're back to where you were before.

2. Execute some % (less than 100%) of one side of the trades so that everything matches up. But then you create a very unstable situation. If I think that maybe only 50% of the trades are going to execute you've incentivized me to say I want to trade 2x of what I really want to trade. But wait, everyone else is also thinking the same thing. So maybe 4x? Repeat ad nauseam. This is a very unstable game-theory heavy sort of situation that's actually pretty dangerous.

And really even if you do this you haven't removed the need for high speed computers & communication. Lets say that a block closes at precisely noon. I definitely don't want to input my trade at 11:59:01. What if there is news in the next 59 seconds? I really want to wait until the last possible microsecond to input my trades for the block right? So...we're back to where we were before anyways.

The real world is continuous. It's not really possible to make markets based on the real world operate discretely.

That's totally right, I blundered my hypothetical scenario. Let me try again.

1. 10AM catastrophe happens

2. 10AM + 1 ms: news reach some HFT

3. 10AM + 2 ms: I put out a market order, ignorant of the news

4. 10:01AM: the news reach me

That's what I meant. But I suppose that given the competition between HFTs, the market price should have reached 20$ by then, so I get to sell it at the better price.

Does that mean that HFTs do not generally make money from getting the information before everybody else unless some non-HFT trader used a limit order?

If some HFT holds a lot of some company Y and gets the news early that Y had a bad quarter, won't they make money from that by selling Y to a non-HFT investor who hasn't received the press release yet? But maybe if no one but HFTs use limit order and everyone has a decent-speed internet connection, this is a vanishingly small occurrence?

Ah yes. Everyone eventually hits on this idea, but it doesn't work. What happens if, at a given price, the # of buyers and sellers doesn't match up at the time you want to execute all the trades?

1. Execute the trades that put in their orders first. Well then you're back to where you were before.

All good points. Although there's probably a game-theoretic solution to that...

> Does that mean that HFTs do not generally make money from getting the information before everybody

In general this is somewhat true. Market makers would actually love it if prices were very stable. Just sit there and buy at 9.99 and sell at 10.00 and make a penny a trade all day long. They need to respond very very quickly not so much to make money, but to avoid losing money when prices change.

Makes sense. Thank you for the discussion, I appreciate it.

You're welcome!

I don't buy that the order books are shallower than previously. I'd love to see evidence of it. People also complain that 'as soon as the markets move, the HFT firms get out of the way and the liquidity disappears.' As if the humans in the pits were making markets during big moves and getting run over (losing money) during high volatility. Give me a break.

"A U.S. Government Accountability Office study in 2005 showed that bid-ask spreads compressed by as much as 80% immediately following decimalization, well before high frequency trading came to dominate markets, and well before this author's time frame."

http://www.amazon.com/review/R28Y8AI4PIPYZ2/ref=cm_cd_pg_nex...

FWIW: I used to work at the predecessor to Virtu Financial with Peter Kovac, who wrote an excellent counter argument book to flash boys.

http://www.amazon.com/Flash-Boys-Insiders-Perspective-High-F...

He's one of the most honest and intelligent people I've ever had the pleasure to work with in my entire career. He was at Madison Tyler while I was there roughly 4 years until the Virtu merger (at which point many of us left).

This book, for what it's worth, is fantastic. If you're interested in nuts-and-bolts how-stuff-works, and less in the screenplay-friendly narratives that Michael Lewis wanted to peddle, it's a much better book than Flash Boys itself was.

Yep, I mention that in the first paragraph :). I loved the book. It was fantastic. Both for explaining stuff, as well as totally smacking down Lewis's nonsense.

It left me wanting to learn more. I'm currently reading Trading and Exchanges, as well as Market Microstructure in Practise (I wanted a newer book as stuff has changed in 2002 and 2010.)

Here's a lengthy critique of the Kovac book alleging it misreads Lewis's point:

http://www.amazon.com/review/R28Y8AI4PIPYZ2/ref=cm_cr_dp_tit...

"Kovac misfires by supposing the "front-running" condemned in "Flash Boys" refers to a particular kind of abuse, and not surprisingly it's the kind he plans to refute. In common usage there are at least two meanings to the term. One common meaning is as shorthand for "front-running demand," which is what happens when someone tries to figure out whether your mutual fund or your pension fund is buying or selling stock, realizes that it will change supply-and-demand dynamics for the stock, affecting the price, and front-runs that demand to profit.

Imagine if someone made a reasonable guess you needed milk, raced ahead of you to the only store in town to buy up all the milk before you got there, and then offered to sell you milk at a higher price than the store sold it to him; this is one example of front-running demand, just as described in "Flash Boys." (It's also called "trading ahead.") Another common meaning is "front-running a customer order," a type of insider trading, where a stock broker believes his own customer's order will change supply-and-demand and tries to profit from that; this is what Kovac apparently intends to refute, and it has nothing at all to do with "Flash Boys.""

This is a review written by someone who, by their own (repeated) admission, hasn't read past the introduction of the book; further, the premise of the review is that Kovac doesn't know what he's talking about vis a vis trading, which is something we know to be false.

Correct. Peter Kovac wrote the entire realtime risk and clearing system that is now live at Virtu (formerly California Applied Trading Science which split into EWT and Madison Tyler, which became Virtu. This is all public knowledge if you dig deep enough). Again, the source is I worked with him for several years.

"Front-running demand" is just anticipating where prices will go. That's what all competitive trading is. At its core, that's what business in general is. Observing supply and demand playing out through public information is completely natural.

The reason we have terms like "front-running" and "insider trading" and laws against those practices is because of ethical obligations between people. If I see a lot of women wearing blue dresses this season, and buy more anticipating they'll sell for a profit, that's just business. If my accountant also works for the shop on the next block and tells me they're about to buy lots of blue dresses this season, that's unethical.

> ..in "Dark Pools", they discuss how some of these traders would call up Nasdaq wondering why a certain trade did or didn't work how it did. And they'd find out all sorts of strange rules. Which makes me wonder, wtf? Are Nasdaq's rules hidden? Or were people doing this as a living just too lazy to read the specs?

Sometimes the functionality is obscure or poorly documented, so it's difficult to understand how it will work, or you make an assumption that turns out to be wrong. Sometimes, different order types interact in unexpected ways (these are effectively complex systems).

Sometimes, the exchange just gets it wrong. On a number of occasions, I've investigated an incident and it turned out that there was a bug at the exchange's end.

These are Patrick's levels. I'm writing up Erin & my levels now. If you liked Microcorruption, well...

Here's, like, I don't know, a taste, or whatever:

https://www.dropbox.com/s/8l2d814o1cuiumv/Screenshot%202015-...

Almost every fun project I've worked on required me to learn something new. Heck, the last level of the Stripe CTF3 required me to learn how to fake a distributed fault tolerant SQL database!

> The game right now - as this article, this mail shows - requires you to learn about finance systems before you even start playing the game. It's a game where the rule book is - from this pov - too long and already uninteresting.

Given that Starfighter's business model seems to be helping filter programming talent for recruiters, filtering out developers who aren't interested in programming financial systems is a good first step. My day job involves understanding the finance "rule book" and translating part of it into something that developers can understand and build to. But, even with my translation, the developers still need to be capable of understanding a certain amount about the "rule book" so they can delivery code/functionality that actually fulfils the requirements.

If a developer isn't interested in (or capable of) doing that, they're not going to be of interest to someone who's looking to recruit developers to work on finance systems.

That is not our business model. We are recruiters (or at least: we plan to figure out how to be). We do not provide information to outside recruiters or hiring managers.

Aren't you using the CTFs you run as a way of filtering out the best talent?

We intend to get in touch with very talented players. We’ll be interested in talking about the game, about what you’ve built, and about what you want to do in the future. If your future goals include “Get a better job!”, we can help make that happen. If not, no worries.

We have agreements with a small number of companies to find qualified engineers and introduce them to the companies. These clients pay us for introductions which result in placements.

He didn't dispute that, just that their model does not entail sharing your information. They aren't planning to be middle-men to recruiters, but to be a specialized form of recruiters themselves.

I'm with you on this one. When I first heard about this and signed up for the newsletter, I was genuinely excited. However, the more I hear about it, the more disinterested I am.

Reply on news.ycombinator.com