A veteran programmer explains how the stock market became “rigged” | Hacker News Reader

A veteran programmer explains how the stock market became “rigged”

washingtonpost.com

92 pointsxmpir120 comments

Here's an animation showing the SIP vs Direct Feed issue: http://www.nanex.net/aqck2/4599.html

Much of the games being played no would no longer work if the SIP was the defacto source of "real-time" prices (as specified in Reg NMS). Sure, there would be new games, but we have rules specifying the SIP as the key to Reg NMS. At least there would be a decent audit trail if rules were being followed. People would be able to detect delays (currently obscured by the changing time stamp) and many games would be exposed.

It's helpful to actually read Reg NMS - a link is provided on that animation page.

P.S. I'm the veteran programmer in that article.

Can you describe in detail what "games" this enables and how often they actually happen in the data? You claim SIP slowness enables a fictional trade called "latency arb" where HFTs can trade with orders at stale prices, for example:

-A user is pegged to mid at a pool where they think NBBO is bid: 20.01 ask: 20.02

-An HFT with direct feeds sees the market go to 20.02 bid 20.03 ask

-The HFT buys the midpoint order on the dark pool at 20.015

-User regrets trading since they could have sold 20.02

How often does this really happen and how is it really any different from the following scenario:

-A user is pegged to mid at a pool where they think NBBO is bid: 20.01 ask: 20.02

-An HFT with direct feeds sees the market as 20.01 bid for 100000 shares and 20.02 ask for 1 share.

-The HFT predicts the price will move up with near certainty and buys the order for 20.015

-User regrets trading since they could have joined small 20.02 ask and traded quickly

If the user or broker is pegging their order to the midpoint and not adjusting it based on market conditions, they're going to trade at disadvantageous times no matter what. I'm not sure how pricing off the SIP makes a big difference. It's not HFT's fault that the broker sucks and can't trade well.

thanks for cutting through all the bullshit and showing the real technical facts. I'm tired of listening to HFT defendants pointing out the necessity of their service in their own overly generalized and useless models.

This is not a case of the markets being rigged. This is a case of regulators (actually congress) failing to understand the CAP theorem.

But for obvious reasons, an article on WAPO can't use words like "partition tolerance".

The problem is that the SIP must be available and it must be consistent. As a result it sacrifices partition tolerance - high latency (which is equivalent to a network partition) causes things to break sometimes.

So for all the conspiracy theorists out there, here is the fundamental question: which two should be provided for the markets not to be "rigged"?

That's a false dichotomy. Even something seemingly binary as CAP is actually subject to many assumptions and preconditions. Change one and the question is of whether CAP still holds for the system, is relevant, etc.

For one example, partitioning comes in degrees. You can accept partitioning up to a certain threshold and only then choose to sacrifice availability until partitioning crosses below the threshold again.

Ok - what complicated tradeoff do we make for things to count as "not rigged"? Please be specific.

Don't sell privileged access to the stock exchange and don't enable abuse of the trading system.

Can you tell us what you mean technically by "privileged" access? Is there a threshold in dollars you have in mind for what is an acceptable fee for connectivity, or is it something else?

Is it a matter of the stock exchanges profiting from this, or the fact that it's possible at all?

I can't see how it can ever be possible to give everyone the same information at the same time. Latency will always be there, and renting rack space in a building near to the exchange will allow the information to reach you before the opposite coast of the country. Although this might mean that the owner of the neighboring building is the one making money, and not the stock exchange itself.

Firstly, I wasn't arguing that the market is rigged: I was arguing that your argument doesn't hold, because there is actually an infinite amount of options to choose from, instead of just two (or one, as you are implying).

Secondly, you are not being constructive in your questioning, as you have already marked any answer I would give as 'complicated' (as if the system isn't complicated already).

Thirdly you are unfairly asking for 'specifics', where you yourself are arguing in generalities. Why won't you be specific: explain why the market is not rigged? 'The market' is not 'innocent until proven guilty' or 'efficient until proven otherwise'. You claim the market is fair: prove it. Please be specific.

Fourthly, as long as you are asking: I think we don't need the amount of Availability the system currently has. Quote stuffing is currently possible and you can get rid of that by reducing the availability of the system. Any form of rate limiting is a deliberate sacrifice of availability to achieve some other goal. Rate limiting the number of orders per licensed party, per instrument, per unit of time is eminently doable.

In conclusion, from the whole of your posts here on HN, I think you are way too infatuated with capitalism and unable to see her for the bitch she sometimes is. Your response was uncivil and uncalled for.

I didn't mean to imply there were only two possibilities. I merely wanted to ask what "not rigged" means to you? As the current article shows, there are real technical details necessary to figure out and the CAP theorem limits what can be done. It isn't a simple question to answer, and I personally lean towards "the SEC did a generally good job".

I don't know why you feel I was "uncivil" - I merely asked a question. However, please accept my apologies.

If you want to see assorted defenses of specific aspects of HFT, I have quite a few on my blog. I just posted a new one today. If you look way back, you'll also find a tutorial explaining the basic mechanics of HFT. I'm planning to publish an explanation of "quote stuffing" (aka screwups in your algorithm) shortly, as well as a takedown of Mark Cuban's idiotic article on the topic.

http://www.chrisstucchio.com/blog/tags/high%20frequency%20tr...

As I read the article, what you claim happens "sometimes" is rather "most of the time, let us say 99.999999% of the time", which is not so acceptable because it almost always (i.e. always) benefits HF traders.

So, I had rather it would be designed in a different way which does not benefit the people with more money. For example, sacrificing latency so that there is no partition. This way, everybody has the same information at the same time, which is the basic tenet of an equal market.

Also, having different routes for different actors (as per the graphs) does not look like a road map to equality.

BTW: I do not get why you have been downvoted although in some sense your comment might be seen as a misunderstanding of the text. But I am surprised.

>So for all the conspiracy theorists out there, here is the fundamental question: which two should be provided for the markets not to be "rigged"?

There is no need to make any complicated tradeoff. For it to count as non-rigged, you simply need a system where:

* You cannot pay to gain privileged access.

* Abuse of the system (e.g. quote stuffing) gets you punished.

I find it amusing that you call us 'conspiracy theorists'. These aren't allegations of a government cover-up of alien UFOs - it's mundane criminal behavior with a very minor twist.

To avoid payment for access, presumably access would need to be free. Who would pay for it, then? Should we have state-sponsored exchanges? The networks and rack space required to let people trade in near-time are not cheap (and using the internet doesn't really work because people would go right back to paying for lower latency, higher throughput, and/or better reliability).

We could make the markets go super-slowly, but you know what will happen then? Correlated securities in other countries will trade at high frequency and dominate the price discovery process for "our" listings. That won't help our economy, it won't help reliability, and it won't help us assess taxes.

>To avoid payment for access, presumably access would need to be free.

There is absolutely no reason to presume that. It could be one flat rate.

>We could make the markets go super-slowly, but you know what will happen then? Correlated securities in other countries will trade at high frequency and dominate the price discovery process for "our" listings. That won't help our economy, it won't help reliability, and it won't help us assess taxes.

If our intention is to kill HFT (and it probably should be), we could simply levy a very tiny transaction tax on trades. It would instantly render the practice unprofitable.

Retail and institutional investors wouldn't even notice the tax (although they would probably notice the fact that HFT isn't skimming money from their trades).

Thanks for pointing out the tax idea (a fairly popular one these days). It has some of the same issues as slowing the market down. For example, a transaction tax on one market will push transaction volume to other markets, like squeezing a water balloon. A lot of securities have correlated or even 1:1 equivalents in other jurisdictions (e.g. Europe vs. US), which results in a tricky situation. Basically, whatever market pushes their costs up first risks suffocating itself.

I'm not sure what you mean by "one flat rate." Exchanges already charge a flat rate for connectivity (the rate is per month, usually a couple thousand bucks or so).

>For example, a transaction tax on one market will push transaction volume to other markets, like squeezing a water balloon.

Actually the exact opposite will happen.

Only the HFTs will want to move away, the rest will want to move to such a market. For the rest of them, operating in a market that allows HFT is pretty much the same thing as operating in a market for lemons: https://en.wikipedia.org/wiki/The_Market_for_Lemons

Asymmetric information in markets means 'the bad drives out the good'.

Just as running an honest used car market will attract buyers and sellers, running an honest stock exchange will attract traders.

Then why hasn't a stock exchange that imposes a transaction fee appeared and pulled in all traders, leaving the HFTs behind?

This is what I can't understand about this stories, and makes me doubt the claims against the HFTs.

network effects work heavily against new exchanges. it's not like starting a SaaS to deliver email.

But there are already 13 of them that are regulated. At this point BATS owns 4 of them, the only reason to have 4 different exchanges is to have slightly different rule sets and fee structures. Any one of the existing exchanges (especially the small ones) could change their rule set or fee structure if they thought it would bring volume.

This is nonsense. There is a lot of competition in exchanges, a new one is sprouting up every few months. Not all of them are able to get Michael Lewis to shill for them, but a new ATS/Dark Pool/etc sprouting up is hardly uncommon.

You should consider asking yourself why most such trading networks actually go to great effort to encourage HFTs to trade on them. (Hint: no liquidity no customers.)

A transaction tax will increase spreads. That is a really big argument against that such an exchange would be popular.

HFT did not reduce spreads. electronic trading did. actually sometimes hft can be a liquidity taker and thus increase spreads.

I don't know where this meme started, but it's clearly untrue. Nanex likes to make this claim, but they claim "HFT" only started with Reg NMS in 2007. Tell me this doesn't sound like the most stereotypical HFT-style trading in 2005: http://www.bizjournals.com/kansascity/stories/2005/01/17/sma...

And it wasn't just Tradebot doing it. There were many firms already active with similar styles in the early 2000s like GETCO, ATD, Madison Tyler (now Virtu) and a bunch of smaller guys. HFT does not depend on Reg NMS, fragmentation, or rebates. HFT firms are super successful in futures markets with a consolidated limit order book and fees for both sides of the transaction.

Canada tried a transaction tax (technically a regulatory fee increase) in 2012. The net result is that retail lost, institutional investors won, and spreads increased by 9%.

http://qed.econ.queensu.ca/pub/faculty/milne/322/IIROC_FeeCh...

Everyone would notice a 9% increase in spreads.

If you believe HFT is "skimming money from their trades", can you explain the specific mechanics (i.e., what orders are placed when)?

Didn't Canada loose out because people went to other markets instead? I.e. not because of the tax/regulation per-se but rather because other markets didn't have that tax/regulation.

From what I learned from a guy who did HFT back in 2007 HFT skimming money from their trades, thats why they make more money when markets are volatile.

In some ways it's just a more sophisticated version of the traders who in the nineties made their money from each trade they made happen whether it went good or bad.

The more customers they go to investe, re-invest etc the more money they made.

Don't think thats a big secret.

"Canada" didn't lose out. Individual investors in Canada did, because the bid-ask spread widened.

If that happened because HFT market markets decided not to use the TSX anymore and instead haunt exchanges that were amenable to HFT, that's evidence that HFT was helping individual investors.

Thats like saying tax havens are evidence that paying less taxes is helping individual tax payers.

Sure but that is not really the point is it?

You're implicitly begging the question by assuming that HFT is bad, and therefore permitting it is bad, regardless of any further evidence. The point here is that if an exchange successfully removed HFT, and that materially hurt people, that is evidence that HFT isn't bad.

I'm simplifying a simplification, and I personally would be among the first to point out that this is still just a one-point sample of a horrifically multidimensional space. My point is not to specifically claim that HFT is good, a matter that I still have no personal opinion on, but to point out the question begging. If HFT is shown in fact to be a net gain for consumers, then you should not blithely carry on with the presupposition that it is bad.

I was replying to this:

*"...If that happened because HFT market markets decided not to use the TSX anymore and instead haunt exchanges that were amenable to HFT, that's evidence that HFT was helping individual investors..."

Which claims that because people move to places that are not regulated/taxed it's a proof that HFT was helping individual investors.

That was in return in the context of me asking whether the reason why the Canadian market lost out was because it did so alone.

In other words. If ALL markets included taxation would the same drop happen? Or in even other words. If something can't handle being taxed is that an argument for not taxing it? Wouldn't it just be evened out.

I am not assuming HFT is bad or good. I am actually trying to figure out what exactly it's good for.

I understand how it's good for those who trade on the market, what I have not yet seen is any examples of how that transcends to society as a whole. And until someone can give me obvious examples of that I don't believe it's irrational for me to be sceptic.

Didn't Canada loose out because people went to other markets instead?

Most Canadian stocks are not traded on US exchanges. So no.

As for "skimming", I'll repeat my question: can you explain the specific mechanics (i.e., what orders are placed when)? Note: if you believe FIX/OUCH has a "SKIM" command, can you link to it?

With regards to the mechanics. All I have is what was explained to me by someone who started doing HFT more than 10 years ago and from those traders I know who made money by trading the same stock several time a day and taking a small fee each time they did. Back then they just didn't have computers or the knowledge to do it fast enough I guess.

> If our intention is to kill HFT (and it probably should be), we could simply levy a very tiny transaction tax on trades. It would instantly render the practice unprofitable.

That would also kill a lot of mutual funds, exchange-traded funds, asset managers, and others. These depend on the availability of frequent rebalancing of portfolios at low transaction cost. Add in a tax to this and you penalize it significantly.

I forget if it was on The Daily Show or 60 minutes, but the "solution" the new exchange came up with was to actually slow down his trades by fractions of a second such they all arrived at the same time. Once he did that his order fill rate went up and the HFT got cut out. He did this while a 60km fiber optic spool

So "super slowly" isnt exactly what he was doing

It was the Daily Show

http://thedailyshow.cc.com/extended-interviews/o7grtj/exclus...

As I understand it, that link is only good in the US. So for everyone else, that is the extended interview of the April 1 episode. I'm sure it can be found somewhere.

Thanks for the link.

It works fine from Denmark. I've never had problems watching it from a non-US country.

Also, pro tip if you want to avoid ads between segments: right before the video gets to a horizontal line (ad break) on the seek bar, skip to the part right after the horizontal line. You'll skip the ad. Also: if you work as a web developer at Comedy Central, you never saw this comment.

GP: >>You cannot pay to gain privileged access. [my emphasis]

>>To avoid payment for access, presumably access would need to be free

(Also, high frequency trading can't have much to do with finding the price points of stuff.)

I don't have a horse in this race (I don't know the subject area), but most of that comment was a straw man argument.

I realise that was probably a mistake in reading, but even I can see the implementation ought to be somewhere between simple and possible: Publicise the algorithm for a random delay between a tenth of a second and (more seldom) ten seconds for all buys/sells. Or something similar.

Make certain to have good random number generators.

Edit: If it is a good thing (or possible) to do this I don't know.

So everyone needs to pay several thousand/month for direct access? I don't know why it's bad that people who don't need low latency are allowed to not pay for it.

"Quote Stuffing" is not the nefarious activity that Nanex portrays it as. It's generally the result of your algorithm screwing up. It provides no benefit to the trader and exposes him to risk - specifically, an HFT who is quote stuffing will always be at the bottom of the book. As a result, he won't get any retail fills, he'll only trade with informed traders who are buying up the whole book. It also lowers your fill rate, which can result in a trading halt or financial penalties.

As a result, most HFTs put effort into reducing quotes and making their algorithms more stable.

I shouldn't have used the term "conspiracy theorists". I just find this topic frustrating since there are a large number of comments that display a complete misunderstanding of basic market mechanics and a complete unwillingness to be specific. No shortage of moral posturing, however.

I don't know why it's bad that people who don't need low latency are allowed to not pay for it.

For years, mil-spec GPS time was masked with randomness and unavail to the public. Why not do this with the markets? The regulators only care about mean prices being unbiased. They should equilibriate just fine with some non-sequential information introduced. Why or why not?

The regulator cares about more than just the mean price. The idea behind Reg NMS is that anyone who places an order at a protected venue should get the best price available, not just something close. It isn't enough for the differences to just average out, because some people get poor execution in one direction, and others get poor execution in the other.

Agree but consider this:

If the timing is perfectly tractable the system is gamed. A small amount of noise that prevents manipulation is a good thing over time for the market. It means the stock is going to have less vol which increases the value of the market equity given constant cash flows. That's another policy goal more broadly. That is, if it works this way.

> I just find this topic frustrating since there are a large number of comments that display a complete misunderstanding of basic market mechanics and a complete unwillingness to be specific.

Not to mention, an abundance of aggressive misuse of terms of art such as "insider trading", "front running", and others. These terms have a well-defined meaning in SEC and FINRA regulations, and it does no one any good to accuse HFT of doing things that it isn't doing.

"I don't know why it's bad that people who don't need low latency are allowed to not pay for it."

With the prices I am hearing it's not about those who don't need it but rather those who need it and would like to have it but cant afford it.

If I want to move one lot or ten lots through the market, I don't need low latency.

If I want to move 10k lots through the market, I can afford low latency.

And why would you want to move 10K lots through the market any others point of view than the HFT traders? It's a genuine question.

Your English broke down there, and I'm not sure what you're asking. "Why would anyone but HFT traders want to move 10K lots through the market?"?

Sorry English is not my first language and sometimes I end up writing convoluted sentences.

Let me rephrase then.

Why would anyone but HFT traders need to move 10K lots through the market?

"Sorry English is not my first language and sometimes I end up writing convoluted sentences."

No worries at all. I certainly didn't mean to imply any sort of a judgement; I just needed clarification.

"Why would anyone but HFT traders need to move 10K lots through the market?"

To a first approximation, it's never the HFT traders that need to move 10K lots. In commodities, it's big producers and consumers - Chevron needs to secure crude stock, ADM needs to buy up a ton of corn, some mega-farm (or co-op of large farms) wants to lock in a price on what they're producing, &c. In equities, it's institutional investors, hedge funds, and possibly exceedingly wealthy individuals, who have decided they want to hold many more or many fewer of something. It's clearly going to be a big player - transacting 10k lots of a stock priced at $10 would be $10 million. Most people - including HFTs - aren't trying to move millions of dollars in one direction in a short time. Those who are have resources, almost by definition.

One of the core complaints about HFT regulation is that the regulators refuse to update their definitions to include HFT activities that follow the letter of the regulation but clearly violate the spirit of the regulation. Therefore arguing that HFT detractors are not relying on precise definitions does more to show that you're not really listening to their argument but rather just finding something to nitpick. It looks like an avoidance strategy, not an honest defense.

First off, my quibbles about the definition of front running vs predatory trading are unrelated to my defense. The defense I gave was about price discrimination. It doesn't matter what label you apply to the definition - if you want to argue against me then say why price discrimination is bad. Or say I'm wrong about price discrimination.

Second, the "spirit of the regulation" about front running is about allowing people to trust their agents. Other regulations in the same spirit are lawyer-client confidentiality and "financial advisers can't take gifts from companies they give advice on" regulations.

There are no "you can't kick people's ass in the stock market" regulations, in letter or spirit. If you play the stock market everyone is trying to take your money. You are trying to take theirs.

>So everyone needs to pay several thousand/month for direct access?

I think you're overestimating the cost of running an exchange by several billion dollars.

>I don't know why it's bad that people who don't need low latency are allowed to not pay for it.

FRONT RUNNING

>"Quote Stuffing" is not the nefarious activity that Nanex portrays it as. It's generally the result of your algorithm screwing up

No, it's generally the result of an HFT intentionally trying to clog up the order stream (in order to push everybody else back), or trying to gain an information advantage.

>It provides no benefit to the trader

Now you're in cuckoo land.

> I just find this topic frustrating since there are a large number of comments that display a complete misunderstanding of basic market mechanics

Yes, i find it frustrating arguing with you too. What's worse is that you think using industry jargon will intimidate others into thinking you know better.

I'm planning to write a blog post on quote stuffing - explaining why it is dangerous for the trader and why it is easily caused by software bugs. I hope you reads it.

The exchanges do send the information to the SIP at the same time that they publish it on their own direct feed. NYSE actually got in trouble for not doing this properly. This isn't actually an easy problem, because there is a TCP interconnect with the SIP, and if it gets bogged down, then the SIP packets can be backed up.

The real issue is that since the SIP (CQS/UQDF) are cheap; and big/sophisticated players don't actually use them, no one cares about them. Beyond that the performance is necessarily slower than a direct feed, because if I wanted to learn the price of a NYSE listed-security on NASDAQ the path that the SIP would take is far less direct. Specifically, if I have a machine sitting in Carteret (Nasdaq), listening to the proprietary Nasdaq feed, the data from Nasdaq will be sent to me and the SIP at the same time. But to get it from the SIP it has to go to Mahwah and then back, instead of staying local to the data center.

Nasdaq makes its feed available to everyone, for a cost. Someone who is providing execution services, or market making, needs the direct feeds so they can provide good execution and know when to get out of the way. A small investor looking at a screen doesn't care about the SIP or a direct feed. They are going to submit their order, and their broker is responsible for executing it.

What do you think would happen if direct feeds were prohibited and participants had to listen to the consolidated feeds only? This could even include getting one's own fills, if there were a consolidated feed with order IDs included (i.e. you would not know your fills any earlier than anyone else).

I don't know, but I can't see very many good side-effects, though can think of some negative ones. The first thing I would think is that participants would gravitate towards trading where the SIP is to avoid transit latency, reducing competition for those exchanges that run their own data center and SIP. Also, I'd imagine that there would be a greater risk of getting run over, so spreads would widen to account for it.

Then HFTs would still win but pricing would widen out since some liquidity is made available because of ability to hedge in another product. Think about if I'm quoting SPY because I can buy/sell the basket better than my quotes. If the feeds are delayed, I'm less confident in where the basket price really is, so I'm going to demand more compensation in bid-ask spread to put up a price. This wouldn't be a big deal in slow markets but when things really started moving, most derivative liquidity would be pulled entirely.

In single names HFTs mostly use these feeds to protect themselves from other high-speed traders and they want the granularity of information for their models. They're already faster than retail guys and banks and always will be. It's not an "unfair advantage" like people think. They just have to get on it if any competitor gets on it, similar to all the fast lines. If 20 HFT shops get on a new fast line, none of them are making more money from it.

The other thing people should understand about the consolidated feeds is that they only show the aggregated size at the best price at each exchange, whereas the direct feeds show you every displayed order in the book. A direct feed is a firehose and requires significantly more network and processing power to actually make sense out of it.

I think there should be a new kind of logical fallacy: appeal to grandpa.

Today's world is not the same as your grandfathers world, we use computers to do many that they are better at than humans, executing trading stragies in submicrosecond latencies is one of them.

Submitting an order to a market causes the price to change, submitting a large order means that you are likely trading against an informed trader, by making a bid they are disseminating information about price to the market, if you don't adapt to the new information (by raising your price) you are an missing a huge opportunity.

In an ideal market to trade a large block you'd pay a risk premium less than the price of disseminating that information over a longer period and possibly paying more as supply of the stock at a given price evaporated.

Everyone can read the order book to see the market depth for a trade of a given size, if you showing your hand all at once is the same as showing it over a prolonged time as you eat through the depth and the market becomes shallow.

Essentially what the anti-HFT people are saying is that for a public company it's rational to expect to buy the entire stock are the current market value, as anyone who has seen a takeover go down, the purchaser must offer a premium over the current market cap in order for a bid to be successful.

Simply making an offer usually causes the share price of the underlying company rise to a very similar to the offer discounted for likelihood of regulatory approval and the time value of money. HFT is expected market behaviour simply occurring over a time period of nanoseconds rather than minutes and hours.

The issue seems to be that, when you want to buy a somewhat-large amount of stock, regulations exist which:

1. Force you to split up the purchase among multiple exchanges, and

2. To compensate for this disadvantage to the buyer, are supposed to prevent news of the order causing a price change at the other exchanges by spreading faster than filling of the order, so that you don't have to deal with an artificially-inflated price on top of artificially-imposed split of your order.

But while (1) is enforced, (2) is not. Which means buyers and most sellers have to play by one set of rules while people who want to inject themselves as middlemen get to play by a different set of rules.

At least, that's what seems to be being described here, and I certainly can see problems with it.

(not to mention that that "liquidity" and "efficiency" arguments fall flat -- the types of HFT schemes being described here work only by creating artificial illiquidity, and I don't know of any rigorous argument that creating more middlemen improves efficiency)

There is no regulation which forces you to split up your order. People split their order because one venue does not have enough liquidity.

To compensate for this disadvantage to the buyer, are supposed to prevent news of the order causing a price change at the other exchanges by spreading faster than filling of the order,

This is simply incorrect. There is a regulation which requires exchanges to transmit news of price changes to the other exchanges as rapidly as possible. The purpose is to ensure that all fills happen at the NBBO.

In concrete terms, if BATS has a BUY@$20.00 and ARCA has a BUY@$20.01, BATS is forbidden from filling your order.

http://en.wikipedia.org/wiki/National_best_bid_and_offer

If you believe the description is wrong, your issue is with the article.

Direct quote from article:

With Reg NMS, it essentially took x amount of available stock at one place, and made it one tenth of that x at ten different places.

If that isn't a regulation forcing large purchases to split up, then it's on you to explain why.

The most charitable interpretation of that quote is that after RegNMS, multiple trading venues became legalized. Liquidity providers then chose to spread liquidity out because it was now legal.

Here is RegNMS. Find me a place where it says liquidity must be divided between all possible exchanges: https://www.sec.gov/rules/final/34-51808.pdf

If you believe the description in the article is wrong, then as I said your problem is with the article.

Seller(s) of stock are not required to place shares on different exchanges. This is a side effect of the process of trading, as the NBBO price bounces around to different venues.

If I offer 100,000 AAPL at NBO at exchange X, and you are willing to purchase 100,000 AAPL at NBO, your buy order matches mine exactly and the transaction takes place. This is highly unlikely, of course, that an enormous stock transaction like this ($50M) will take place this way.

It's a lazy question, but what mechanism is the regulator using to score the effort of the exchange?

A related topic would be how often they take enforcement actions.

Actually, the exchanges are REQUIRED to split up your order, if the bid/offer is better on another exchange.

Example: NYSE order book

100,000 shares offered $10.02

100,000 shares offered $10.00 (quote)

NASDAQ order book

100,000 shares offered $10.01 (quote)

You send a "buy 200,000 shares $10.02 limit" order to the NYSE.

The NYSE is REQUIRED to route 100,000 shares away to NASDAQ@$10.01, rather than filling the extra 100,000 at $10.02.

An HFT intercepts the trade, buys the 100,000 at NASDAQ @$10.01 ahead of you, and also can buy the 100,000@$10.02 at NYSE ahead of you. Now you have to pay $10.03 to buy the remaining 100,000 shares.

Please explain how My. HFT "intercepts" the trade in this case? How does he know your limit price? How does he know your parent quantity?

Imagine an alternate scenario: You send an order "buy 100,000 shares $10.00 limit" to NYSE. Does the HFT spend $4MM buying up 200k on the off chance that you want more than the 100k you already bought? If not, why?

The feed the HFT gets is faster than the NYSE to NASDAQ feed. That lets them trade ahead of you.

You placed an ordinary 200,000 share limit order. You didn't use one of the special order types.

It is riskless for the HFT to buy 100,000 from NASDAQ, knowing they can sell back to you when you buy the other half of your order. Other algos, seeing a big buy, move up the price, so the HFT (may) be able to safely buy and then sell to them.

> The feed the HFT gets is faster than the NYSE to NASDAQ feed.

Assuming for the moment that someone can see an execution from NYSE and route an order to NASDAQ in response in less time than NYSE can send their own order to that Exchange you haven't answered the rest of the question.

How does the HFT even know there is an order in-flight to NASDAQ? How does the HFT know that the buyer is willing to pay $10.02?

> You placed an ordinary 200,000 share limit order. You didn't use one of the special order types.

Is this supposed to be an answer? 100k executed at NYSE @ $10.00, this is public information. NYSE routed the other 100k to NASDAQ. Are you saying there is some way for Mr. HFT to know the particulars of that order to NASDAQ?

> knowing they can sell back to you when you buy the other half of your order.

You seem to be under the impression that at some point along the line someone besides the NUSE is seeing the parent order details (buy 200k @ $10.02). Please explain how you believe this information is transmitted to third parties.

The HFTs get to see the list of orders as they come in. That's the advantage of colocating with the exchange.

If a 200,000 share limit order is placed at the NYSE, the HFTs know that regulation NMS requires the NYSE to route shares to the BBO on other exchanges, before filling the order from their own book.

>The HFTs get to see the list of orders as they come in.

No, they don't. Third parties cannot see orders "as they come in". Only orders displayed in the exchange's book and executions already completed are sent out to data consumers.

In the example you created the 200k buy order is never displayed and as such is not visible to any other user. All an HFT can see is the 100k execution.

Again, an HFT rushing to buy in front of this order cannot know how big it actually is, nor what the limit price is and as such whatever buying is purely speculative and not a "guaranteed win".

>(not to mention that that "liquidity" and "efficiency" arguments fall flat -- the types of HFT schemes being described here work only by creating artificial illiquidity, and I don't know of any rigorous argument that creating more middlemen improves efficiency)

A lot of HFT is market-makers competing for positions in the book. They're simply providing liquidity and literally the more trades they make the more money they make. Market-making is socially valuable (decreases the cost of trading aka the spread) in a similar way that arbitrage is, and as I understand it this is why people are so hesitant to regulate HFT, despite some of the problems it causes. As far as preferential treatment, consider that exchange fee schedules prefer these kinds of traders for a reason.

Exchanges are for-profit and their liquidity providers don't execute trades for the fun of it, there has to be incentive, even if the incentives are at times perverse

If someone was already willing to sell 50 shares of Acme Corp. at $20.00, having an HFT bot get advance knowledge of an order, buy those shares and then re-offer them at $20.01 is not market-making and is not providing liquidity -- it is injecting an artificial middleman, with the cost of a middleman, for no other reason than "we can make money off injecting a middleman".

My point is that regulating this kind of trading can also damage socially useful trading activity and that's why it's unlikely to happen. There's a lot of rabble rousing about HFT in these discussions on HN (in a similar thread someone went as far as to say arbitrage was not socially useful), I thought it was worth pointing out its more complicated than "HFT IS A BIG SCAM"

I'll admit its funny to see the libertarian bastion of HN get so worked up about HFT. Free markets are all fun and games when it's Uber or AirBnB, but HFT harming retail investors seems to really hit a nerve

That's the funny thing too, retail investors are fucked beyond believe and it has nothing to do with HFT and everything to do with not having satellite pictures of walmart parking lots, etc. They trade at such an unbelievable disadvantage that it's laughable that HFT could make any difference. (Plus retailers dont move big enough blocks to get into HFT issues)

How does an HFT bot get advance knowledge of an order?

The scenario most commonly presented is something like:

1. Buyer A wants to buy 50 shares of Acme Corp.

2. The order hits Exchange A first, and Exchange A has 25 shares of Acme Corp. at $20.00 (currently the best price).

3. The order is partially filled, and then on to Exchange 2 to try to get the other 25 shares.

4. Exchange 2 also has 25 shares of Acme Corp. at $20.00 at the time the order hits Exchange 1.

5. An HFT bot saw the order hit Exchange 1, and, in the time it takes for the order to move on to Exchange 2, is able (due to having much faster access) to buy those 25 shares at Exchange 2 before the order gets there, then offer them to the buyer at $20.01.

The fact that the HFT bot was able to act in Exchange 2 with advance knowledge of an order that would arrive at but had not yet arrived at Exchange 2 is what people complain about. And, more generally, people complain that this provides no benefit to buyers or sellers; the buyer was going to buy those 25 shares, and the seller was going to sell those 25 shares, no matter what. So no additional liquidity or market-making went on, and the buyer's order likely is not filled any more quickly than it would otherwise have been.

(and several articles complain that this practice actually leads to loss of liquidity, in that orders go unfilled because HFT bots racing ahead from exchange to exchange have reduced the previously-available shares)

The workaround that's been presented is to have the actual initial order take circuitous, deliberately time-delaying routes to the exchanges in order to have it hit multiple exchanges within a short enough time that an HFT bot can't inject itself as an artifical middleman.

What you describe is called predatory trading. It will almost never occur with a half-lot order, but can occur with large orders.

Rather than repeat myself, I'll just link to a blog post I just wrote about it.

http://www.chrisstucchio.com/blog/2014/fervent_defense_of_fr...

What you describe as predatory trading is different from ubernostrum's scenario. In your predatory trading example, Little Billie trying to move a large block of shares, there are no existing orders in the book that match the entire block; so once the initial partial fill of Little Billie's order is done, he will have to change his offer for the rest of the block to move it. In this situation, yes, the predatory trader is communicating market information that might not otherwise get communicated.

In ubernostrum's example (which is also the one appearing in the OP to this thread), there are existing orders in the book that match the entire block. So the HFT in between adds nothing; the entire trade would have occurred at the original offer without him.

How does the HFT know the size of the buy order?

Alter the scenario here:

1. Buyer wants 25 shares, buys all available shares at his limit price on exchange A.

2. HFTs 1-n see the entire quote taken at Exchange A and rush to buy all available shares at that price on Exchanges B-K

3. HFTs now own a few hundred shares of Acme corp that they need to unwind.

HFTs are only accessible to a select few. And are only worth doing over large sums of capital.

Is it rigging if my dice has more sixes than yours?

This is incorrect - many HFTs are just one programmer investing his savings. Many of the big guys started this way.

THIS is incorrect. From the article:

>A direct feed from one exchange could be $10,000 to $60,000 a month. If the SIP is no longer real-time, it means that people who want to get real-time data have to pay, well, significantly more. It’s no exaggeration here to say you go from $1 a month to well over $100,000, with your own networking staff and a room full of computers just to get real-time pricing.

$100k a month is now something you can stump up out of your savings is it?

You don't need to pay $100,000 month for access. My NDA forbids from revealing pricing info, but I can say I worked for an HFT fund that was just a programmer working out of his living room.

... Neither of your statements invalidated my statements.

Unless you are saying that programmers are a majority, and their savings are typical of the same majority?

EDIT: (And reading the other response, it seems you are in fact incorrect).

Here's a few examples from at 2010 CNBC story of top HFTs. It's a pretty diverse group but most were programmers and most started (relatively?) small.

http://www.cnbc.com/id/39038892

Yet the price to "play the game" does not exactly say "small guy".

Trading is the only industry on earth where we even debate whether Joe Schmoe off the street should have a chance at competing with huge institutions, and oddly enough it's probably one of the most purely competitive things in the world. Most guys who play ball at the local park can't be Lebron James. A guy sitting at home has no shot at market-making liquid stocks, and low latency is only the tip of the iceberg as to why.

In any case it is not expensive to start a firm like this relative to most things in finance or even other physical market-making businesses like running a retail store. You could do it with a few million dollars and a dozen technologists/researchers. That's why this debate is so stupid. Banks who are mostly the ones complaining about having trouble executing could easily hire up a team to build better routers and execution algos for them. Most of them have, even, which has cut HFT profitability considerably. The laggards prefer to go on CNBC rather than invest in technology to keep up.

So you are comparing startup cost of a ball with the startup cost of a "few million dollars" to start a HFT company?

I have no issue with the market in general or people making money providing investment to others who need it.

Where I am having problems is when the market becomes it's own closed loop and the money is made inside of that loop. I been on the sideline of this industry for many years now and I have yet to see any company who is successful in the long run be so because they play a fair game.

Reading stuff like this:

http://moneymorning.com/2012/10/15/the-truth-about-high-freq...

certainly doesn't help.

"...What they are doing is trying to influence, I call it manipulate, what other traders and investors do with their bids and offers. They are trying to fake or set-up other market participants to react to the quotes the HFT players fire out onto the exchanges for all the stocks they trade.

Only the HFT quotes sent out aren't meant to be acted on. They aren't looking to buy on their bid quotes or sell on their offer quotes.

Instead, they are sending out orders to "ping" markets.

Ping refers to how sonar works. For example, a submarine sends out a sonar beep which hits a target and sends back a sound (which sounds like a ping) which the sonar operator "reads" to determine the pinged object's distance and shape.

HFT players are constantly pinging stocks where their quotes are housed and displayed. They send out their orders to manipulate others to adjust their quotes, which get fed into the HFT algorithms to determine any directionality; then, if an opportunity exists the HFT computers buy or sell shares that someone else has put onto the market.

They aren't quoting constantly as bona fide "market-makers" are supposed to do, which they claim they are acting like. They are simply putting out millions of fake bids and offers which they pull almost immediately, just to read the movement of other market participants who react to the HFT come-ons.

It isn't illegal. But it is manipulation.

The buyers and sellers the HFT crowd trades with aren't forced to trade, they are willing to trade -- it's just that the prices they trade at may have been manipulated..."

Can you explain to me how HFT benefits society in general? What would happen if we didn't have HFT? Would the market crash? Would there suddenly be no money to invest with?

There are not many businesses you can start that cost so little. Even a McDonalds franchise would cost more to start than a small trading firm.

The activity he describes is illegal. Every order you place must be because of a bona fide interest to trade at that time, and not as an attempt to induce others to act or gather information.

If I could get filled on every bid and offer I have in the market I'd be very happy. That's basically what happened to the MMs on the other side of the Knight debacle.

HFT moves liquidity across time, place, and product. It drives down spreads by intermediating trades for a lower cost than human market makers had to charge. The market would not crash if HFT went away, but markets would become less efficient, more trades would go off away from fair values since nobody could correct small-scale mispricings, spreads to customers who immediately want to buy and sell would increase, and short-term volatility would go up since nobody would buffer transient pricing errors. It'd be like if all the grocery stores around you folded and you had to trek up to the meat market and buy things in bulk. It would be very inconvenient and expensive, but not the end of the world.

In the digital space there is.

Comparing it to a McDonalds franchise make little sense IMO both in terms of what the investment is used for and the returns you stand to gain.

With regards to the inconvenience and price of not having HFT around. That again sounds like an issue for the traders not for the companies the stockmarket is supposed to help.

The returns aren't that high though. Even the top firms now barely make anything after costs.

The stock market is designed for traders! The primary purpose of a stock market is to let stockholders transfer risk most efficiently. The capital raising was done when the company issued shares. People buying stocks are not investing.

But then we are back to the original question. What value does it provide then?

You claimed that it would be much more expensive if HFT wasn't around. So surely HFT should be able to reap the benefits of what they save no?

Not in a competitive market, and on-exchange liquidity provision is the most purely competitive market out there. My quote is no better than anyone else's, so anyone can undercut my price.

Imagine the grocery store example I gave. Say meat costs $5/lb wholesale and your transportation costs to drive to the meat market in a car are $1.50. A grocery store using a truck to pick up the meat and freeze it in their store can transport the meat for $0.50. In a non-competitive market, they would charge you $6.49 for the meat, and you'd prefer to buy it there since it's less than your alternative of paying $6.50. However, if there are two grocery stores across from one another, they can compete on price to win your business, and the price of meat will end up being marginally above the costs of whoever can transport/store meat the cheapest, maybe $5.55 or something. The end consumer captures most of the surplus value.

The same thing happens in trading. Professional traders are more skilled at managing positions and trading across markets. By electing to trade with a market-maker's quote, you are effectively giving them your trading problem. Unlike the grocery store example where only a handful of businesses may compete, there are dozens of firms engaged in automated trading all competing for your business by offering tighter prices. They'll continue to do this as long as it's profitable, but competition will erode their profit margins until they're slightly above costs.

HFTs made a killing when they had first-mover advantage, just like if you were the first company to use smart logistics software or any other innovation. Now, not so much.

thats still not showing transcendence. How does HFT help the grocery store, the startup, the autocar dealer, amazon, mictosoft etc. you only showed inner market workings not why we as a society need it.

We don't need much more than safe shelter and a shirt on our backs. We could get by with once a day call auctions or even trading stocks face-to-face without an exchange at all. We don't even need stocks or companies. That doesn't mean intermediation in real-time markets isn't valuable.

Say you have a car and want to sell it to buy your future wife an engagement ring. You could take it to CarMax and sell it instantly, or spend your time researching a fair price, posting up ads, cleaning it up for sale, meeting with potential buyers, etc. CarMax has all sorts of "unfair" advantages: they've developed superior pricing models, they have access to historical auction data you may not be able to get, they can sell their inventory in the inter-dealer auction market, they've invested heavily in their own sales force to resell the car, through economies of scale they can hire their own detailers and mechanics to fix the cars up at a lower price, etc. CarMax is clearly superior to you at the "reselling used cars" game, though all the advantages they have are things any competitor could choose to purchase. They make money doing this, but people still choose to deal with them because it's a net benefit to be able to sell your car instantly.

Some people prefer to sell their car to CarMax, others try their hand at doing it themselves, and through their patience they may save a bit of money. Likewise you can do the same in the financial markets. You can work your own limit orders to enter or exit positions or trade in the opening/closing call auctions on each exchange, but many traders prefer to trade instantly with quotes placed by a market maker. Being able to liquidate your stock holdings to spend the money on something else instantly is valuable to some people, and they pay for it through the bid-ask spread.

You're basically saying people want the wrong things.

no i am basically asking whether anyone besides the people doing HFT benefits from it. If they do then it should be easy to point to the positive consequences of HFT in society and a lot more people would back it up.

If its only beneficial as a scheme to get money out of the stockmarket then many people are naturally going to be against it.

Claiming i am on the wrong side of what people want seem to be a tad ignorant of reality, dont you think

Countless studies point to a lowering in bid-ask spreads (the effective "fee" impatient traders pay to trade) when electronic market-makers entered the market, and an increase in this spread when they leave or are forced out:

http://marginalrevolution.com/marginalrevolution/2014/04/a-s...

For me to win trades as a market-maker, I have to put up prices that are tighter (buying higher and selling lower) than my competitors. Through this competitive process, the rents earned by market-makers decrease and the end consumer of their liquidity reaps most of the benefit. Since markets have moved away from a monopoly specialist model and become more electronic, spreads (and thus compensation earned by market-makers) have decreased dramatically.

Nobody is backing it up because a.) firms involved tend to be secretive so as not to give away any competitive information, b.) there are a lot of slow or inefficient brokers speaking out against it "talking their book", and c.) to the end-user, HFT is like plumbing. I don't thank the city workers every time I hop in the shower, or write a glowing blog post about Visa when I swipe my card to buy goods and services all over the globe. The infrastructure exists and it provides a benefit, but it's faceless and not something most people think about. Most people don't even consider who is on the other side of their trades or where the prices even come from. They just want to click a button and get in or out of a position, and thanks to competing HFTs they can do it instantly, cheaper than ever before.

This perhaps used to be true, but the HFT arms race has pushed the cost of entry up to the point where you are looking $50k-$100k a month in infrastructure costs just for a seat at the table, and many times more than that if you want to play with the big boys. And that is in addition to the money you actually plan to invest.

I had this long reply about how HFT allowing you to execute large trades, etc is a bullshit argument, but then I deleted it. I weighed in on how it affects pricing, how it affects the seller, re-read your post, and realized something:

I'm grandpa.

A veteran geologist explains how the medical industry became "rigged"

Can anyone explain to me what wouldn't be possible without HFT? Is HFT allowing for some useful types of investments/capital?

Buying a lot of shares at once. I wrote a blog post explaining how to not trade with HFTs if you don't want to:

http://www.chrisstucchio.com/blog/2014/how_to_not_get_ripped...

Quoth a trader: "Let me tell you this as trader with 15 years direct bank and hedge fund execution experience. Your suggestion of using such algorithms would result in me never getting filled even remotely close to where I wanted to get filled."

tl;dr; He wants to buy 10,000 shares right now and HFTs are the only ones offering that.

So its good for the trader who wants to do that. Is there any benefits to the market overall that he wants to do that?

Edit: Can someone explain to me why I am being down voted from asking a question? It seems like every time I get into a discussion with yummyfajitas I am being down voted?

Yes: limit orders provide liquidity, and market orders sap it.

Allowing a trader to make large trades makes price discovery happen faster and more accurately. I've written a more detailed explanation on my blog - read HFT apology 1 and 2.

http://www.chrisstucchio.com/blog/tags/high%20frequency%20tr...

As for the downvotes, I lead a small cult of traders, rationalists and scala programmers - tptacek is my #2 sycophant.

So again the trader not the market/society.

Whats the benefit outside the stock market to society in general? I.e. how is HFT a net profit to other than those involved in HFT?

You are asking people to be specific, can you you?

The tutorials I linked to address your questions in detail.

You must be able to give examples.

An example is provided in part 2 of the tutorial. Are you demanding I copy&paste it here?

Yes please

o.O

Amusingly: this comment was downvoted 8 minutes after I posted it, despite the thread being 6 days off the front page. :)

" they will signal up to their other machine to say, “Buy everything that’s available.”"

This sentence is very problematic as it implies that HFT are injecting themselves between 2 intermediaries that don't need the HFT in the first place. What is actually happening is the HFT is not buying or selling anything. They are changing the prices they themselves were offering originally. There is no intermediary involved.

Here's an interesting counterargument: http://streetwiseprofessor.com/?p=8333

A direct feed from one exchange could be $10,000 to $60,000 a month.

That's way off the mark. It's more in the $1k - $3k range.

http://cdn.batstrading.com/resources/regulation/BATS_US_Mark...

http://www.nyxdata.com/page/1084

https://www.nasdaqtrader.com/Trader.aspx?id=PriceListTrading...

This is the SEC filing with proposed fees http://www.sec.gov/rules/sro/nysemkt/2013/34-70285.pdf

That's for a full colocation - not just a direct feed. The originally posted links are the correct prices for direct feeds.

You are referring to direct feed content, I was explaining (in the interview) direct feed speed and content. Who buys a direct feed for a speed advantage and then doesn't colocate?!

"The market is a pendulum that forever swings between unsustainable optimism and unjustifiable pessimism." - Benjamin Graham

Those Washington Post articles are a pain to read on mobile. I will avoid them like the plague.

Reply on news.ycombinator.com