Hacker News Reader

Comment by pmb | Hacker News Reader

These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop. There is no chaos monkey for the stock market, and transaction rollback is available only in the most extreme circumstances.

I have yet to see a convincing rationale for how high frequency trading adds value to the system - it certainly doesn't seem to add pricing stability. Because this is hacker news, I suspect a few people reading this either work in the industry and/or have strong opinions about it, and I would love to hear why I am wrong. Because I do like it when clever computer scientists make money, and that seems to be pretty much the only social benefit of HFT, at the cost of flash crashes and things like the example above.

View on HN

These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop.

This implies that humans are actually better than machines at making data-driven decisions at high rates. They're not. They're astoundingly not once you compute the cost of humans versus the cost of machines on a per-decision basis.

Liquidity in the market used to be provided by large groups of sweaty, overpaid alpha males yelling at each other. We replaced them with dueling robots. The robots can provide liquidity for a fraction of the price. The alpha males really, really hate competing with robots, because the alpha males invariably lose, so they complain that robots are stealing the money that the alpha males used to extract from their customers by right of being the one with a license to be shouting and sweaty at a particular physical location.

Occasionally a robot blows up. Not a problem -- robots are easy to replace. Besides, humans blow up all the time. We only ignore their ridiculous strictly-inferior-in-every-way-at-this-task nature because they look more like us than the robots do, and because these particular humans being displaced used to be rich, whereas e.g. telephone operators tasked with manually doing call routing (also clearly inferior to highly reliable distributed algorithms) were poor.

Another way to look at this: it's a well-known phenomenon that expert radiologists will misdiagnose an X-ray image, even contradicting a past judgment they made on the same X-ray. It's feasible that a properly machine-learned computer could make better judgments on a long-running average basis.

But when a computer screws up an edge case, versus the many, many times a doctor will screw up a diagnosis, which instance will get more attention. People are scared of black box machines and will take particular note of the times an algorithm has screwed them over, regardless of the times a human has screwed them up.

Also, when the possibility of examination exists, computerized decision making is far, far easier to audit (i.e. determine blame)

Exactly. It is the same as the problem with robotic cars.

Cars with human drivers kill thousands, but nobody blinks. The day a robotic car kills a single person, all the news networks will have a field day.

I think you're right about this. Maybe it has to do with appearance of control/self determination. It was a mistake (an accident), but it was my mistake/accident (as opposed to something one didn't have control over).

I once posited the question to a group of friends, would you rather drive a car with a 60% chance of crashing fatally in your lifetime, or use a robotic car that as a 2% failure rate. Everyone chose the 60%, even though statistically it makes absolutely no sense.

Not (entirely) true. Robots are not strictly superior to humans in this case because the emergent properties of the system change due to the speed of robot action.

As more robots crowd into the very fast strategy space, there tends to be more collision of strategies and thus more extreme price movements as many robots all try to buy/sell the same thing at the same time. If they all pile onto one side of the market, liquidity disappears and price movements become dramatic. In the very fast strategy space, there is less capability for complex strategies requiring long memories. If you want to save some nanoseconds to win at this winner-take-all market, you need to reduce your memory and processing time.

http://arxiv.org/PS_cache/arxiv/pdf/1006/1006.5490v1.pdf

This isn't saying that humans would never display herd behavior, but that they do so less often.

Humans absolutely do display herd behavior. This has been documented in financial markets and other venues.

However, they do it (by definition) on a time scale that is perceptible to other humans.

I agree with your premise -- the issue is that when humans blow up it doesn't cause a cascading runaway failure like robots are susceptible to.

Humans are susceptible to cascading runaway "ethical" failures. I'm thinking the $2 billion UBS loss. Humans also don't scale very well so risk tends to be distributed across several of us. Perhaps robots should be setup in the same way; that there are several of them with different algorithms tuned for different levels of risk... or perhaps it really doesn't matter all that much because this is one firm and not the entire system so the risk is already distributed correctly.

That's the first time I've heard an explanation of the benefits of HFT that actually adds up. Maybe that's because it's appealing to my prejudices, but thanks.

I think I stole "dueling robots" from tptacek or yummyfajitas, but sweaty alpha males are totally mine.

Edit to add: The dueling robot at HNsearch says a) it was tptacek and b) I should trade with it because it remembers HN comments better than I could ever hope to.

Well put.

A lot of this "but ... but ... but ... we NEED a human in the loop" handwringing that we're seeing on the news channels is obsolete people who used to be ridiculously overpaid complaining that they have been replaced by fast computers and efficient code.

(Disclaimer: Non-sweaty alpha male HFT algorithm writer here ... ha.)

There are cases where having a human in the loop is appropriate, particularly manufacturing. A human, thanks to a combination of slowness and observation skill, will pick up on a fault after the first error while a machine, thanks to a combination of speed and lack of observation skill, will produce fault after fault until things come to a crashing halt or someone notices before that happens.

This $440m example shows that a human should be thrown into the mix. Let the human/machine strengths cancel out each other's weaknesses.

The 'Does HFT add value?' question gets flogged to death every time one of these articles gets posted.

http://news.ycombinator.com/item?id=3894302

http://news.ycombinator.com/item?id=3852341

http://news.ycombinator.com/item?id=2828538

To return the discussion back to your main question, there is strong evidence that high frequency trading, or algorithmic trading (AT), increases liquidity: http://www.afajof.org/afa/forthcoming/6130p.pdf (See Figure 2). Increasing liquidity and reducing the bid/ask spread adds value to the system by increasing efficiency and information dissemination.

For a period after the introduction of autoquote in 2003, providers of liquidity captured most of the surplus and enjoyed larger margins on trades (See Figure 3). However, this advantage quickly dissipated as more parties implemented AT. The first-mover advantage doesn't apply in the world of equity markets; competitors quickly duplicated AT strategies and competition swiftly lowered spreads in the second half of 2003. Today, spreads on equities are much lower than pre-2003 largely thanks to AT.

As for pricing stability, let me disregard AT glitches for the moment. Algorithms can tirelessly monitor market information, whether media reporting, filings, event rumors (eg M&A), order trends, etc. Humans are are relatively constricted to a few information sources when executing trades in comparison to AT. In addition, AT reacts faster to new information sources and can adjust bid/ask near-instantly. Therefore, price volatility increases as a result of increased information efficiency.

Glitches and fast-crashes are a negative counter-example to the information efficiency argument above. I leave it to the reader to decide if liquidity benefits justify the occasional flash-crash. However, recognize that this phenomena is not exclusive to AT: many human traders have caused similar crashes of their own -- I'm looking at you London Whale.

People make massive mistakes all the time. http://dealbook.nytimes.com/2012/06/28/jpmorgan-trading-loss...

What would happen if a computer trading operation used the JPMorgan London unit's flawed risk model and a computer executed trades based on it as many times as the algo saw "opportunities"... until the humans who created the model noticed a problem with it?

Tens of billions? Hundreds?

The usual criticism of high frequency traders doesn't really apply to Knight. Knight is required by law to execute orders that are sent to them, instead of 'investing' in stocks for the long term.

"These sorts of problems will occur in any distributed system which runs at a high rate without humans in the loop"

You write a risk check gateway once. It rarely if ever needs to be touched.

If you do this right, you roll out changes on one machine with strict risk controls (1000 share position, for example) to prevent such a blowout. Then you transition everything else.

Ironically, this should be exactly what other tech firms should do: roll out changes to a small area, check basic problems, and then move on to a larger rollout.

> You write a risk check gateway once. It rarely if ever needs to be touched.

Such checks are only as good as the imagination of the author.

> Then you transition everything else.

Which is all well and good until you have an issue where two independent systems creat a feedback loop. Such problems are only evident 'at scale'.

I'm not blaming or trying to exonerate anyone. I love armchair quarterbacking as much as the next guy but a small trading shop runs a little differently than america's largest electronic market maker.

I went through this process myself in setting up a self-clearing BD, so I am fully aware of all of the relevant rules and regulations.

>Such checks are only as good as the imagination of the author.

SEC and FINRA regulations require very specific risk checks. One of those involves looking at potential positions if all of your (buy or sell) orders get filled. You have to sum your max potential gross position across all symbols and decide if you will trip a limit. If so you have to halt.

This is not imagination; this is codified. If you send an order for 100 shares, you HAVE TO ASSUME they were filled when you send the next order. That's the rule. This was not left to imagination.

"Which is all well and good until you have an issue where two independent systems creat a feedback loop. Such problems are only evident 'at scale'."

There should be no feedback where the trading engine affects risk check. In fact, the new rules require separate code bases and separate legal entities.

You created a self-clearing broker/dealer? Thats incredible. I would hope that the benefits of that paperwork outweighs the incredible costs of setting that up.

Nobody except for HFT people has ever argued that they add value to the system.

Everyone else believes that HFT exists because it's profitable, and that it subtracts value from the system.

You can choose who you want to believe.

Market makers used to be humans. Now, due to advances in technology, computers can do that job. Computers are (generally) cheaper than humans so they can do the job for less money. You can see this in the fact that spreads have decreased as market making has been turned over to computers. That's the value they are adding.

Market making has always been work so it's always cost money. But now it's cheaper. There's always been money subtracted from the system, but now it's less than it used to be.

Reply on news.ycombinator.com