Algorithms take control of Wall Street | Hacker News Reader

Algorithms take control of Wall Street

arstechnica.com

53 pointsshawndumas37 comments

This article is very poorly researched. For example, it refers to high-frequency traders as "flash traders". I have never heard that term before. The cited Wikipedia article discusses flash orders, which was a quoting mechanism that DirectEdge experimented with in 2009. Flash orders are not some most automated market markers even care about.

Also, the article mentions Harold Bradley as "among the first traders to explore the power of algorithms in the late '90s", completely omitting much earlier pioneers such as Peter Muller and David E. Shaw.

And of course it goes into depth about the Flash Crash without once mentioning Waddell & Reed, a discretionary (non-quant) fund whose errant trade started the crash to begin with.

Reading this was like watching the 11-o'clock news discuss a medical discovery.

Nothing unusual then. Pretty much every tech article in the WSJ is of this caliber.

Except that I see no indication this is a Wall Street Journal article. The site is Ars Technica; the by-line says "By Felix Salmon (wired.com), Jon Stokes" and the end of the article says "Felix Salmon (felix@felix salmon.com) is a blogger for Reuters."

I think what @w1ntermute means is that we can expect Ars to screw-up finance stories just as the WSJ screws up tech stories. Kind of an admonition for when journalists choose to venture beyond their expertise.

Yeah, sorry I didn't make that clearer. I made the comment while this[0] poorly researched/written WSJ tech article was still on my mind.

0: http://news.ycombinator.com/item?id=2086738

That's odd; Felix Salmon writes almost exclusively for the financial industry, and used to run his own blog. Reuters hired him for his finance industry audience. It's surprising that he would make elementary errors regarding his industry of primary coverage.

It's easy to be wrong when it's so hard to make out what's right. Algorithmic traders are unusually secretive about their modes of operation. The primary knowledge dissemination mechanism is via employees moving.

Uh, but the Wikipedia article on the topic has the relevant information in this case.

>Also, the article mentions Harold Bradley as "among the first traders to explore the power of algorithms in the late '90s", completely omitting much earlier pioneers such as Peter Muller and David E. Shaw

Don't forget the super pioneering work that Claude Shannon and Ed Thorp were doing way back in the 60's and 70's too!

which was a quoting mechanism that DirectEdge experimented with in 2009

Could you go into more detail about what you mean by that? Unless you're being pedantic about something specific, I don't get how you could say that. Flash trading (in several variations) is basically the only thing Direct Edge offers, and they currently carry roughly 12% of the market's daily trading volume.

Everything else you said was spot on, but this is wrong:

> And of course it goes into depth about the Flash Crash without once mentioning Waddell & Reed, a discretionary (non-quant) fund whose errant trade started the crash to begin with.

Waddell & Reed had nothing to do with the big flash crash, nor with the 10 slightly smaller crashes that has happened since -- at least, not any other non-HFT trader had that day. If you want something which isn't some ridiculous CYA, read Nanex's analysis, not the SEC.

> "Automated trading systems will follow their coded logic regardless of outcome,"

Of course, so will humans. They won't make the absurd trades referred to, but a system of trading humans can still, say, get stuck in a positive feedback loop. Of course, they aren't pure trading algorithm, so they may notice the problem, but actually putting a stop to it requires coordination and is hard. Mostly the loop just occurs more slowly.

Yep, which is precisely why economic bubbles happen. Algo traders weren't responsible for housing prices in the last decade.

Comforting to know that the economy is being gambled on a roshambo programming competition.

http://webdocs.cs.ualberta.ca/~darse/rsbpc.html

Why do you think this is true? For every trade, there is someone on each side. The market as a whole does not care if you make bad trading decisions. Someone else makes up for it.

I think our money needs to be working towards things that we actually care about: advancing science and technology, creating art, socializing, and so on. Putting it all into a big game of craps is not going to improve the human condition in the long run. I mean, it's an interesting game, but not interesting enough to justify the entire economy revolving around it. And people don't play because they find it interesting.

"Even if each individual algorithm makes perfect sense, collectively they obey an emergent logic—artificial intelligence, but not artificial human intelligence. It is, simply, alien, operating at the natural scale of silicon, not neurons and synapses. We may be able to slow it down, but we can never contain, control, or comprehend it"

I can see algorithms being used to automate decision making in almost every other industry, not just Wall St. It's both scary and exciting to see what happens with the resulting "emergent intelligence".

The systems dynamics folks point out that systems with more rapid feedback often have more variability.

The classic example is a car dealership deciding to order cars from the factory. When should they be ordered, and how many should be ordered?

Suppose they decide that every day they place orders based on the 7 day moving average. Then they get a busy day and over-order for a week. That's an expensive mistake.

Next they decide to shrink the window to 3 days (more rapid feedback). Now after another busy day they find stocks dropping precipitously and order big to make up, follow by drops, followed by big orders. In an attempt to stabilise the system, they've made it worse.

The modern stock market has extremely short feedback times, on the order of microseconds. When a feedback loop forms on the market it can spiral out of control within seconds and minutes -- hence the 'flash crash'.

Two ways to deal with this might be to develop some kind of balancing feedback loop (for example, higher prices for more trades-per-second, or a progressive price for trade based on delta with the last trade) or to reduce the feedback rate. An economics blogger I host, Nicholas Gruen (you may know of him as Australia's Gov 2.0 inquiry leader), has suggested just that:

http://clubtroppo.com.au/2010/08/07/a-modest-proposal-to-rem...

http://clubtroppo.com.au/2010/10/08/a-self-denying-ordinance...

Or to do more orderly crossings at regular intervals.

The market makers (and shadow market makers) on the ECNs get paid for providing liquidity, rather than having to pay.

This kind of stuff isn't supposed to happen on the NYSE, as all the trading goes through a single specialist.

The systems dynamics folks point out that systems with more rapid feedback often have more variability.

You've got systems dynamics folks completely backwards. Smaller, more frequent controls tend to stabilized systems, not make them unstable. A simple example we've all seen: take a stable ODE. Now try to discretize it - if you are unlucky or uncareful, your discrete approximation can easily blow up exponentially.

The example you provide is different - you are describing two different control strategies, one of which fails to correct for noise (and note: any HFT who makes this mistake loses money FAST). If you made orders every day based on the 7-day moving average, it would be better than making orders once per week based on the 7 day moving average.

This is exactly what we saw with the flash crash - there was a large exogenous shock and the system self-corrected within minutes.

Good followup, thanks.

The neural network that Bradley was working on sounds interesting.

NB: This article (and several more like it) was in January's wired.

As an AI researcher, I get suspicious when I see anyone talking about Genetic Algorithms and Neural Nets. These are techniques that current researchers simply do not use (Neural Nets are used very sparingly, GAs should never be used at all). They make up for their technical failings by being approachable, particularly for journalists. In short, these methods intuitively sound like they should work much better than they actually do.

What techniques are now used in place of GAs?

GAs are just a really complex version of a local search algorithm. The problem with them is that they're just too complicated - you're trying to solve some non-linear problem, and your first step is to introduce several more non-linear problems that also need to be solved (picking chromosomes, mixing, population size, etc.)?

I'm kind of interested if you could point us in the direction of specific techniques. I always thought neural nets and genetic algorithms were interesting and I'm curious what other techniques are out there.

Your query is so vague it's hard to know where to even start; you're looking for "artificial intelligence" or "machine learning". Here's a halfway decent start, actually: http://en.wikipedia.org/wiki/Artificial_Intelligence#Tools , it at least arms you with a lot of search terms and introductory overviews.

Neural nets and genetic algorithms are plenty interesting, it's just that they're way more interesting than they are useful. (Also note I didn't say useless. Just, not that useful.)

For solving local search problems, I use tabu search (hill climb with a "recently visited" list) or beam search (simultaneous hill search). Both are simple techniques that show remarkable emergent behavior. Many search problems are better phrased in terms of numerical optimization or what have you - if your problems maps a continuous space to a continuous space there's probably a standard numerical technique that solves it better than a local search hack.

For Machine Learning type applications, SVMs are very popular. Briefly, both sufficiently deep neural nets and sufficiently dimensional SVMs are arbitrarily expressive, but SVMs give you a better perspective on what is actually happening with your problem. If you're interested in Machine Learning, you should check out Andrew Moore's very well-written tutorials: http://www.autonlab.org/tutorials/list.html

There are lots of techniques. Really simple and effective methods are naive bayes and kNN. Other methods include decision trees and logistic regression which seems to be popular at Google from what I can indirectly infer from their papers. More complex methods include bayesian and non bayesian graphical models, kernel methods (SVM, RVM) and not too hard but powerful methods are boosted trees, random forests and ensemble methods in general.

Layers of unsupervised learners (clustering) feeding into a supervised learner form a very powerful technique known as deep learning. This technique hasn't found a niche though and can be outperformed by shallower methods for much of where they are used [1]. And due to all this big data mumbo jumbo on-line learning methods are getting to be more important.

[1] http://ai.stanford.edu/~ang/papers/nipsdlufl10-AnalysisSingl...

Actual Genetic Algorithms are not really applicable for this (or most other) AI problems.

Simulated Annealing with restarting is a simpler, less finicky and robust one to use in place of GAs. But since the No Free Lunch theorem for optimization says that the general applicability of any optimization algorithm is as bad as every other and no better than random search the answer is like anything else. Understand your problem and pick the right tools. You can get an overview of the numerous techniques here. http://en.wikipedia.org/wiki/Category:Optimization_algorithm...

* Actually its not quite any since it holds not exactly but to a very very good approximation and there are a few, like Coevolutionary approaches which provide a loophole out of that. So that might be something you want to look at.

What is the loophole that coevolution provides? I didn't realize there was a loophole to NFLT.

Well the loophole is not really a loophole more like an escape hatch in that NFLT was shown to not be applicable for certain types of coevolutionary algorithms like say pareto coevolution. The way I understand it is that in essence the no free lunch theorem only applies when candidate solutions are not re-evaluated in terms of relative performance. In coevolution we continually pit a population of candidate solutions 'to play' against each other in order to evolve a champion. Since fitness in this case is relative and not absolute - making performance tricky to measure - NFTL can be wiggled out of. as long as you can evaluate/term your problem within game like heuristics.

- http://www.no-free-lunch.org/ - http://cs.calstatela.edu/wiki/images/1/15/Wolpert-Coevolutio...

And in trading applications (as with many other domains), GAs can lead to fitting if the quant isn't careful. What looks like a great strategy in a back-test suddenly blows-up in production. I worked at one hedge fund whose safe-guard against GAs was to make traders submit their algos a month before production so they could be tested again out-of-sample.

I am not sure I agree with your take on neural networks. Neural network is a vague terminology. But most people mean multilayer perceptrons which I agree are not very useful. But logistic regression is a very widely used 'single layer neural network'. Even the original single layer perceptron has its place in this big data world (with appropriate modifications: e.g. adapting ensemble and on-line techniques , using the kernel trick etc.).

Then there are auto-encoders, Restricted Boltzmann machines and Deep Belief Nets that are certainly getting a lot of attention and are a type of neural network.

I think the OP was refering to neural network as used with their analogy to their biological counterpart, which has indeed fallen out of fashion several decades ago. I rarely see "neural network" used in recent machine learning/AI literature - most usages are historical, and as you mention, you could fit pretty much anything into neural network if you really want to, so the word is basically meaningless.

I was wondering if anyone else had that reaction. I'm looking at a story about how Bradley was using elaborate GAs to choose the weightings on a small handful of variables that, from the description, sounds like they should've yielded to a regularized linear regression producing a fast cheap optimal answer; and I'm thinking to myself, "Either the journalist completely failed to understand and/or convey the problem that was being solved, or nobody designing this system understood anything about statistical learning and were just throwing cool-sounding tools at the problem, or they were just having fun at the reporter's expense."

For those unlearned in AI, an analogy: the original article reads like someone talking about how they used Haskell monads to add 2 + 2.

Reply on news.ycombinator.com