The stock market is controlled by algorithms that are fighting with each other | Hacker News Reader

The stock market is controlled by algorithms that are fighting with each other

io9.com

94 pointsDanielRibeiro67 comments

Something that I've never understood.

We have markets that are based on allowing trades to happen at any agreed on price, as fast as people's minds can meet. Which can therefore result in very high volatility.

Why hasn't anyone tried building a market that goes the opposite way? Prices drift at a fixed rate (eg min(1 cent, 0.05% of price)/minute) in a direction determined by whether there is an excess of buy or sell orders, and trades execute as soon as both a buy and a sell are available at the right price. This would give greatly reduced instant volatility in return for uncertainty about how quickly orders will complete.

The advantage of the latter strategy being that it is very resistant to attempts to compromise it via high frequency trading techniques. That 1/100th of a second time margin doesn't matter because the price changes too slowly. Therefore a long-term investor knows that they aren't getting charged "rent" via the automated algorithms.

It might be objected that lack of trading volume would make such a market initially useless. But arbitrage would keep its price close to the mainstream market. Hmmm..I guess it initially comes down to whether you're paying more in rent on arbitrage or on HFT...

This reminds me of my idea for a compiler that guarantees performance but not correctness, instead of the other way round.

I have a compiler like that. It turns any code into "return 0".

You mean C?

http://blog.llvm.org/2011/05/what-every-c-programmer-should-...

3D rendering and media compression are almost entirely based on the idea that "close enough" is vastly cheaper than "do it right". At the code level, though... You'd have to be fairly careful about what you allowed, and you'd probably have to be able to enforce correctness to some degree to get any meaningful use out of it.

I like the idea of a compiler making lossy simplifications to mathematical formulas for speed - "You told me to divide by seven. This (faster) machine code divides by 8 most of the time, and divides by 4 every now and then." Blindly trusting the branch-predictor could be fun, too, as long as it took into account the correct answer in its subsequent predictions.

Were you thinking along similar lines, or did you have something completely different in mind?

I like those ideas a lot. I hadn't thought about it in enough detail to come up with similar ones, but that's exactly what I was hoping for.

The problem, of course, is that it's hard to find a metric for "close enough" that works for every program. Perhaps it's easier if you're compiling from a high-level language than from C.

This would be a great idea, but will never get implemented because wall street has almost perfect regulatory capture.

There's been a lot of academic research on call auctions (e.g. http://www.springer.com/business+%26+management/finance/book...) which are somewhat similar what you describe in terms of their effects on price stability (though pricing happens differently), but they've not been successful in practice.

What is the advantage of such a market? Near as I can tell, what would happen is that whenever the market believes the price should move, liquidity will drop to 0. Eventually the market will allow trades at the right price, and liquidity will return. Why is this a good thing?

Also, HFTs will still exist. Suppose the "right price" is $50 - even if the market hasn't adjusted (say it's stuck at $51), HFTs will race to have their orders first in the queue at $50 for when the market does readjust.

For markets to work one must try to get as much trading as possible on them. Only if the majority of trading happens on markets will markets be able to effectively discover prices.

If one puts such artificial limitations on markets, than there is always a danger that people will not trade there. If people trade in side deals instead of the markets, then markets would not be able to discover prices and all kinds of inefficiencies would result from the uncertain prices. For example if you put in your limitation, there will be occasions where both buyers and sellers will want to buy and sell at a certain price but the exchange will simply not let them. In this case the buyers and sellers will try to meet and do the transaction outside of the exchange and that is usually a bad thing.

I think there are other ways this can be fixed. For example one may even the playing field by simply adding a clock to the market so that all trades must happen on clock "ticks". This would prevent the advantage of HFT. Or the exchanges could just stop selling offer data to the HFT people. This would not cause the market problems discussed above, because it would only remove things that actual market participants do not have anyways. Thus, these measures would not remove any actual supply or demand from the market.

The solutions are not hard to figure, the problem is that exchanges do not want to implement them. Exchanges make a lot of extra money from HFT and want to keep and grow that money even if endangers uncertainty in their markets.

For example one may even the playing field by simply adding a clock to the market so that all trades must happen on clock "ticks". This would prevent the advantage of HFT.

No it wouldn't. At 12:00:00:001, all the HFT's will race to place an order and be the first in the queue. At 12:00:01:000, the fastest HFTs will be the ones to trade.

Additionally, if news occurs at 12:00:00:998, the HFTs will all race to cancel their orders as fast as possible.

Well it largely depends on whether you want price stability or price discovery from your markets.

Please explain where the source of capital for the arbitrage comes from, it's not exactly clear to me who would be willing to take huge arbitrage losses from this market. (Maybe the fed could back it, they seem to like taking a bath on arbitrage)

HFTs aren't charging us long-term investors any "rent". They're actually giving us better trade execution. Some HFT algorithms might temporarily distort the market price of certain securities but over the long-term those minor inefficiencies get arbitraged away.

Short term trading is a zero sum game. Therefore the profit that HFTs make is entirely money that in their absence would have been captured by other investors. Given that HFTs consistently do make profit, that is a form of rent charged on other players in the market.

There's no evidence that the majority of HFTs consistently make a profit. It may just look that way due to survivorship bias. The HFTs that do make a profit are harvesting their gains from other active traders, not from long-term investors.

The market is a place for risk transfer. That is where the profit that HFT makes comes from. Long term investors transfer the risk of being or not being in a position to HFT market makers, who then (in expectation) make a small profit per transaction.

The bigger picture that your argument misses is that trading is zero sum in wealth, but NOT zero sum in utility. That is the amazing thing about trading (and by trading, I mean trading in general). Say I have three oranges and you have two bananas. It turns out that I really really love bananas, and you have equal preferences for oranges and bananas. I would be happy to exchange two of my oranges for one of your bananas. The next result is an increase in both of our individual utilities. That is how trading work (and why people trade), and HFT is no different.

Here's a better analogy:

Bananas and oranges go by nightfall after arrival in the day by ship, and there are no other sources of bananas and oranges. For every banana a person eats, they must also eat an orange.

Ships arrive sporadically throughout the day, and a ship can only contain bananas or oranges, not some of each. Only a few ships arrive each day. An equal number of bananas an oranges are shipped each day from the sources, but some of the ships sink on the way, so that there may be a surplus or deficit of bananas or oranges (remember, to eat one banana you must eat one orange).

Spotters see the ships coming at roughly the same time, with some randomly variation, and immediately adjust the price of existing bananas or oranges based on what the incoming ship is carrying--after all it didn't sink, and the fact that it didn't gives some information on whether or not there will be a banana/orange deficit.

One day a man invents a telescope and begins seeing the arriving ships five minutes before everyone else. He provides no net-wealth to society as a whole, but with the money he makes via arbitrage he begins commanding a disproportionate share of society's wealth for his own use.

*Galileo originally used his telescope invention for exactly this (in conjunction with a wealthy family)--one of the least useful things he ever did. Would you rather someone work in HFT (scope out banana boats 5 minutes earlier than the next guy) or work in something useful like scientific research/general technological improvement (Galileo's discovering of deeper secrets of the heavens/improvement of navigation and cartography enabled by the scope)?

So you'd rather he not invent the telescope? Because inventing the telescope "provides no net-wealth to society?"

Lets say it is a hypothetical telescope that has no other use than banana/orange arbitrage. If you don't think that addition makes the analogy more accurate, you haven't seen market-data parsing FFPGAs.

And don't forget this:

"By March 2009 Spread was moving dirt. Soon it had 125 construction crews working at once. [...] At 825 miles and 13.3 milliseconds, Spread's circuit shaves 100 miles and 3 milliseconds off of the previous route of lowest latency. [...] Spread won't disclose cost, but Jason Cohen, the chief operating officer of Allied Fiber, which is building a nationwide network, says laying cable through easy terrain runs $200,000 per mile. Half of Spread's route, however, is through tough virgin terrain, pushing forbes' estimate of its cost toward $300 million."

http://www.forbes.com/forbes/2010/0927/outfront-netscape-jim...

And, back away from the hypothetical, obviously from my post I think Galileo's telescope was great overall.

"One day a man invents a telescope .... He provides no net-wealth to society as a whole"

Financial incentive drives creation. Programable gate arrays that can keep a limit order book drive sales of the cards. If more cards are sold, they get cheaper. If they are cheaper, they are more accessible for medical imaging.

What impact on the national fiber network has the Spread line had? What has it taught us?

Sometimes the Randian suggestion doesn't meaningfully differ from the caricature of Keynes: "dump a portion of production into the ocean to help the economy."

World War II spurred development of new metal alloys, better vehicles, and thousands of other innovations. Does that make Hitler a hero? Was war the only path?

(Sorry, I had to Godwin it before this thread got too long; the more general war analogy isn't too far off though: $300-million in heavy-equipment laden fiber laying probably resulted in some injury--is it too much to ask that we both get the beneficial externalities of R&D and put in the fiber for a useful purpose in the first place?)

The company I work for (Red Hat) has made a ton of contributions to the Linux kernel to improve real time performance. The driver and money for these contributions is high-frequency traders, but the benefits accrue to all sorts of areas and kernel users.

Your analogy is pretty good, but the part about the telescope is critically flawed. You assume that there is very little (zero?) variance in the future once the man with the telescope sees the incoming ship. In the scenario you described, that may be the case.

Unfortunately, in the stock market, there is no where near that level of certainty about the future. There is great risk to posting prices that may in fact be terribly wrong in the future. That is the risk that market makers have to grapple with and quantify when they post or take liquidity.

Suppose that every time you spotted a ship through the telescope, it would magically explode 2 minutes later. This happened roughly 50% of the time, as observed over 1 million ship sightings for the past few years. This is a lot closer to what the stock market is like. The other difference is that every serious player in the stock market also has a telescope :(

The risk you mention can be worked around with loss limiting orders placed on the other side.

You make it sound so trivial and easy. Do you actually believe it's that simple?

My day job is making low-latency trading systems. =)

So is mine. I develop HFT trading algorithms using statistical analysis. Placing limit orders on the other side is as dangerous as anything else. Are you on the strategy side or pure development side? If you are developing strategies, then you know what you said in no way mitigates most of the risk that we talked about.

Pure development. So yeah, there's some handwaving there.

Let's say Galileo had had to climb a rickety ladder to the top of a tower at significant risk of falling and breaking his legs. Would that in any way have altered the social utility of his end product?

re: "every serious player in the stock market also has a telescope," see my comments on the Chicago fiber installation elsewhere in this thread.

Is that 5 minutes of advanced information truly worthless?

Does the average wealth of society increase or decrease with the addition of the arbitraging Galileo?

I think it increases.

Inequality increases, true, and there are other dangers in inequality.

But the median wealth of everyone also increases because now people know 5 minutes earlier if they're going to eat oranges tonight. Presumably that information is worth something, and I think there is a good argument to be made that the value of that information is equivalent to the profit of all the arbitrageurs.

I believe in wealth redistribution, but this question is less about inequality and more about the utility of that advance information.

Intuitively it does seem ridiculous that 3ms of ping is worth $300,000,000. But we're dealing in a global economy with nearing 7 billion people, and 3ms * 7,000,000,000 adds up to 8 months of human life.

It would be a problem if the fiber line owner, with his wealth, gains some kind of hegemony over the world and enforces some kind of human-life-crushing totalitarianism. But the inequality argument is most sensibly solved by wealth redistribution - taxes, welfare, public healthcare, etc. Rather than outlawing arbitrage.

> Is that 5 minutes of advanced information truly worthless?

>But the median wealth of everyone also increases because now people know 5 minutes earlier if they're going to eat oranges tonight.

Since we can assign an arbitrary utility value to that knowledge while not changing the profits telescope man makes (how much is the existing stock of oranges and separately the existing stock of bananas now worth, now that telescope man sees a boat of bananas has arrived unsunk and each banana must match an orange? He can use this to revalue the two and sell or buy from/to people who only know the old valuation).

The utility you claim isn't directly related to the telescope-man profit. Certainly not equivalent as you seem to think:

>I think there is a good argument to be made that the value of that information is equivalent to the profit of all the arbitrageurs.

What makes you think there is a one-to-one correspondence? Can you make the argument, or do you just think there is a good one to be made? If you already made the argument I didn't see it, and if it involved multiplying ping delta by current world-population then I am definitely not clever enough to understand it. ;)

("go by nightfall" in my original post should have been worded "go bad by nightfall")

My point regarding pings was that tiny advantages in speed, if leveraged across enough people, become large advantages in speed, and if a single person provides a tiny bit of value to many people, then that adds up to a lot of total value produced by one person and therefore a big paycheck for that one person.

The telescope information is valuable to fruit sellers and fruit consumers alike. If Bill is very hungry today, but Joe is not, then Bill is rightly willing to pay a higher price for fruit.

Let's say that the loss of one ship raises the fruit price from $1 to $10. Bill wants to eat today and so he is saving his last $10 for when the fruit arrives.

As the ship crosses the ocean, the closer it gets to shore, the less risk there is in Bill spending his other $9. He wants to buy beer but is unwilling due to the risk of not being able to afford fruit.

To Bill, there is clear value in knowing how far along the ship has gone. The sooner he knows the better. Since knowing the ship is coming frees up his $9, this information also provides value to the brewer who will be able to sell his beer sooner. If the brewer sells his beer sooner, the brewer will be able to buy himself new shoes sooner, and so on. The 5 minutes of foreknowledge results more economic activity, and higher wealth in general. Bill gets his beer sooner, making him happier sooner. This has value. Mr. Guinness the Brewmaster gets his shoes sooner, easing his ankles sooner, and this is worth something. Every little bit of foreknowledge helps.

All arbitrage follows this same model. Whether the arbitrage profits are collected by one person or many is not relevant. There is utility in prices that accurately reflect supply and demand. The sooner prices reflect reality, the better.

If there are going to be fewer oranges available, the price should be higher to ensure an efficient distribution - a distribution to those who most value the oranges.

Galileo made prices 5 minutes more accurate, and while that is unsexy, it saved 5 minutes of inefficient trading. In aggregate, those 5 minutes end up being worth a lot.

Those minutes provide some level of utility equal to or greater than Galileo's profit. If Bill did not get at least $10 worth of utility from his oranges, he would not buy them, and Galileo would not receive Bill's money. These are voluntary transactions after all. Bill can always starve until tomorrow, or boil his shoes, or go forage for berries.

Fast information is worth something in itself.

First you were arguing that the profit equaled the utility exactly. Now you are saying something different. Taking your new example, since the trading profits are roughly the same whether the advantage is 5 minutes or 20 seconds so long as the ship is seen first by telescope man, yet the probability of Bill mistakenly wasting his money on beer when he should have saved for bananas does depend on that length of time, then the utility and the profit aren't equal; and as the time advantage approaches 0, (and thus the utility you are talking about here approaches 0) the profit remains the same.

You are saying there is always some utility. Ok fine, but it gets arbitrarily close to 0. In the case of HFT there is also some cost (brain-drain for proprietary R&D, power drain for datacenters, etc.). Which outweighs which? What timescale do we have to get to? We are already well below it.

I was wrong saying profit equaled the utility exactly.

There is clearly an opportunity cost with HFTs, and I would hypothesize that at a certain point there is a better ROI in other fields for those people. There should be diminishing returns to investments in HFT, right?

This is such a theoretical argument and my conclusions are based mainly on the idea that if there is no utility people would not pay for it. It does seem ridiculous that moving 3ms faster wins you so much, but I'm just not convinced that there is definitely no utility there. Often economic utility is very hard to intuitively recognize -- often it is not recognized until people start intervening in the market and bad things start happening. If you ask computer scientists which jobs are useless, provide no utility, and should be eliminated, they would eliminate huge sections of the economy.

HFTs tend to earn the spread - this is what it means to sell liquidity. If the bid/ask is $10.00/$10.05, the HFT might make $0.05 on that trade (assuming the market doesn't move before he can execute the other side).

If you don't wish to pay the spread (the HFT's "rent"), use post only orders. Your are guaranteed not to pay the HFT rent, but you run the risk that your order may never be filled.

http://www.nasdaqomxtrader.com/content/ProductsServices/Trad...

Continuous auctions have the advantage of being continuous at the expense of volatility.

You could instead have a single auction every day and this would be more fair but less available.

Your example is not unlike how specialists control the price for markets.

You should probably look into how these things actually work.

Bitcoin is a market that grows at a fixed rate!

The stock market is kind of a joke...

Artificial Intelligences warring with each other on stock markets?

Witness the birth of Skynet, it's not going to be a military machine but a predatory corporate AI. Originally designed to devour competing AIs on the capital markets only to turn on its masters when the funds run out.

Woah. Spooky. Someone tell WIlliam Gibson and Vernor Vinge. Hell, I'd write that story if I had the time!

EDIT

oh no! It's a predatory Corporate AI that has built in new monitoring functions. The news monitoring subprocess has discovered that there's a predictable correlation between WAR and stock market fluctuation.

What does it do, suddenly places a whole stack of extremem long shot PUTS and SHORTS then hacks into the military computers to instigate a war between two nations.

We've got a grizzled reporter or plucky young cop (maybe an SEC investigator?) who's doing routine investigations into this and that and uncovers a series of small time conflicts in bananna republics. (these were A/B split testing 'dry runs' the damned AI was reading the visual website optimizer's blog posts!)

Just as he (or she's) connecting the dots on the small time stuff there's a major tunabout and the AI has put a bunch of shorts ON US SOIL.... !!! (oh noes! all of america is at risk!)

Boom! Race around trying to uncover the source... but now the AI is on to him (or her) and has started hacking public records to declare him (or her) as public enemy number one.

Then ... ROBOT ATTACK!! Just as he or she is on the final sprint to disconnect the AI's network cables - military robots spring out of their boxes and start shooting. PEW PEW PEW.

ALso - after the cable gets unplugged of course we learn that it's already made it out into the CLOUD!

I love this story :) Just the right balance of total cheese and hard Sci Fi.

"Woah. Spooky. Someone tell William Gibson and Vernor Vinge."

cstross, actually: http://www.antipope.org/charlie/blog-static/fiction/accelera...

the news discovery portion fascinates me. I know that they are sucking in all of Twitter, as much of Facebook as they can, tons of RSS, etc.

recently there was a case where the actress anne hathaway was trending on twitter and it caused the bots to short berkshire hathaway stock, which dipped ~6%

arms race going on which will only open more opportunities to exploit these situations. feeding in all this public news data and automating trades makes a mockery of SEC insider trading regulation (because it is so easy to hide - the computer did it!)

I would love to read a full feature of what goes on inside GS et al but I know that the quants aren't allowed to speak about it and the companies do not discuss it. We get drip fed small pieces of info but nobody has really assembled a complete picture of what is going on.

I do know that the investment banks are some of the largest customers for most of the large enterprise IT co's (IBM, Sun, Microsoft, EMC, etc.). The IB's are also hiring mathematicians and compsci graduates like crazy - the tech these guys are producing must be cutting edge and at a very very large scale

Be sure to update this with your Kickstarter URL for contributions to have you write/publish this.:-)

In Daniel Suarez's Daemon (and sequel, Freedom), a dying programmer develops an AI that maintains a fair economic system for its participants, and harshly punishes violators. Great read. http://www.amazon.com/Daemon-Daniel-Suarez/dp/0525951113

Sigh. VWAP isn't an algorithm, it's a benchmark. Volume-weighted Average Price. An algorithm that is attempting to trade VWAP is trying to get the day's average price, whatever it is. And a good algorithm is quite hard to detect.

Most shops call their VWAP-benched algo "VWAP".

http://www.thetradenews.com/algorithmic-trading

VWAP algos try to beat VWAP over the order's timeframe, which is almost never the entire day.

no you read it wrong, they are detecting the true volume weighted price of an instrument, by sniffing all the little sneaky trades jettisoned into the market.

Brutal title (Not OP's fault).

The article basically suggests that algos are reducing market volatility.

Algos don't try really hard to get VWAP. VWAP is REALLY easy to achieve, _beating_ VWAP is what most banks are trying to do when they trade large blocks.

A lot of the stories about warring algos and detecting each other are just that: stories. Yes, it has happened, but by and large thats not the game thats going on.

Citation needed for your assertion that VWAP is "REALLY easy to achieve".

In the extreme case, consider you were on the other side of every other trade that happened in the market. You would by definition have achieved VWAP over some time period.

Trading randomly during any given time period probably comes close to achieving VWAP. I'm sure something a little more intelligent could probably achieve it in expectation.

VWAP is "volume-weighted average price," not "average price". Therefore, to replicate it, you need to participate in line with volume-at-time, not 1/N per time bucket. This requires you to estimate the volume envelope over the time period of interest. (Often a day, or in the case of hedging certain new issue convertible bonds, up to several days.)

Yeah, that's why I said in the extreme case you have to participate in all the volume. Of course randomly trading doesn't prove that you achieve VWAP, but I'm just saying that it's probably not terribly far off.

And how far is "not terribly far"? Pennies? Nickles? Dimes? Dollars?

I can say with a great deal of confidence that a "random" VWAP algorithm would, on average, suck.

For context:

Many brokers will guarantee an execution at VWAP. It's a competitive business. The last time I put in such an order, I traded a double-digit percentage of the day's volume and paid 10bp commission (0.10%).

Brokers guarantee VWAP only if they also are paid a very healthy commission. They generally lose to VWAP, but makeup for it in premiums.

Plus, if you are dealing with a large broker they may just cross your order with an offsetting order (or their own position, execute nothing in the market and keep the entire premium.

Yay! Building a volume model was one of my first projects as a quant.

Neat. Tell us about it?

Source article:

http://www.lrb.co.uk/2011/05/19/donald-mackenzie/how-to-make...

Not set to be published until the 19th of this month.

If you've done any daytrading, you've seen these algorithms in action. I know a few folks who could make a small fortune daily just by "fooling" the algorithms by setting extraordinarily high or low order prices, and rarely actually filling one of those orders. There is lots of money to be had (and lost) by mathematicians in the stock market, and there is a lot to be said for technical analysis... but it's a jungle out there.

there's nothing to be said for technical analysis. it doesn't work.

PBS documentary on Paul Tudor Jones from 1986:

http://www.tudou.com/programs/view/XH5W4vffBbY

Sorry that the first 10 seconds are shady japanese yogurt commercials, last time I looked the videos were $300 on ebay reportedly because PTJ had gone around buying up all the tapes (maybe the glasses?)

even though technical analysis may not have any merit, many people do use it in their trading decisions. thus, sometimes it is worth paying attention to.

If you depend solely on technical analysis and you aren't a total pro, then it won't work. For most people it should part of their overall trading strategy.

"Could make a small fortune..."

This spikes my BS mater just a bit. Are any of these people actually doing this? Doing it consistently? Making a fortune? (How small is small, btw).

If there was free money for the taking, someone would be taking it and squashing the opportunity out for the others. It's how markets (and especially the stock market) works.

For those curious, here's a link to the full text of the Isaac Asimov short story that was mentioned in the text, The Evitable Conflict: http://members.multimania.co.uk/shortstories/asimovconflict....

For people who speak dutch, there's a great interactive documentary for the iPad called "Money & Speed" that examines the flash crash and high-frequency trading:

http://itunes.apple.com/nl/app/money-speed-inside-black-box/...

The Core Wars, on a huge canvas.

HFT is the legalized way of insider trading and front running

Absolutely! If the computer 'decided' on these illegal strategies theres impunity!

Shouldn't we be hoping that more and more bots enter the markets? Sure, there will be a period when hackers will be able to exploit their weaknesses, but in the due time they will evolve and market will equilibrate, making today's speculative traders' job much harder.

Reply on news.ycombinator.com