The rise of the financial machines | Hacker News Reader

The rise of the financial machines

economist.com

153 pointshourislate89 comments

Machines took over finance a long long time ago. The big firms like Renaissance and D.E. Shaw that ushered us into that age were all largely founded in the 80/90's.

yeah, this article lacks any material substance.

I'd recommend "Flash Boys: A Wall Street Revolt" for more substance

I don’t. As someone in the industry, almost nothing in that book is actually correct. Moreover, HFT is a very small part of the industry and is not particularly profitable.

Quantitative investing as a whole is becoming more and more popular, while HFT isn’t.

As someone in the industry, you're also unavoidably biased. This isn't snark or accusation, just a factual observation.

Someone in the industry might just have a slightly better idea of what's really going on and that's largely what HN is about.

Another factual observation is to look at Virtu's recent earnings numbers. They're basically pivoting as hard as they can to get out of speed-based HFT and into execution services, because the profitability of the former is cratering.

Yeah Virtu's market cap is about the same as Groupon. I hardly doubt they're emptying the pockets of every American here.

See also (as a rebuttal): "Flash Boys: Not So Fast"

A much more entertaining and informative book than it's title or pedigree would hint at. Genuinely recommended reading.

One of the most surprising details in the whole book was the trading system devised by the biggest HFT-complainer, "Thor". And not even the system itself. Trading firms have to connect to practically all the exchanges, because they need the ability to send in orders directly to one of them. In trading, adverse selection is a thing - if an institutional trader sends in a big order, liquidity providers (HFT firms) can get hammered pretty badly. So when someone is buying or selling in one exchange, the HFT shops will adjust (read: pull) their quotes on other exchanges before the same executions would hit them there as well.

The basic assumption of a liquidity provider can be summed up as: if someone is executing trades, they must know something more; the prices currently on offer are clearly wrong. Pull quotes before things get expensive.

Thor wasn't exactly an optimisation engine, it was more a synchronisation engine. It kept track of transmission latencies across all the exchanges and could coordinate order creation times to such a degree that the orders that it wanted executed would land in all the exchanges at almost exactly the same time. This would allow it to execute its orders on all the exchanges at the prices available at that very time, without giving HFT firms the time to communicate across exchanges to pull their quotes. If I remember correctly, even the book used the term "slam" for the behaviour.

Thor used a strategy that any half-decent engineer should come up with in less than 5 minutes. But it was considered unfair by all the other market participants. The book didn't tell much more about Thor, other than that its use was discontinued shortly afterwards.

Rather amusing, nonetheless.

that book is garbage

As someone who enjoys a lot of Michael Lewis's books, I completely agree. The arguments and explanations throughout make no sense, because it's a lot motivated reasoning trying to justify a false narrative.

It all made perfect sense to me. I'm not sure why people get so upset about it.

I suspect people can't tell the difference between the arbitrage that will happen no matter what (Chicago to New York, etc.) with the front running that came from seeing orders too large for one exchange and buying the rest from the other exchanges so they could sell it to the original buyer.

I mean that part is true and is exactly how markets are supposed to work, but the conclusions he draws are just bizarre. He makes the argument that HFT is the classic story of the fat cats stealing even more money from the little guys. But it couldn't be further from the truth. HFT's are helping the retail investors and actually "stealing" from the "fat cats" (institutional investors). Retail trading is now cheaper (free in a lot of cases), easier, and faster today than it has ever been, largely due to HFT firms and low-latency liquidity providing strategies. Also the ETF revolution and the massive democratization of financial asset classes and strategies is entirely predicated on HFT arbitrageurs.

But Lewis realized that the idea of HFT actually helping the little guys and hurting the big guys wouldn't be as sensationalist and provocative a story, so he decided to twist the facts to suit his own narrative.

Wait so you're saying that trade sniping is good for more than just the handful doing it because side effects? I've heard this pro-HFT argument before and always assumed it was biased BS.

That's really not true.

Trading is cheap now... because it's all been computerised. The exchanges are digital - as soon as you can do all the transaction processing by computers there is an entire army of (expensive) clerks that can be laid off who used to handle the paper work. HFTs occupy a niche where they can exploit latency between exchanges for profit, with occasional other activities that were considered to be far more questionable when it was people playing those games, not algorithms.

Just because things are computerized doesn't mean firms are willing to provide liquidity. The number of natural counter-parties at any one time are low, this is where HFT comes in, as market makers. Also if you look at the pricing mechanism of ETFs, they simply couldn't exist without HFT. Arbitrage is essential to ETFs, they simply wouldn't exist without low latency creation and destruction of shares.

>Trading is cheap now... because it's all been computerised.

This is true but the (fairly recent) move to essentially free trading for retail investors exists because firms will pay money to brokerages like Robinhood just to get the opportunity to trade with a pool of traders that are unsophisticated.

Am I reading this correctly that firms subsidize Robin Hood in order to fleece their customers? Curious to hear more.

GP is referring to https://en.wikipedia.org/wiki/Payment_for_order_flow

Paying for order flow doesn't fleece customers. Those customers are guaranteed to trade at the best price by NBBO.

So why would people pay for order flow? The people who do this are liquidity providers. What they want to do is sit there and buy a stock for $10 and sell it for $10.01 all day long. If they are buying and selling to you or me that is a great deal.

But what happens when BIG HEDGE FUND buys a shit ton of stock for $10. BIG HEDGE FUND probable knows the price is wrong at which the liquidity provider will sell a bunch of stock for 10.01 but then instead of being able to buy it for $10, all of a sudden has to buy it for higher. This is how liquidity providers can lose some of their profits.

So they pay brockerages like Robinhood that don't have big hedge funds as customers, not to fleece those customers but just to sell liquidity to them without worrying about getting picked off.

Absolutely.

A few players using them effectively is different than 30-60% of the market being run by algorithms. Granted I can't read much of the article because it is behind a paywall which seems to be very common on HN.

You have to account for the fact that a lot of the algorithmic volume is due to ETF arbitrage. We now have more ETFs than stocks in the US by a wide margin, and each new ETF likely creates another trading opportunity. There are also a lot of factor and smart beta strategies now, which I would classify as more systematic, but these could distort the market more than arb strategies.

Not disagreeing with you, but the arbitrage generally lasts only for a few ms, which makes the article's comment incorrect:

"Visit a trading floor today and you will hear the hum of servers"

In my experience, the machines doing the actual trades are as close as possible to the exchange, generally inside. What's in the trading floor are the client and ML programs that issue the strategies. That said, I've been away from the industry for a few years and it might have changed.

>Granted I can't read much of the article because it is behind a paywall which seems to be very common on HN.

https://outline.com/nGrnKy

https://github.com/iamadamdev/bypass-paywalls-firefox

Thanks this looks good

relevant

weak human + machine > machine > strong human

https://curatedintelligence.com/2017/10/20/kasparovs-law/

But strong machine > all?

IE. Chess / Go / Dota 2

lol, no, that's the point.

A better machine is just a tool.

One that someone who's smarter than you is going to use to win at the game of poker.

Hence Kasparov's law. You know, the chess guy.

>A better machine is just a tool

For now.

Yes, but they weren’t the dominant paradigm. The major push toward automation for both the buy and sell-side came after the crash of 2007/2008.

Moreover, very few funds pursue completely automated strategies. Quantitative giants like DE Shaw, Bridgewater, and Jane Street still execute large amount of discretionary human trades. Of course these trades are informed by models to a large degree, but if that’s the definition of automation, the markets have been automated for over 50 years.

I assure you, stock markets stopped being led by human trading long before 2007/2008.

I assure, as someone who worked alongside of traders and had access to hundreds of hedge fund portfolios, that you are wrong.

In one sense you both are right. When a portfolio manager puts in a large block trade, its been common for a long time for execution management systems to optimally handle the execution, by breaking the trade into smaller pieces of varying size, at various prices over a period of time, to try to get the best execution possible. Proprietary systems have been built which are very good at that. So while the gross holdings of a fund might be set by a human, the executions likely are handled by software.

Having worked on an Equity Derivatives desk for a time, I can tell you much of the exchanges' trade volume--as early as 1990--was from DOT (program trading) and super-DOT executions. Monthly volume leaders included Susquehanna Partners, who did almost all of it's transactions via computer (DOT) trades. Late 90s and early 2000s saw the rise of the Swaps market where one deal could generate literally tens of thousands of transactions. It wasn't Bud Fox with a Quotron and a phone doing those executions.

1) Buy-side and sell-side are completely different beasts.

2) Even today, bonds are traded OTC and mostly by people. In the 1990's, 100% of bonds were traded by people. Bond market cap dwarfs equity market cap.

3) Exotic OTC options were a big thing in the 80s and 90s. While that's not true today, when they existed, all of those were traded by people as well.

4) Most hedge funds today manage their positions with excel and trade based on analyst recommendations. They might have some fancy factor ops engine or something, but it's basically an after thought.

5) Around the dotcom boom, Goldman's equity desk had over 500 traders.

Fundamentally we are talking about a lot of different things. Different asset classes, buy-side vs sell-side, NYSE systems vs pit traders, etc etc. We can slice and dice these numbers in many different ways: total number of discretionary firms vs quant, trade volume vs assets, buy-side vs sell-side, etc etc. There's many ways to look at it that result in different conclusions. All I'm saying is that computerized trading is probably not yet the dominant paradigm and it certainly wasn't the dominant paradigm in 2007, much less 1990.

Sure there were high-touch traders, they were still employed up to and around 2017-2018. Since the early 201x's bonuses have been pared to fractions of what they were and each month, every bulge bracket bank lets some formerly well-paid traders go. Fixed income notwithstanding, it's been a tough road for those guys since 2007, have you been to the floor of a major exchange lately? It's lonely. Gone are all those seven-figure a year specialists. No, low-latency and HFT are the dominant paradigm, they are business today. Want more change? Downtown Manhattan used to be all financial. Every block had dozens of Wall St. firms, trading houses, research organizations, etc. Today, it's residential. It's lots of city, state and Federal Government offices. The legions of well-dressed yuppies rushing to get to the trading floor by 7:30 AM... they're gone. Perhaps your experience is different, but this is the reality.

Some Quant funds like rentech and peers (quieter) are almost entirely model driven though. They also tend to be the most successful.

"Machines are taking control of investing..... Funds run by computers that follow rules set by humans"

If the rules/algorithms are set and monitored by humans, the humans are the ones ultimately responsible. Computers are just tools which provide greater utility and control for investors than ever before.

That comes with its benefits and also with its potential downsides, but does not lessen from the fact humans are ultimate responsible.

With more sophisticated machines you just need less people. When the machines start building the systems and then monitoring the systems then the rules will change.

Humans may be ultimately responsible, but I think it's pretty clear that the more that decisions are made algorithmically, the less accountability firms running those algorithms take.

If there aren't people involved at every step of the way to inject humanity into decision making processes, it's much easier for companies to hide behind their "algorithms."

There is a reason algorithms are considered a laundering mechanism for accountability.

I believe the blossoming buzzword for it may be Mathwashing.

Accountability is in returns and losses to investors.

> "Machines are taking control of investing..... Funds run by computers that follow rules set by humans"

One of the points that the article makes is that this statement is changing, that computers are increasingly creating their own rules. The literal next sentence:

"New artificial-intelligence programs are also writing their own investing rules, in ways their human masters only partly understand."

You're right though, that the responsibility still falls on the humans. Anybody running algos that they don't understand should ensure that they are covered from a legal/ethical perspective.

Actually, the problem is not of the human masters' understanding, it's the ignorance of the human master. Some models are so involved, no one knows how they'll behave under certain market conditions, as we saw in the perfect storm of 2008 with credit default swaps, CMOs and CBOs. That's the danger, no matter how smart and clever the creator believes they are.

Is it really that we didn’t know how the models behaved in certain market structures, or that we were feeding them the wrong inputs?

The former. LTCM blowup in the late ‘90’s is a perfect example of a generally correct trade, but the market moving against them long enough to force insolvency.

Apparently, none of the Nobel prize winners thought to model out that particular adverse condition.

Bunch of smart people + one missed market condition = failure (eventually).

What I meant was the machine learning systems that have developed in the last decade since the 2008 crash. Many of these are black boxes with thousands to millions of variables, and it's very difficult to understand exactly why a ML algorithm made a decision that it did.

With ML this isn't entirely true. Proven by ML models trained to play games, they have strategies that were unknown to their human opponents.

It's disturbing that machines are the ones making investment decisions concerning human needs. It explains why real innovation has stalled. Algorithms can only make predictions based on existing trends and data, they don't actually have a vision for how to improve human society.

The machines are proxies for humans, they are programmed to suit humans and are turned off if they don't do what suits those humans. This is smoke and lies, it's camouflage to prevent people from focusing on the people standing on their necks.

real innovation has stalled because it has been prevented from happening by regulation. Just look at any real industry that involves creating actual things: Automotive or Housing, or Medical insurance. They're all choking from mundane regulations we never hear or talk about.

These machines are just trading and making value calculations on existing markets, not directly investing in businesses. I would hesitate to use the term “invest” at all, I think the term “allocate” is much more accurate

https://outline.com/nGrnKy

I see no annotations on the article, why did you post this link?

Outline can be used to bypass the paywall

Also, it's a long and annoying HN tradition to post the outline link (useful) without mentioning that it just bypasses the paywall (not useful).

If homo economicus does not exist we must surely be obliged to create him...

I never thought about it before but that's exactly what corporations are meant to be: "personhood"s that think only in economic terms. People get lazy and they run out of greed once they become millionaires and they want abstract things like love or popularity.

> they run out of greed once they become millionaires

In my experience, the opposite happens.

But they still don't turn into homo economicus. If they lose all care for love and popularity they might just as well find other outlets for irrationality, like bravado, hubris and generally taking things too personal (worst trading strategy ever).

I think of it as cancer. Society = body. Humans in society = cells. Some cells are small, some are big. But some cells, for whatever reasons decide to take far more resources than they need to fulfill their function. These are the cancer cells. They have a runaway feedback loop that will never stop on it's own.

Greed is good, it's how people eventually lose their money.

Quant funds have existed for a very long time. Rentech has been using neural nets, HMMs, speech and language processing in some capacity for decades

How do you know what they actually use? I thought they are super secretive.

Things leak. And knowing which models someone uses aren't that important. The devil is in the details of all these things. Which datasets, what features to feed in, how to clean them, etc.

It's actually my understanding that the core of Medallion is the data processing engine. The models themselves aren't that much more advanced than everybody else (iirc). Rather, financial data is extraordinarily noisy and often not i.i.d. and thus things like deep networks, applied alone, will drown in noise. The use of techniques from advanced statistical signal processing, information theory, conpressed sensing, information geometry, etc. allows the data processing engine to automatically and autonomously extract this signal from a sea of noise.

Ya, that's likely correct. It's all about how you transform the data you feed into the model, and which data you choose to look at, not to mention how you group companies together. That part is huge too. Once you've done all that correctly, off the shelf ML models will solve your problem just fine.

It's my understanding that RenTech essentially takes in any data and lets the pipeline figure out what's important.

One founder of RenTech studied under Geoffrey Hinton (some call father of deep learning). This was before DL was known outside of Toronto.

RenTech is also known for hiring/poaching almost everyone from a former IBM voice research team.

Leonard Baum of Baum-Welch/EM algorithm fame (for training HMMs) was one of the first recruits to rentech. They pretty much scooped up the entire IBM speech and language processing team (Mercer's 1980 paper is cited as eqn 1 in the GPT-2 paper). Robert Frey, formerly of rentech, confirmed their past use of NNs.

These are all public. Beyond that I can't say :)

How do you know that? People in the industry haven't confirmed much of anything about them except they make a retarded amount of money.

Copy/paste my other comment:

These are all public. Beyond that I can't say :)

I am hopeful that wide deployment of decentralization technology can help monitor and aggregate real world data relevant to human needs so that these algorithms can be tunes to serve people in general rather than the 1%.

If this even remotely interests you, then I'd highly recommend "Homo Deus: A Brief History of Tomorrow" by Yuval Noah Harari

As someone in the industry I would caution everyone techy to take such articles with a grain of salt. Most of these articles are PR pieces by WS to lure tech talent. Yes tech is making a huge impact and it has been for a whole but not replacing everyone.

I'm still waiting for an AI that can help me invest in the most lucrative Dutch tulips.

This is known as "tech unicorns" these days.

Ask Softbank, as I suspect they have found it.

Agree with comments. Not only quant funds have been around since forever but in fact the new trend is in a hybrid approach of discretionary/fundamental & algo/quant strategy -- see DE Shaw article on their "quantamental" approach in:

https://www.ft.com/content/0364850c-3ebf-11e9-9bee-efab61506...

There's a lot of opportunity in human aided systems, i.e. CEO says they are getting a lot of new web traffic after a redesign, but aggregated website analytics show that it isn't the case. Using credit card data in retail, etc. to either trade around a position, or inform a thesis, etc.

http://archive.is/p3RKW

"Machines are taking control of investing—not just the humdrum buying and selling of securities, but also the commanding heights of monitoring the economy and allocating capital."

I think machines investing is less than 1%;

Total US equity market cap is around 30 trillion. Total money in ETFs is 4.5 trillion. So at the very least machines are investing over 10% of capital. Probably more like 20-30%

Of course this only US and only equities, but it's still well above 1%.

> Industries from pizza-delivery to Hollywood are being changed by technology, but finance is unique because it can exert voting power over firms, redistribute wealth and cause mayhem in the economy.

This is quite the sensationalist bullshit. What does this even mean? The rest of the article is paywalled, so I don't know for certain where the author is going with this, but what mayhem is there to speak of?

As noted by other commentors, retail investors are much better off these days because of tightened bid/ask spreads.

After working in HFT for nearly 10 years I've learned the firms that do the best do so by proving value to the market. Through competition with one another, retail investors are much better off by getting better prices from market makers who do a better job.

This article isn’t very good or informative. Many, maybe even most, of these “black-box” strstegies are derivarive of work done over 40s years ago at Chicago. Certainly there are a few firms doing truly innovative work such as RenTec or 2 Sigma. But a lot are like AQR: just using publically available academic knowledge but doing it better than everyone else.

Also I think this article is overstating the prevalence of quant strategies and funds. At my old portfolio analytics firm, we had hundreds of hedge fund clients and not a single one was doing fully automated trades. Of course they had optimization engines and tech to squeeze a few bips here and there, but the fundamental ideas were still generated the old fashion way (through sleep deprived analysts).

> (through sleep deprived analysts).

You misspelled 'material non-public information'.

Reply on news.ycombinator.com