Hedge funds embrace machine learning up to a point | Hacker News Reader

Hedge funds embrace machine learning up to a point

economist.com

96 pointsjosephby41 comments

Funds would not share what they are doing because then others would do the same, narrow spreads and reduce profits for all participants. But you can get a good idea of what they might be working on by some of their workers' expertise. Notice how places like RenTech (https://en.wikipedia.org/wiki/Renaissance_Technologies and https://www.bloomberg.com/news/articles/2016-11-21/how-renai...), Two Sigma, and others hire loads of NLP and machine learning academics.

Perhaps these newer hedge funds just want to give off the perception that they know what they’re doing — I’m more sure about a company like RenTech since they have been through a recession and market crash, and has still come up net positive. So how much of this is simply the market rising; how much of this is due to chance? If we can remove these confounding factors then we could maybe actually evaluate the efficiency of ML-based hedge funds (or regular ones).

"That's my quant".

[1]: https://www.youtube.com/watch?v=FoYC_8cutb0

Many quant funds are market neutral. For every dollar long they have a dollar short. Such funds aren’t necessarily benefiting from an upward rise in the market.

Are you referring to gamma hedging or another strategy?

It's called delta neutral afaik.

Agreed. Both rentech and two Sigma were at NIPS this week. Probably a good indicator of where some of their interests are.

As were Cubist, Citadel, Winton, Optiver, Jump, MAN AHL, Voleon, DE Shaw, Hudson River and PDT. Not sure if I missed anyone.

I worked at a hedge fund for 9 years. Regression models are used for EVERYTHING. Regression is machine learning. This article has a narrow focus and doesnt understand what machine learning is.

Usually « machine learning » is used to talk about things beyond basic statistical methods that predate computers. On the other hand, basic statistics are often good enough...

Other machine learning techniques widely used on wall street for over a decade - Dimensionality Reduction - Clustering - Classification techniques using trees, SVM, bayesian models

I feel like machine learning is one of those terms that has lost all defined meaning, under a bombardment of hype, corporate buzzwordism, and other horseshit.

The article distinguishes between using a regression to generate trade ideas for a human (say, data science), or actually having the machine trade on the idea without intervention (that would be AI).

The article claims the former happens but not the latter. This is entirely compatible with your statements.

What was your ultimate salary?

An anecdote from 1990s continental Europe.

A friend of mine, upon reaching a status of a "high-net worth individual", was invited to invest in a hedge fund associated with a bank of regional importance (major for that small country). They demonstrated the best of the tech of the time, and then shifted the tone to, "but we also use more traditional approaches".

Unusually large portion of their "consulting research staff" was female and spread Europe-wide. Not because they were progressive or pursuing diversity agenda; these were call girls. The call girls were paid for timely tips. Even if they only report that a particular top executive stayed overnight far away from home, was in a foul mood, and a couple of top-ranking colleagues were present, this can be used to detect an significant internal event not reported publicly. If the girl was crafty enough to extract more information, even better.

I don't think technically sophisticated funds are going to be sharing any details at all for pieces like these. There is bound to be fancy ML in play at hedge funds, though - do not underestimate the lengths to which people will go to get an edge in the markets.

"Share-price time series going back decades still contain far less information than, say, the image data used to train Facebook’s facial-recognition algorithms."

What?

Clearly these ML models aren't trying hard enough. Correlating stock movements with the weather, geo populations, ... there are potentially infinite patterns buried in the data that a hedge fund could uncover.

This is a fluff article and they know it, plenty of major quant firms analyze satellite images with neural nets to make trades.

Do they do that ? And why exactly do they do that ?

I can pay for high resolution images of every walMart in the US- Use math- estimate foot traffic- estimate average sales per person- I can estimate profitability of Walmart. Let's say it's a good quarter. I buy a bunch of Walmart stock - they then disclose quarterly earnings- investors see the good numbers- the share price goes up. My position is long-I sell. Profit.

edit: obviously many steps of this process can use machine learning in general and deep learning in particular

no you can't. take the absolute highest resolution images you can buy, and take 100 of the best mathematicians and using only maps you won't even be able to guess within 20%...analysts can already estimate within 1% of expected earnings

> take the absolute highest resolution images you can buy, and take 100 of the best mathematicians and using only maps you won't even be able to guess within 20%...

Why is that?

that example was more hypothetical of possible uses of satellite images. idk whether or not its too coarse to be a profitable signal

https://www.cnbc.com/id/38722872

Walmart’s last quarterly EPS beat analyst consensus by >3%, according to an aggregation by the WSJ.

Wouldn’t it be easier to create an incentive for customers to directly give you data about where they shop? Like make an app that pulls up coupons specifically for Walmart..

For starters satelitte images can help infer crop farm size and health and use it as one of the many signals to predict corelated stocks.

i think some firms buy analyses of imagery to predict oil market movements by looking at whether oil storage containers are empty or full. could probably imagine doing similar things in other commodities markets (corn etc)

Yes, but not necessarily to trade stocks. There is a more direct use of this data - to predict demand.

not to trade stocks necessarily but to trade commodities

There are about 30,000 institutionally-investable stocks in the world. They have ~252 closing prices per year. This is the thing you're trying to predict, so it forms the outputs of your training set (be sure to hold some back for out-of-sample testing and there are probably distinct regimes you need to switch between). Yes, there may be lots of features (weather, geo populations, etc., as you point out), but you're often hard up against the curse of dimensionality here. Sure, intraday data is more voluminous, but there are fewer economically reasonable influences on that.

Also commodities, forex...

That only adds a few dozen items. There's only so many currencies and commods, bonds. Even if you include really illiquid stuff maybe you get 150 things.

> There's only so many currencies and commods, bonds

Currencies are traded in pairs. Bonds are numerous, if illiquid off the run. Not to mention the bursting menagerie of derivatives our species tends to.

Highly correlated. I mean you can add option surfaces, but to count each option as separate is a stretch.

Some people are already trying and succeeding. And since a lot of these signals already incorporated into the market prices of various financial instruments, you can apply deep learning techniques to these available prices (and time series) to get similar results, to make buy and sell decisions/algorithms.

Yes, but the quote is talking about “share-price time series.”

Just 32x32 image space is, like, larger the the atoms in the universe. Even per-tick financial data is quite small in comparison.

Or, from another angle, say you've got 10 ticks of a stock per second for 12 hours = 432,000 data points. This is the same as a single (relatively small) 380x380x3 image. If you assume there are roughly 30,000 instruments like this in the world of this volume, that's probably, I dunno, the number of images facebook sees in a second. Those are all guesses but you can see the scale of differences involved here.

Hello Frame Problem.

Some good answers in the thread below, but...

1. This is all observational data - even if you have all the features in the domain it's pretty clear that you won't have seen all of the generative theory. 2. You don't have all the features because the features change, renewable power suddenly makes average wind conditions relevant, and so on. 3. This domain is clearly chaotic; you can induce the various attractors and theories, but you cannot measure the conditions well enough to predict the shift between attractors.

There are many people who are using ML to predict closed parts of the financial world where they have an deep and well founded understanding and excellent data. But there are lots and lots of people who are feeding data into models and getting good results and then going on good runs who are going to get a shock.

I like the shoutout to Numerai. I think they are so left field that traditional hedge funds don't even have the ability to understand how much better Numerai can be.

Huh? I did my commute with a guy who worked for an automated trading shop when I worked in London in 1994 and he talked constantly about what sounded just like machine learning.

Reply on news.ycombinator.com