Hacker News Reader

Comment by nether | Hacker News Reader

One of Quantopian's studies shows that backtesting poorly predicts live trading results: https://blog.quantopian.com/using-machine-learning-to-predic.... From the forums, it looks like 90% of the effort is devoted to getting a good backtest, a yardstick which might not have any bearing on reality. How do real quant traders deal with this discrepancy?

View on HN

> How do real quant traders deal with this discrepancy?

A lot of it comes down to access to information and speed. All the major traders and quants have access to what effectively boil down to mailing lists (that their companies pay top dollar for) that have the best analysts in the world producing in-depth analysis on various topics[0]. Also, many/most have access to a Bloomberg terminal[1] to get the Bloomberg new feeds and the AIM clone, which costs $25,000/year/person.

This is why amateur day-trading is silly. You're not competing in a level playing field- they know more than you do.

[0]http://www.investopedia.com/articles/analyst/03/031803.asp [1]https://en.wikipedia.org/wiki/Bloomberg_Terminal

Sounds like I should get into building tools for traders!

It's a good idea. The user base is a group of people for whom hundreds of dollars per month is peanuts if it gives them a 1% edge over their competitors.

You're right, backtest overfitting is a problem. At Quantopian, we spend a lot of time on education helping people avoid overfitting like this (https://www.quantopian.com/lectures#The-Dangers-of-Overfitti...). It is easy to make a backtest look good by tweaking things here or there and that's just textbook overfitting. If you follow good statistical principles for model development based on a sound economic basis, the backtest is more of a sanity check to make sure that whatever signal you have found survives the introduction of market impact. If you have consistency between the statistical aspects of in-sample performance (your backtest) and true out-of-sample performance (paper trading on entirely new data), there as an increased probability that you have found a viable signal.

Disclaimer, I work at Quantopian.

There was a great article in the WSJ about quant "factors"

>A former student of Mr. Fama, Cliff Asness, founder of quantitative hedge-fund manager AQR Capital Management, said he tries to avoid being caught out by false findings by trading on anomalies he can explain, economically or through investor behavior. To assess whether the market anomalies will continue, he looks for ones which carried on after being identified, can be seen in other markets or asset classes, and where minor changes to how they are defined don’t much affect the result. These include most famously value, momentum and corporate quality, among others.

https://www.wsj.com/articles/an-algorithm-an-etf-and-an-acad...

He also blogs for those interested - obviously geared more to the finance types:

https://www.aqr.com/cliffs-perspective

Real firms use tick data to backtest, not the 1 minute bars that Quantopian uses, and spend a lot of time simulating network characteristics such as data and order latency. They also use machine learning, which isn't possible on Quantopian, to build models, which requires downloading lots of data (and not just equities). No serious quant will ever use such a tool. You might as well go to the casino.

Edit: tick data is usually collected over time, because data sources have different characteristics, and you should be testing over the same realtime data that you'll be using to trade live. You need to know the bid/ask prices and volumes, in order to know where to place limit orders. Otherwise you are just paying commissions and spreads to brokers and market makers.

We are very mindful of overfitting our models, and very skeptical of backtests on their own. Successful strategies often have some fundamental theory about why they should work. If not, the backtest needs to be very robust, and demonstrate that it is not overfit by cross-validation, and other methods.

A Bayesian mindset. When investigating a signal you have to ask: Is it too good to be true? Does it make economic sense?

Then simulate rolling optimisation and trading to see what returns could have been achieved with knowledge only available at the time.

-Out of sample testing. Across markets, across time. -Logical economic explanations.

Reply on news.ycombinator.com