Hacker News Reader

Comment by asdfologist | Hacker News Reader

I don't see how Numerai can avoid the multiple comparisons problem [0]. If people submit thousands of random models, then some subset of them will do a fantastic job in predicting prices in historical simulations but do poorly under real market conditions. As long as the models are black boxes, there's likely no good way to distinguish them from noise.

[0] https://en.wikipedia.org/wiki/Multiple_comparisons_problem

View on HN

This can be mitigated by evaluating all the models on a hold-out test set (similar to what kaggle does and what was done in the netflix prize). The multiple comparisons problem is also mitigated by the fact that the models wont be completely random, there will likely be some positive correlation between them.

edit: Also, by hoeffding's inequality the number of training examples needed for a given level of confidence is only logarithmic in the number of models (even assuming they are independent). See page 6 here: http://cs229.stanford.edu/notes/cs229-notes4.pdf

They would need a model built on top of the submitted models to weight or select which specific trading signals to act on. A plausible model might be walkforward testing on out of sample data, or as simple as trailing performance in live trading, or both. Still, with a large set of models, you won't be able to get around multiple comparisons problem entirely.

That doesn't matter though because you only a need small amount of good signal to be profitable in the money management biz. The larger issue is whether the aggregate signal quality is high enough to be able to pay for development and trading costs.

That's true only if stock market data is completely random and that there's no signal to predict on. That doesn't seem to be the case, considering hedge funds successfully use ML on historical data to capture alpha. You don't need a model to work forever to be successful.

It looks like you missed the point of my comment. I'm saying that Numerai won't be able to distinguish between zero alpha and positive alpha models, if all they're doing is running historical simulations on black boxes.

Reply on news.ycombinator.com