Comment by 2-tpg | Hacker News Reader

Comment by 2-tpg | Hacker News Reader

Parent Full thread

Using purely historical price data it is harrowingly difficult. There are 130 anonymized features, so that's unlikely to be only price data. It could include information on the order book, correlated assets, fundamentals, vectorized/embedded text, etc.

Besides, I bet you can train monkeys to do (slightly) better than blindfolded random throwing. Even with public data (replace satellite images with Youtube mentions, or number of links moving into a company website) it is very possible to do better than average guessing on quite a lot of assets (especially smaller and newer markets).

Most hedge funds, even with specialized expensive non-public data, are not magical unicorns. Their quants really may just run a gradient boosting machine and leave it at that. Some hedge funds even prefer linear methods, because this lowers risk through lower variance. Such models can be beaten by experienced Kagglers for sure. For one, I did.

One thing we need to be clear about is that you're not aiming to be better than average. You're aiming to make a profit. There are probably hundreds of thousands of day traders, there are probably <100 market makers and tradingfirms (far less than that for a some specific products) and you'll probably find 99% of the day traders aren't making systematic profits. There are lots of strategies that are much better than average and still worse than putting your cash in a bank.

You can aim for both. If you just aim for profit, then you can get lucky with just average, or even random, betting. If you find a weighted coinflip (which is not impossible), provided by how many times you can flip that coin, you will see steady systematic profits. Of course, majority of day traders are getting owned by the big players, and they would do better doing more reasoned and long-term investments. Most day traders are not even using predictive models though.

On that point - it's pretty clear that this Kaggle competition is highly likely to result in a decent number of submissions that make more money through luck, than other make through strategy.

Reply on news.ycombinator.com