Hacker News Reader

Comment by ktamura | Hacker News Reader

Their dataset reeks of startup hustling.

I just downloaded the training set [1] and plotted some of its descriptive statistics [2]. It looks that all features are uniform distributions and the response variable is Bernoulli coin-flipping. In layman's terms, you can't really come up with a good predictive model with this training set.

I give them the benefit of the doubt that they wanted to have something in place to push the website live, but I cannot imagine any serious data scientist not noticing this.

[1] http://datasets.numer.ai/57feb95/numerai_training_data.csv

[2] https://cl.ly/051G3Y2Z2O0W

View on HN

The features are all encrypted using homomorphic encryption, which ends up mapping the overall distribuiton uniformly onto [0, 1]. If you play with the dataset, though, you'll find there are some significant correlations between the different features and you can make a model that does substantially better than chance.

I would like to see a comparison between random generated uniform features and these features. Can you fit the noise and come up with "statistically significant" predictions? If so, is anyone doing any better than can be done on random data? If no one is doing better than that, what is the likelihood that these aren't any better than monkey with a dartboard?

I would love to see peer reviewed articles from numerai with some of their behind the scenes results.

Whenever I make a model I always do cross-validation to make sure that I'm not overfitting. If we were just fitting random noise I would always see the performance of my test set being no better than chance (or worse).

The key property that you should be testing for is dependence. It would be surprising if you could identify anything in the descriptive statistics, that would make this all child's play?

I can't imagine any serious data scientist not knowing that.

>to push the website live __ The website has been live since like December 2015. There's a new tournament weekly. (I think it used to be every second week)... New tournament, new data -- always encrypted. Just look at the leaderboard; look at some of the participants' submission history

>any serious data scientist __ - I mean, any human - data scientist or not - would see at least that the website wasn't just pushed live; or at least have noticed the film is from like August... ^are you a serious data scientist? If you are (which I assume you are - just also an eager beaver to download the datasets without learning about the company sufficiently) -- then you'd do well to join the tournament and have a real go before you [attempt] to knock it as you have here -- and then come back and give some real feedback -- I'm sure everyone would appreciate to hear the 'after' report from you -- since you're so kind as already to dish out benefits of doubt :) Looking forward to catching you above controlling capital!

Reply on news.ycombinator.com