Hacker News Reader

Comment by whoisninja | Hacker News Reader

i asked the author where did he source his data, his reply was "i scraped data"

how can you scrape pricing data? not every data in this is on public domain, otherwise there would be no Bloombergs, CapitalIQs selling data for millions(sure they're overrated and overpriced but still!). Or in other words, if he is right - he can sell data and make millions. no need of looking for an investment strategy. just my skeptical side saying :)

you need clean data to accurately test ideas. for instance getting tick data is quite expensive. most universities have free access to Bloomberg, CapitalIQ etc. datasets the reason professors can test and also the reason some smart guys in the industry work for university on the side

View on HN

You can get data for free from places such as alpha vantage

https://www.alphavantage.co/

it's not enough data to test all of those 130 papers, he said he has also tested short term reversion trends. for that you need tick data(or atleast minute/hourly data)

Agreed — something about this post doesn’t pass the smell test.

1. He’s a profitable trader at a tier 1 firm who has the spare time to not only develop a series of algorithms based on 130 research papers, but also sufficiently backtest them in 7 months?

2. He said he looked at the past 8 years of papers, but refers to multiple models correctly predicting the 2008 financial crisis.

3. Where are the code samples?

Edit:

Lol, just realized his medium post ends with a crypto scam.

Ditto on the "130 papers in 7 months." I am not familiar with the field, but I assume the process would look like this:

* Read and understand paper

* Find and download appropriate input data

* Code paper model and validate (he said he wrote his own code)

I can see myself being able to do this for ONE paper in maybe a week. He claims he was doing 1-2 of these per day. Wow. So either there is some exaggeration on his part, or he is a total wizard in his field.

I think you are a quick study; typically it takes me a week to figure out the detail of what's being done in a paper. Getting the data and testing would take longer. Charitably he/she has a framework with all the data required sitting ready to go and is just writing wrappers to the models. But even downloading frameworks from Github and getting them working takes a couple of days - for me. For example I've been playing with the Graph-network code from deepmind for a few weeks - I had to learn how the graphs were represented, how to build them and access them and how the models were made and put together. Just working that out was a solid three day job. Now I can build things and test out what's going on in the examples and get a feel for the framework, probably (if there was a problem) I would be in a reasonable position to say "this doesn't work like they think it does" (it does, but no surprise) but unless you've done that leg work I think you can't really. I think a proper replication effort is really 1 man month of expert time - or really you're just throwing stones.

Depending on the complexity of the model it would take me at least a month for a single paper. What makes it fully unbelievable to me is the claim of detecting p-value hacking in many of these 130 papers while doing 3 papers every 2 days.

To make that claim for a single paper I would 1. have to be able to reproduce their p-value, and 2. spend enough time with the model to understand how/what assumptions were unfairly tweaked to get to that p-value.

Just running your own implementation of a model on your own dataset and getting an insignificant or different p-value is not enough. You might just have implemented the model wrongly.

Reply on news.ycombinator.com