Hacker News Reader

Comment by lettergram | Hacker News Reader

You can get data for free from places such as alpha vantage

https://www.alphavantage.co/

it's not enough data to test all of those 130 papers, he said he has also tested short term reversion trends. for that you need tick data(or atleast minute/hourly data)

Agreed — something about this post doesn’t pass the smell test.

1. He’s a profitable trader at a tier 1 firm who has the spare time to not only develop a series of algorithms based on 130 research papers, but also sufficiently backtest them in 7 months?

2. He said he looked at the past 8 years of papers, but refers to multiple models correctly predicting the 2008 financial crisis.

3. Where are the code samples?

Edit:

Lol, just realized his medium post ends with a crypto scam.

Ditto on the "130 papers in 7 months." I am not familiar with the field, but I assume the process would look like this:

* Read and understand paper

* Find and download appropriate input data

* Code paper model and validate (he said he wrote his own code)

I can see myself being able to do this for ONE paper in maybe a week. He claims he was doing 1-2 of these per day. Wow. So either there is some exaggeration on his part, or he is a total wizard in his field.

I think you are a quick study; typically it takes me a week to figure out the detail of what's being done in a paper. Getting the data and testing would take longer. Charitably he/she has a framework with all the data required sitting ready to go and is just writing wrappers to the models. But even downloading frameworks from Github and getting them working takes a couple of days - for me. For example I've been playing with the Graph-network code from deepmind for a few weeks - I had to learn how the graphs were represented, how to build them and access them and how the models were made and put together. Just working that out was a solid three day job. Now I can build things and test out what's going on in the examples and get a feel for the framework, probably (if there was a problem) I would be in a reasonable position to say "this doesn't work like they think it does" (it does, but no surprise) but unless you've done that leg work I think you can't really. I think a proper replication effort is really 1 man month of expert time - or really you're just throwing stones.

Depending on the complexity of the model it would take me at least a month for a single paper. What makes it fully unbelievable to me is the claim of detecting p-value hacking in many of these 130 papers while doing 3 papers every 2 days.

To make that claim for a single paper I would 1. have to be able to reproduce their p-value, and 2. spend enough time with the model to understand how/what assumptions were unfairly tweaked to get to that p-value.

Just running your own implementation of a model on your own dataset and getting an insignificant or different p-value is not enough. You might just have implemented the model wrongly.

Reply on news.ycombinator.com