Comment by a008t | Hacker News Reader

Comment by a008t | Hacker News Reader

Methodology seems to be poor, like with most publications on finance.

1. How did they split the data for train/validation/test sets? If all of the sets had the data from the same time period, or the same companies end up in multiple sets, it is a major flaw. For example, if outperformance was persistent for the companies over the time period considered, the model may simply learn to identify specific companies by their filings.

2. What is the variance of the out-of-sample performance? Given that their dataset is very small, and the model performed badly at predicting high returns and reasonably well at predicting good returns, what are the chances of getting those results by luck alone?

3. How has the model performed since then, on the most recent filings?

4. Why use a convnet? Would gradient boosted trees not perform just as well/better but be more interpretable? Methods like the ones in the eli5 package can help get an idea of why the model makes a particular prediction, which could help sanity check the model.

The sense check here is "If I had a large team of people reading 10-k reports, could I profit from the analysis?". Well, it turns out this is exactly what many asset managers do. If it was easy to profit from this sort of information, we'd be awash with asset managers beating the market. We aren't.

There are two mutually exclusive conclusions that you could draw from this: one, the informational advantage you get from reading company reports is small, and TFA is overstating their case. Two, convnets have discovered a way of reading reports that conveys a big advantage, and further, human readers can't replicate the process.

My money is on the first hypothesis. The second is an enormous claim, and so demands an enormous amount of evidence.

> If it was easy to profit from this sort of information, we'd be awash with asset managers beating the market.

Unless it's a Moneyball-like situation, where managers theoretically have access to all the same metrics, but actually use inferior squishy human judgement instead of hard data.

Due to the amount of data across the entire industry, it's totally conceivable that "human readers can't replicate the process".

That said, I'm a bit of a pessimist, and concur with your suspicion.

This is not really finance research so do not confuse the two. It is CS/ML with a financial application; two different things. Return predictability low in real data, even if you control for time- and firm-fixed effects.

When applying CS/ML to finance, methodology is absolutely crucial. Yet most work - even work done by quants in major sell-side banks - pays little attention to methodology; it is often naively assumed that the methodology applied to classifying cat images can be directly transferred into the domain of finance.

Here is an article that explains the problem well: http://zacharydavid.com/2017/08/06/fitting-to-noise-or-nothi...

But that's because it is being applied by people who come from classifying cat images and have little domain knowledge.

>"When applying CS/ML to finance, methodology is absolutely crucial."

When wouldnt it be?

In other areas, we can often assume that each example in our dataset is an independent sample from some underlying distribution. We can also assume that the distribution stays constant. Furthermore, we have fairly high-accuracy ground truth labels. Also, we know that everything we need to classify a sample is in the data we are given. We may also have more data to play with than we really need.

Take handwritten digit recognition task like MNIST, for example - the way we write the number '5' does not change much, and the distribution of the different ways people write that number stays pretty much the same over time. The labels we are given are very accurate and everything we need to classify the image is in the data.

All these assumptions mean that picking the right methodology is fairly easy - anyone can read a short tutorial and get it right.

None of the assumptions I listed hold in the world of finance. There is no standard methodology that everyone can agree should be followed. You really have to be very, very careful in order to produce useful results.

It sounds like you just need to use a different methodology, not that methodology is unimportant for whatever ml 101 examples that became popular like MNIST, etc.

Yes, but I also think it's a lot easier to get it wrong in finance without even realizing it than in many other domains. Relatively more difficult to screw up the methodology on MNIST. Many mistakenly assume that the methodology from MNIST transfers directly to finance, which makes the problem worse.

ALl of that means that when talking about applying ML to finance, discussing the methodology in detail is a must. If it is not talked about or only briefly mentioned, from my experience that usually means that the methodology used is rubbish. Whereas you don't need to focus as much on the methodology when talking about MNIST-like problems - one can usually assume a reasonable one is used.

If you use financial performance to measure the success of a model, methodology is critical. It's easy to have models that generate outperformance on paper, in reality it turns out that nearly all of those had some flaw in them.

>"If you use financial performance to measure the success of a model, methodology is critical."

When wouldn't it be? And why does this sound exactly like the other post I just read?

If you compare to other results, small problems with timing usually don't impact results as severely. You don't get completely different results if you forget to move one time series by just a few hours.

As for the second part, I don't know. Probably because we had the same thought? Not sure which post you refer to.

1. Are you really assuming that researchers at MSFT screwed up the training and validation sets?? 4. In what way is GBT more interpretable than an ANN??

Yes they did. I noted them in another comment.

I don't think that was MS researchers - I think they just publicised this bit of research.

With GBT, you can check why a particular prediction was made - roughly - by navigating down each tree for a particular sample and summing up the influence of each feature on the final score. Then you can see if something weird is having a large effect on your score. Can you do something similar with ANNs?

Reply on news.ycombinator.com