Hacker News Reader

Comment by huac | Hacker News Reader

Well, we talked about using cross-validation and the bootstrap/bagging methods. I was surprised that he hadn't heard of these ideas, much less using.

Deep learning variants are definitely farther out; I think the current lack of interpretability makes for a real strenous case wrt economic applications.

I don't really want to draw conclusions from n=1 convos but a couple other links had piqued my curiosity towards this relationship: https://www.quora.com/How-will-Machine-Learning-affect-econo..., https://news.ycombinator.com/item?id=11460412

View on HN

These methods are not usually taught in economics courses. Why? Not enough data. In most cases economists are dealing with survey data, time series data and panel data. The benefits of cross validation and bootstrap/bagging grow with data size. When you're dealing with minimal amounts of messy/misbehaving data these methods lose their power.

Other methods become more important ie: Instrumental Variable estimation, Probit and Tobit models, Vector Autoregressions, Vector Error Correction models. Im sure your econ PhD student friend would know what these are.

A different tool-kit to solve different problems.

"Not enough time" and "harder to do inference" are bigger reasons. It's hard to see why cross validation would do worse than the AIC or BIC for lag length selection in VARs, the bootstrap is widely used for inference for all of the models you mentioned, IV isn't known for its exceptional small sample properties, etc. People are working on this stuff, but it takes a little while to get it to work well for Econ research, and it takes some clear new empirical findings before it becomes mainstream enough to teach it in classes. Everyone recognizes that there's a lot of promise, though.

grayclhn, you might find Rob Hyndman's article on cross validation helpful. http://robjhyndman.com/hyndsight/crossvalidation/

" Asymptotically, minimizing the AIC is equivalent to minimizing the CV value. This is true for any model (Stone 1977), not just linear models. It is this property that makes the AIC so useful in model selection when the purpose is prediction."

Interesting, I think bootstrapping is overused in financial modeling, because most long term financial time series don't obey the assumptions, covariance stationarity, normal distribution of residuals etc. A lot of regime changes, fat tails, latent predictors that you don't realize are important until they break your model.

disclaimer: hand-waving

OTOH there are a lot of situations where you need to model something with a lot of potential predictors and limited data. e.g. testing a macro model with 100s of potential predictors and 100 years of relatively poor macro data.

ML might find interesting relationships in those cases.

Traditional statistics has a strong theoretical foundation, you assume a bunch of things about the shape of the data, and you can prove your estimator is best and what the error looks like based on the amount of data. It makes heroic assumptions about underlying data that we know don't apply.

So it often doesn't work well but we know why.

ML just wants to find things that work well in cross-validation without worrying too much about proofs... ML is a little like QE ... it works but we don't really know why.

Anyway, any sufficiently complex ecosystem is a market design, makes sense that if you have a sufficiently valuable ecosystem you would want some people who study markets.

Thanks, that's a bit surprising, but definitely helped me make up my mind on what I'll teach in the last few weeks of my class next year. The bootstrap is a weird omission, but economists usually only see it for variance estimation and inference, not in other contexts.

Reply on news.ycombinator.com