Comment by brrrrrm | Hacker News Reader

Comment by brrrrrm | Hacker News Reader

I'm heavily involved in this area of research (getting deep learning competitive with computationally efficient statistical methods), and I'd like to note a couple things I've found:

1. Deep learning doesn't require thorough understanding of priors or statistical techniques. This opens the door to more programmers in the same way high level languages empower far more people than pure assembly. The tradeoffs are analogous - high human efficiency, loss of compute efficiency.

2. Near-CPU deep learning accelerators are making certain classes of models far easier to run efficiently. For example, an M1 chip can run matrix multiplies (DL primitive composed of floating point operations) 1000x faster than individual instructions (2TFlops vs 2GHz). This really changes the game, since we're now able to compare 1000 floating point multiplications with a single if statement.

It opens the door to more script kiddies, not more researchers. I really think we need more researchers who understand inference from first principles and make models in view to furthering understanding as opposed to more fit(X,y).

I don’t say this naively. At least in industries, the weight of imposter data scientists I think is getting to a level that may cause the profession to implode due to customer disillusionment within the next 10 years precisely because fit(X,y) is so accessible.

Regarding 1)

I am not sure if you are not trading "high human efficiency" against increased risk of blowing up at some point. Good luck doing forecasting without thorough understanding of priors and statistics in general.

that's a good point. I guess as an addendum it's not just compute efficiency but also "statistical efficiency" (if that has any meaning?)

I think that term already has usage as a proxy for "lowest sampling variance"; for example the Gauss Markov theorem shows that OLS is the most efficient unbiased linear estimator.

I guess this is echoing your point 2, but I would have generally said that "principled" statistical models are less efficient these days than DL (see: HMC being much slower than variational Bayes). Priors are usually overrated but I think the risk is more that basic mistakes are made because people don't understand what assumptions go into "basic" machine learning ideas like train/test splits or model selection. I'm not sure it warrants a lot of panic though.

Agreed, I see the "lower barrier to entry" in this particular case as coming with potentially huge risks. IMO, statistics is vastly, vastly, vastly under-appreciated and under-estimated.

Reply on news.ycombinator.com