Hacker News Reader

Comment by periheli0n | Hacker News Reader

Images -> correlation in space -> Deep convolution networks

Time series -> correlation in time -> Recursive networks

Tabular data without clear correlation structure -> good old ML (ANN, SVM, DT, LR, KNN).

This is obvious when following the field since 2006 or so. Deep Convolutional Networks were considered a special case for data with local correlations at a hierarchy of spatial scales. Same for RNNs in time, although they came much later (when was the LSTM rediscovery again? 2016?)

For most data without clear spatial or temporal structure to exploit, the good old ML techniques will work just fine.

View on HN

Not only will they work just fine, they'll save you compute costs, training time and lower your need for data.

Not to mention the shedloads of “X is all you need” papers which you can ignore, because Bishop’s “Pattern Recognition and ML” book is actually all you need (plus perhaps a good reference on linear algebra).

Not necessarily true that MLPs are very compute expensive. It is maximum a couple of layers and if your input is sparse (categorical) you can gain even more. For some problems it can be the fastest and decent non linear model from my experience.

I don’t think that was the claim… MLP/ANNs are fine except for the difficulties around interpretability. DTs and LR are preferable on that front. Or an SVM if you know a kernel/similarity metric that kills it in your data.

It is true that MLPs are classic, but the regularizaton methods that apparently make a big empirical difference at this paper are new concepts (data augmentation, skip connections/residual blocks, dropout, batch norm, lookahead, stochastic weight averaging, etc.). They compare againts a good old MLP without the bells and whistles at Table 2 and the classic MLP is quite a poor performer (XGBoost beats a classical MLP very significantly). Which leads to the conclusion that we need all these recent deep learning advances on innovative regularization techniques to make the difference.

Reply on news.ycombinator.com