Hacker News Reader

Comment by niksmather | Hacker News Reader

Do neural networks work better than other models? They can definitely model a wider class of problems than traditional ML models (images being the canonical example). However, I thought where a like for like comparison was possible they tend to worse than gradient boosting.

View on HN

Gradient boosting handles tabular data better than neural networks, often because the structure is simpler, and it becomes more of an issue to deal with the noise. You can do like-to-like comparisons between them for unstructured data like images, audio, video, text, and a well-designed NN will mop the floor with gradient boosting. This is because to handle that sort of data, you need to encode some form of bias around expected convolutional patterns in the data, or you won't get anywhere. Both CNNs and transformers do this.

Would you agree/disagree with the following:

- It's not gradient boosting per se that's good on tabular data, it's trees. Other fitting methods with trees as the model are also usually superior to NNs on tabular data.

- Trees are better on tabular data because they encode a useful inductive bias that NNs currently do not. Just like CNNs or ViTs are better on images because they encode spatial locality as an inductive bias.

Absolutely agree on both counts. Gradient boosting is the most commonly known and most successful variant, but it's the decision tree structure that is the underlying architecture there. Decision trees don't have the same "implicit training bias" phenomenon that neural networks have though, so all of this is just model bias in the classical statistical sense.

Can NNs be made to be better than trees on tabular data with some further constraints, or something?

You might find this interesting: https://arxiv.org/abs/1909.06312

On tabular datasets less than ~250k samples, tabular foundation models now outperform boosting. Of course it remains to be seen how they'll scale to significantly larger datasets as the models improve.

https://huggingface.co/spaces/TabArena/leaderboard

Reply on news.ycombinator.com