Do you know of any (families of) examples of tabular datasets of any size (you can choose what "big" means) where deep learning convincingly outperforms traditional methods? I would love some quality examples of this nature to use in my teaching.
Consider: Predicting how much a customer might pay by end of month, with information we have at the start of the month.
In this example, if a customer had a record $10m of open invoices due by EoM and the largest payment amount received in prior months of $5m, the decision tree cannot possibly predict the payment amount will be ~$10m, even when the best feature indicates the payment will be $10m.
There are some hacks/techniques which can maybe reduce this issue, but they don't always work.
Also all models are a “mean of the subgroup of the data.” The prediction is by definition the conditional mean as a function of the input values.
PS: Its weird that you are being down-voted. I think your opinion is reasonable.
Retraining - online training solves this for the most part.
Frameworks - the only battle-tested batteries-included one I've seen is Vespa. Noone else publishes any of interesting bits. KDD is the most relevant conference if you're interested in the field. IIRC Xiaohongshu has some papers that can only really be done with NNs.