I've put in production numerous models with millions of tabular data points and a 10^5-10^6 feature space where tree-based models (or FF nets) outperform more complex DL approaches.
I'll chime in with billions of data points and 100-300 feature space with some smart feature engineering outperforming DL in runtime/compute (by orders of magnitude) and performance. But the domain was very specific and everything prior to the tree was doable with matrix operations, with the tree model summarizing a mixture of experts that chose optimal features.
Damn, this sounds fascinating! Have you shared more anywhere eelse?
What kind of freakish tabular data do you have with a million columns??
A badly defined one probably? At least for one of the test arms.
Categorical variables in the datasets of large tech companies can take a lot of different values.