Is there a balance to be struck between simple hierarchical models and

I think they have totally different goals typically. For example, let’s say we are doing a sampling procedure. How do you estimate the sampling error? I’m not aware of a machine learning technique that will help, but you can use Bayesian and MCMC techniques

I think this is accurate but mostly because statistical modelling aims for interpretable parameters. That very strongly regularises complexity.

Yes and you can even build symbolic engines that do this for you. I think the real question we must ask ourselves as data scientists or statisticians or whatever is whether we believe these data models represent the space of data fully or by happenstance. And if by happenstance is it because the data doesn’t capture the underlying processes that produced the data or are they uncapturable in this way and function approximators like neural networks or gradient booster machines are better. And is that because those function approximators capture interactions between the driving processes that otherwise go unseen or is it because those processes have fractional dimensions that control their impact that are not captured by data models. This all is summed up well by Leo Breimans two cultures paper in my opinion. I have gone back and forth on which “culture” is the correct representation of how processes produce data. If you buy that only function approximators truly capture the complexity of whatever processes you are observing then you have to wonder why physics works so well. That’s because, at least in my opinion, from the statistical point of view physics has spent centuries developing equations that are linear combinations of variables that are essentially data models according to Leo. I hope this opinion generates discussion because I don’t know what the answer is or if it matters that there is one.