An Introduction to Hierarchical Modeling | Hacker News Reader

An Introduction to Hierarchical Modeling

mfviz.com

112 pointsandyxor17 comments

Though the visualization is cool here, I think this is actually not a good example. The final "hierarchical" model presented doesn't say anything about a relation between the coefficients for the different departments, so one can view this as really just having separate models, one for each department.

To me, the point of a hierarchical model is that one can assume relationships between these parameters (e.g. suppose that they are from the same distribution). Then when one department has very few data points, some of the information in your beliefs about the parameters for that department come from what you learned in other departments. E.g. the base salary is likely to be similar to base salaries of other departments.

Agree. The example is not a Class hierarchy - it is a group with different coefficients. But the concept is useful, and the related visualization (for Class comparisons) would be useful.

I love the visualization and animation, but I think it should be specified that this is Linear Hierarchical Modelling.

I was expecting something bayesian-related given that bayesian statistics lends itself to hierarchical models quite nicely.

I don't quite understand what's hierarchical about this example. seems like a few different mutually exclusive segments that can then be independently analysed?

It's hierarchical because of the nature of the data, not because of the model itself.

What if you simply add department and department*salary interaction terms to a standard regression model?

Then for any given data point, salary for all other departments will be zero (dep1salery = dep2salary = ... = 0) so data for each department will become separate and you will in effect have i number of different models, although they all share the same bias term / intersect.

You could argue that a hierarchical model is already multiple models. If each department has a dummy variable then they wouldn't share the same intersect. The only difference is how you interpret that intercept. The department intercept would now not be the salary of an individual with 0 experience, but the difference between the salary of an individual with 0 experience in a department compared to the all other employees with 0 experience.

Well, two things here:

You wouldn't actually add salary as an interaction, as it is salary you are trying to predict. You'd use an interaction between experience and department to predict salary, as you correctly point out.

But usually when doing interactions, you don't apply it to the bias term, so you will get this:

    salary = b0 + b1*d1*exp + b2*d2*exp ...

And this is essentially a model for each department, but all models share the bias term, b0.

If you also interact the bias term with the dummy variable for department, you will in effect have completely independent models,

   salary = b0,1*d1 + b1*d1*exp + b0,2*d2 * b2*d2*exp ...

But I don't think that is what is normally done when applying interactions between features.

Anyway, I think we both agree, once we get our terminology aligned.

You'll run out of data pretty quickly.

https://statmodeling.stat.columbia.edu/2018/03/15/need-16-ti...

Hierarchial model has the same problem, since it splits up the data into categories.

I think it gives you the same answer without having the intuitive picture of your kernel that people like for toy problems.

How might this be combined with a training/testing data splitting paradigm, or even using an ML model?

Probabilistic machine learning is still machine learning. If you're referring to one of the popular tensor frameworks, pretty much all of them have probabilistic features or add-ons [1][2][3]. There are also frameworks developed specifically for this kind of work [4][5].

This [6] is a more in depth look at hierarchical / multi-level modeling. The prediction section [7] specifically goes over cross validation and inference.

[1] https://www.tensorflow.org/probability/examples/Multilevel_M... [2] https://pyro.ai/examples/forecasting_iii.html [3] http://edwardlib.org/ [4] https://mc-stan.org/ [5] https://docs.pymc.io/ [6] https://docs.pymc.io/notebooks/multilevel_modeling.html [7] https://docs.pymc.io/notebooks/multilevel_modeling.html#Pred...

ML are algos not models (estimators to be precise). This model could be fit by many of them. However, fitting the model directly with the linear model will be advantageous for computational reasons.

Is this a narrow special case of an SVM?

Why not use a SVM?

Very cool animation.

Reply on news.ycombinator.com