Hacker News Reader

Comment by marmaduke | Hacker News Reader

Have "actual machine learning" experience, no idea what "high leverage" means.

I'm referring to leverage in statistics [1], and particularly how high leverage observations may affect your model and how people deal with them. Clearer?

For context: one of the candidates I interviewed told me that the best way to pick a model was to pick the model that would have the highest R squared when fitted to the whole dataset. I asked him about overfitting and he didn't know what I was talking about. Same guy whose CV showed 4 years in a research lab doing stats [2]. A lot of people are just going through the motions.

[1] https://en.wikipedia.org/wiki/Leverage_(statistics)

[2] reposting for fun: http://4.bp.blogspot.com/-IJo_Tkw95-o/VANnTdCFgHI/AAAAAAAADV...

Yeah... Have actual machine learning experience as well, never heard the term leverage as applied here. Sometimes the nomenclature can confuse people, and I guess the original point about more in depth interviewing processes was related to discerning whether the fundamental knowledge and team fit are there, not whether the candidate is aware of the particular subset of CS trivia the interviewer is currently interested in. I say this while admiting that I've made this mistake in the past while interviewing candidates for CS positions...

What do you call high leverage observations then?

This is similar to "multicollinearity". I have two graduate degrees in applied statistics and machine learning, and did math as an undergrad, but I did not ever even hear the term "multicollinearity" until I got my first job after grad school in a place that used a lot of regression models.

I had studied "spurious correlation" in some machine learning courses and in some research, but almost all of the methods I ever studied or worked on were meant to use data-driven methods to account for spurious correlation, or dimensionality reduction to find subspaces in which the data's natural structure was best preserved without correlation-caused redundancy. Literally none of them ever even mentioned the phrase "multicollinearity" -- which seems to be more popular to people coming from classical statistics or econometrics programs.

If I had been asked to describe "what is multicollinearity" during the interviews, I would have been rejected, yet after they told it to me, I learned about it in about 2 hours on Wikipedia, and within two months I had actually done a research project for them where I showed them how you could use ISOMAP or randomized PCA to effectively handle regression multicollinearity better than their ad hoc covariate-averaging techniques.

This kind of trivia stuff is useless for hiring. All it does is let the interviewer feel smart and tout their favorite particular buzzwords to see if the candidate is in the same "club" as them.

I can imagine some diehard frequentist caring a shit ton about "consistent estimator" or various special tests like Kolmogorov-Smirnoff or F-test or likelihood ratio test. If you say, "I always do Bayesian stats... those test thingies are stupid junk" then it means "This person is not in the Frequentist Club, let's reject their lame ass and tell them it's cause they are 'just not a good fit.'"

Same would be true in reverse if some Bayesian diehard interviews a frequentist person.

Same for someone who sees deep learning as the hammer for every nail. Etc. etc.

People don't want to admit they treat these things kind of like baseball trading cards, and it's more about your cool-kid status than your actual tech skill.

However, if you ping me on my profile email after I put up a job on Who's Hiring, and I send you back an email saying "hey, can you answer these few filter questions, take your time" and one of them is "what is multicollinearity", chances are you'll google it and tell me what you typed above and it will be fine. It's not like you can't use Google or take your time.

What I'm dealing with is people who are dumping their CV and a standard cover letter on me at high frequency and I do not want to put an HR filter between them and me, because I want to spot the guy without a LinkedIn profile (HR red flag), who dropped out of college (so no brand - HR red flag) but has plenty of useful experience (so no buzzwords, because real experience looks unremarkable on a CV - HR red flag). I know no more efficient way. Would love to hear of any.

I did make the mistake many years ago of rejecting a PhD in stats because she admitted not knowing about neural nets ("but my lab buddy is doing some research in them"). I'm still kicking myself, but have learnt a bit about hiring since then.

By way of example, I do not have a LinkedIn profile, and I left a Ph.D. program with a master's only. Though my degrees are from pestigious schools, I don't think school status should count for as much as it does.

I am lucky that the prestige of my schools is high, that you can easily Google some of the actual research work I did, and that I am very highly ranked on Stack Overflow.

If I didn't have those, I think more employers would simply reject me for not having a LinkedIn account, because employers use the dumbest shit (like LinkedIn) as hiring cues.

Ah we usually call them outliers and use robust techniques or not-L2 stats when they are pervasive in the data. Still, in my experience, handling outliers is an application dependent thing, as they may be benign or they may suggest another stat model entirely.

Otoh, I agree not being aware of over fitting is definitely grounds for setting aside a candidate for ML job.

There's a difference. Imagine a simple one dimensional linear regression.

A value with x close to the mean, but extremely high y would barely move the line as a whole up or down as you vary y. The gradient would not be much affected.

A value with extremely high x does not need to move much to change the gradient of the line. You might even change the sign of the gradient.

Both are outliers but the second has higher leverage than the first.

Edit: illustration: http://imgur.com/vVKwjHh (right hand side are high leverage)

Do you mind if I ask you where you work? I find most people practising "machine learning" have barely gotten past kmeans, naive bayes, and SVMs with very little actual understanding on how or why they work and where they fall apart.

I run my own data consulting firm based in Asia.

Originally, we were going to build high dimensional statistical learning systems crunching large datasets on GPUs in Haskell.

90% of clients cannot feed me the data, so I end up building them a data warehouse first; in some cases we even redo their data model. 100% of clients have datasets too small to justify getting out of R and single CPUs, so our Haskell ML libraries are, so far, on the wishlist only (I think Tweag has had more success there, but I still have feeble hope that one day...) Hence the RM focus as well as ML. The ML side is sort of picking up this month.

That being said, I'm not actually that well versed in ML. Just the guy running the company. I've built (and read) just enough to know how it works, how it applies to clients and who to hire to get the job done properly...

I think what maybe tripping people here is your use of the word machine learning. Personally, as you're using it is how it should actually be used: applied statistics. However, a lot of people think of machine learning as a collection of algorithms they can use to make predictions about some given dataset. Use sklearn to model.fit(xtrain, ytrain) and model.train(xtest, ytest). I personally blame Coursera and the online courses for this trend.

Do you ever have the impression that the model you are going to apply influences how you clean the data? What happens then when you redo the model?

Interesting, I guess Leverage is one of those terms that one might not encounter frequently except in some specific situations

I don't claim to be an expert but at least I know what Moment is

There are an infinite number of such concepts in "machine learning." Just because you liked this one doesn't make it a good interview question. If anything, you are failing at theory of mind - as evidenced here, most people aren't making lots of use of the concept. When interviewing, don't be a dick. Test people for general ability and willingness to learn, not stupid questions you got out of a random textbook.

Everyone learns different vocab depending on what discipline you learned about stats/ml. Leverage is a very standard stats term, but I've never seen it used in an ml context.

Reply on news.ycombinator.com