Abolish The Reference Check | Hacker News Reader

Abolish The Reference Check

nirandfar.com

39 pointsnireyal43 comments

There are many, many bogus things about this post.

First of all, I'm ready to stab myself in the eye every time I have to see someone they only hire "A" players. This is getting tiresome and unbelievable - I don't know of any startup that has nothing but "A-player type work" to do. Almost every plausible startup has some work that will be comfortably addressed by a range of abilities and the idea that you're going to stuff your startup full of absolute super-stars (and do well as a result) is just posturing.

I am also not entirely sure that someone whose references are, I dunno, busy doing actual work or "on holiday" deserves to be penalized with respect to someone whose references are all hanging around in their office with nothing to do but promote the living shit out of each other in some kind of mutual admiration society.

Speaking from experience, I have seen many such cliques of utterly mediocre, self-serving people who would unhesitatingly tell happy stories of how wonderful everyone in their clique was to anyone who asked - even if a more conventional response from further up the management chain would have given a very tepid review of the actual performance of said people.

Finally, if you're going to contact a bunch of people out of the blue, adding self-promoting line about your blog is not appropriate. Under the signature line, maybe, but putting it the body text is a dick move.

>mutual admiration society

That's very true. I browse Linkedin sometimes, and I'm often surprised to find out x# of people have recommended a terrible ex-coworker.

Yes. Thank you. I'm glad I'm not the only one who wants to retch when I read posts like that. The underlying ego and assumptions are laughable, at best.

Beyond that, however, the poster needs to learn about ROC curves (http://en.wikipedia.org/wiki/Receiver_operating_characterist...) – for just a moment, let's assume we're genuinely only hiring "A" players – then the challenge is a signal detection problem, pure and simple – sorting the signal from the noise and avoiding both false positives and false negatives.

While I have huge issues with the post's method, it's clearly only focused on avoiding false-positives – avoiding hiring someone who's bad but is sold as good/great. Unfortunately, there are real issues with false negatives as well – you may miss out on a fantastic candidate, first and foremost...

But, of course, decisions aren't that simple. Anyone who understands decision making knows that decisions depend highly on context – including the context provided by the other options... and if you systematically fail to account for eliminating some number of options, you may well be (foolishly) limiting the range of options and the context of your decision. In short, "A players" are relative...

And that idea goes even further. Many stellar employees, innovators, coders, etc. don't start out that way. Sure, some do, but those aren't the people applying and giving you references.

Your job, when hiring from a resume pool, is precisely to find those diamonds in the rough – those false negatives.

This is particularly interesting because it is common to hear complaints that (A) it is hard to find "good" people, and (B) it is hard to go through huge quantities of resumes. If those are both true then clearly it's time to worry about false negatives. Maybe failing to hire isn't such a big deal? Maybe there's an element of bravado in claiming that nobody's good enough? Or is a lot of tech hiring just horribly misguided? I'm not sure.

Personally I've got to say that I also find the "rock star" rhetoric to be a huge turn-off. When I see it in a job ad I think of how horrible it would be to have to work with a club of self-described "A" players.

The gruelling interviews are another perverse system that I think has a negative effect. If you can easily get lots of interviews are you going to bother with the five stage interview with four hours of quiz questions and "homework"?

Well – the one thing I'll say is that interview processes like that screen out people who are both good and know what they're worth – which is one of the subtle subtexts in much of the hiring processes being discussed/described here – everyone wants the cheap rockstar – not the rockstar that expects rockstar prices and rockstar access... or well, few people do and those people don't use these sorts of hiring mechanisms.

Agree. Hiring all A-players is something people say who do very little hiring (or have ridiculous amounts of money, though I've never had the luxury myself). Hiring all A-players means your paying way too much for your staff. A great manager with a great process will be able to get results from a team of all skill levels.

There are indeed many bogus things, but I do like the idea of an opt-in vs opt-out email.

Reference checks, as they exist today, are useless. Doing them as an opt-in email at least gives you a small chance of getting some signal. No, you can't use it to sort them into your A-B-C rock star instant hire buckets.

But it's a signal. Signals are good.

I thought the line promoting his blog was amusing - he put it on an example letter for the reader to use (it says "sincerely (you)" at the bottom) as a joke. I don't think it was a actual e-mail that he would send out.

Eric Paley had a great post titled "The Curve of Talent" in which he said "Those who suggest that startups should only hire A players are grade inflators": http://epaley.posterous.com/the-curve-of-talent

A long time ago, I was a VC/growth equity investor. Reference checks were incredibly important, as a startup's chances of success, are, on average, very much correlated with the strength of the team in a variety of areas.

First, obviously, we were looking for red flags. We had ways of phrasing questions designed to get people talking (esp about negatives). Also, more importantly, we were looking for names/contact info of other people who worked with the person we were checking that weren't on the reference list they provided to us. We'd do the same thing with customer references.

Second, we didn't know most of the people provided as references. A random stranger's opinion is interesting, but not that useful, especially when phrased in terms of "good/bad" or "exceptional/not exceptional." What was more valuable was to find out what kind of person we were dealing with, which is something that you really can learn from a few random strangers. For example, in past lives, has the CEO been an effective salesperson? Does s/he need a strong COO because s/he tends to let details drop here and there? The line of questioning went something like: "If you were going to build a team around this person, who else would you want on it? What qualities would the other team members have?"

In other words, references are just another data point, and I believe their value is not a binary determination of that person's ninja-rockstar-astronaut-whatever-ness, but a more nuanced read on that person's abilities as they contribute to the team overall.

Is it just me, or does the person who wrote this sound like a snob?

* "In a startup, you can’t afford to hire B-players."

* "...a technique I’ve come to call it the “average-need-not-apply” method."

* "You saved yourself from hiring a B-player, or worse."

* "...employers can make better hiring decisions and make the hiring process better for everyone involved – except maybe the B-players."

"You saved yourself from hiring a B-player, or worse."

As somebody that's probably not up to "A-player" status for this guy, I'd consider myself lucky if this method saved me from getting hired by anybody using it.

This is the language of "TopGrading", a hiring methodology.

Oh dear. Is this a thing now? Thanks for the heads up...looks like I have another set of snake-oil buzzwords to watch out for.

Oh, good, a methodology! That should help with their lack of actual brains and heart.

Don't forget the Stanford name drop.

I'm not really fond of references for a couple of primary reasons:

1) IMO, they are relatively unreliable indicators for technical candidates

2) They leak the candidates active job seeking status to potentially untrusted colleagues

1 occurs primarily because references are self-selected to be people that the candidate has friendships with rather than people who will give the the most honest assessment. In large companies, this can often be especially easy for a candidate to game. The technique in the linked article will do nothing to resolve this, IMO.

And 2 is made worse by this person's desire to interview people who aren't even listed as a reference by the candidate. I've worked for companies before where I would have had to turn down a restriction like this.

The obsession with finding super-employees is the flip side of the obsession with finding the perfect process.

Does pretending you hire only exceptional employees help you believe that you're exceptional? Or does running an exceptional workplace help you hire exceptional employees? Would an exceptional employee REALLY be able to make a difference where you work? Or would they just ground down by an ordinary organization?

My definition of "A" people is "can do the job without being taught how to do the job." This is a necessity at a start-up where there is no one else to teach you how to perform your role, because your role is new.

"B" people are fine for large organizations where there is someone to show you the ropes, but smaller orgs need people making fast and correct decisions without organizational tradition/experience to support them.

This technique exposes you to a new pool of mediocrity. Sure, you're guaranteed exceptional candidates, but what how are they exceptional?

Did they, in the past, work for exceptionally abusive employers who need them back? (The new guy quit after his fourth month of unpaid overtime!)

Did they work for exceptionally dimwitted individuals who can't measure their new workers? (The new guy doesn't understand the old system! Fred said it was SO EASY!)

Did they do exceptionally niche work, where almost no other candidate will do ? (e.g. managing physical linotype inventory -- no one ever got our 'e' and 't' counts in the same place as Fred! His database management was magic!)

Using former bosses as an uncalibrated barometer isn't very much better than phoning them up and trying to draw inferences from their tone of voice. Garbage in, garbage out. (But I bet it saves time on collecting your crap data.)

My answer, not really, but what that part of me that flies off the handle feels like answering:

Dear interviewer,

This is not a reference. I don't respond to douchebags with manipulative reference requests.

Thanks for letting me know that Jane Doe is in the job market. In the case that Jane is indeed exceptional (and she might be for all you know, or you wouldn't be annoying me at my place of work), I have done the following:

I have just written to all my friends in a position to hire developers to let them know there is an exceptional candidate doing the rounds, and they should call her for an interview asap.

In fact, I may have done this even if I don't think Jane is exceptional. By definition, only a few people can be exceptional. Some developers are just solid and dependably conscientious, creative and productive, and I also hire and recommend those whenever I can, because in doing that I help two people: the developer and the employer.

Which is also why I would never give you references for good people. The industry needs good people, and good people need good places to work. Which isn't your shop. Good developers deserve better than your manipulative sports-metaphor elitism.

Regards, etc.

[edit: took out an unprofessional bit of invective at the end]

This method falls victim to self-selection bias. Consider this scenario: Tom is an mediocre developer but he has a great personality. People who don't like him or his work won't answer. But his friends will.

Ah the good old stalking technique. I remember the first time a potential employer tracked down and telephoned my mother.

Sure that would have been OK if this was an FBI background check. But it wasn't. So the answer to the job offer was "Sorry friend, but go to hell."

I think most employees (and perhaps many managers) fail to understand the purpose of a reference check. It is _not_ to find out whether the employee will be any good - that can only be determined by observing the employee in your new workplace for some probationary period.

The hiring decision is traditionally made on the strength of a hiring-manager's conviction and only the hiring manager - the interview pool can inform, but never make a decision. I've seen many situations in which the entire interview group said no to an individual who was hired, and became a star player. I've also seen scenarios where everyone said yes, and the employee was terminated for non performance within a year.

The interview is just a numbskull filter - you usually can't identify great people, but usually (particularly for technical positions) - you can eliminate the people who don't know anything.

So, what then, is the purpose of the "Reference Check" - it's to determine if this employee you are hiring burned down, assaulted, or otherwise was convicted of some horrible crime at a previous company. In general, with larger corporations, all you'll ever get out of them was "He was an employee from XXX to YYY in the position of ZZZ" - but, if he was terminated for killing a coworker, you might discover he was terminated for cause. (unclear whether they'll tell you _what_ cause, but do you really need to know?)

I've done over a 100 reference checks for employees - and, over time, I've realized that's all I'll ever get. And, when an employee does not give you _any_ prior bosses (particularly if they've had a lot of them) - that's a warning sign as well, that you are allowed to discriminate on in at least California.

Am I the only person here that is offended by "Don't let the candidate return the call" or was that meant metaphorically?

I might not be "A-level" yet but as a potential candidate I would be furious if my employer kept my options away from me, especially if he's likely to toss me away if everything isn't going perfect for him, as the language of this post seems to dictate as a very real possibility.

I assumed that meant "Give a good enough offer that the candidate doesn't bother returning the other call", as a truly exceptional candidate is likely to say no if you press them to decide immediately.

This seems like a good strategy but I would only try it if you had a few different good references to check. The danger that the email gets ignored because the person let it slip through the cracks is fairly substantial.

Maybe it could be improved to be super short and only ask for a simple "yes" answer so they can answer it right away instead of flagging and moving back to it.

I wouldn't respond. I would just find it douchey.

You should already be convinced that the person is exceptional (in talent) or at least good enough to hire before you start messing around with references. The purpose of a reference check is to make sure the person isn't either (a) lying about his background, or (b) seriously harmful. For (b), I don't think references are the solution but I can't think of anything better. Unfortunately, there's no character test that can be done in a few hours of interaction.

Not only would I not respond, I'd let my (former) colleague know what kind of potential employer (s)he was considering.

I don't know that I've encountered such an ill-conceived idea in quite awhile. Forget all the false negatives this method would be sure to "detect", I wouldn't feel comfortable spamming the candidates contact list.

Yeah, all this nonsense about "we don't hire B-players" is ridiculous. Narcissists always think their loyalists are the best people in the world and everyone else is an idiot. Classic sign.

If we're going to ignore the douchebag baggage ("douchebaggage"? 'tis a new word) associated with "A-player" and "B-player", I'd say I really want the person who turns B-players into A-players.

I'd probably ignore the mail specifically because of the marketing effort for the blog at the bottom (and would have already been a bit hackle-raised at the attempt to define exceptional as top-10%. One, that's wrong. Two, it's patronizing.)

There is something reasonable related to this, though. Let people recommend with different weightings. Say, a way to say "no, this person is a bozo" without directly saying it.

It's like you could come up with a simple web UI to do this. Assign -1 to "this person is a bozo, avoid", 0 to "I am only doing this out of social obligation", 1 to "weak rec", 2 to "good, competent", 3 to "indeed, far better than average", and 4 to "you are an utter moron if you don't hit this."

Maybe add a comment field at the bottom.

The key is to give the recommender a social out -- they can tell the candidate "yes, I recommended you", even if it was a 0 or 1 (or, in some cases, a -1).

Facebook and Google do "recommend" (no-op) vs. "strongly recommend" (actually significant), but adding more levels is good.

I've done some hiring of math tutors for a small tutoring company I used to run and I was extremely diligent about reference checking. It made a big difference. I never got a bad reference, but I got some where there were pauses on the other end of the line that made me curious. I was naive then and hired the people anyway. In one case it was worth it, but in both cases, it became clear early on what might give pause. Other than that it helped because a lot of people with cheerful demeanor and astonishing resumes simply disappeared when it became clear that I really did make calls and verify every degree they listed. That happened with a lot of people.

I'm not sure this is the solution. The problem is hiring (or accepting for YC, College, etc) in general is a complicated process where the generally accepted solution doesn't work as well as we want it to.

The only real solution would be to get to know the people better, which takes too much time. We resort to things like the reference check because it's the only option that doesn't require massive amounts of time. Unfortunately there is no substitute for the time it takes to get to know someone.

Companies complain and complain about the hiring process, with people like the OP even resorting to subversion in an attempt to "de-risk" the process.

"The only real solution would be to get to know the people better..."

I totally agree. Read the resume/CV, run the technical test (if required), conduct the interview. If they seem good enough, hire them.

All the jobs I've ever had (perm and contract), had a probationary period during which I could be fired at will.

If a company cannot handle the financial strain of hiring and subsequently firing someone within the probationary period then perhaps they shouldn't be hiring someone in the first place?!

Hire base on employee referrals have always worked the best.

This deserves a downvote.

Horrible idea. Most companies have their policies of not giving references either way. Lawsuits over good references (for people who turned out not to be good) have occurred, and even though the probability of a plaintiff victory is low, who wants to deal with that pain in the ass? Most people are fine saying that someone was essentially decent and worked hard, but aren't going to go out on a limb to say that someone who no longer works for them is "EXCEPTIONAL".

Also, what if the person doesn't remember his boss's email or gets it wrong? What if his boss is three companies away by now? Lots of false negatives here.

I think reference checks are not essential for junior hires-- not having references is a warning sign, but I'd take a chance and decide fast on someone if necessary if he wasn't going to be in a managerial role-- but I'd require references from past subordinates (2 per company) from anyone being hired into a managerial role. I actually think subordinate and peer references are far more useful than managerial ones: if you want to see someone's true character, ask people he had power over.

Or perhaps it can be a horrible idea, but deserve an upvote, because it leads to interesting and useful discussion.

It was written by some dude who "sold two companies". Quite out of touch with the working man who needs references. Safe to ignore in that case.

Oh, come on! If you're just going to be an employee, possibly for ten years making money for other people, instead of taking some initiative and having your dad set up a lunch between you and the VCs he prepped with and having them teach you how to work for yourself and join Those Who Have Completed An Exit, well then... then you deserve the mediocrity you get!

Congratulations! Looks like you found a star

Uh, why are you looking for a star? Why does everything think they need or even want a star? How about just someone who gets things done?

As for reference checks, they're a great way to assure that a past, toxic work environment will continue to haunt a candidate indefinitely.

This article is just more of the "I eliminate X out of Y candidates, so I know I'm hiring the best!" B.S.

Reply on news.ycombinator.com