The Hiring Post
sockpuppet.org
sockpuppet.org
I can talk in a pretty good amount of detail about how exactly our process worked, if anyone has any questions.
And, to head off a concern a reviewer gave me: from 1997-2005, I was a full-time software developer; I shipped shrink-wrap boxed software on Windows and Unix in the 1990s, then appliances deployed at tier 1 ISPs. I'm a "software person" more than a "security person".
For example, a good portion of doctors absolutely hated using checklists. Yet, when pressed, readily admitted that it prevents simple mistakes and that they would prefer to have them rather than not to. Another is that entries that address more human concerns, e.g. "Have everyone introduce themselves", have a place on good checklists.
[0] http://www.amazon.com/Checklist-Manifesto-How-Things-Right/d...
At a high level, this is all about Deming ( http://en.wikipedia.org/wiki/W._Edwards_Deming ) and TQM concepts -- if you want to achieve a high-quality output, measure the things that matter, and understand the variation present in the system. Once you have a stable system with good data achieved by good methods, you may then begin improving it. Attempting to improve a complex system without knowledge results in unpredictable changes—we call that tampering. Simple but beautiful.
So, in essence, this is an extremely natural and correct application of quality management principles to the hiring process. Stellar.
discussion: https://news.ycombinator.com/item?id=8830903
In your post you say, "At my last firm, we had the “first-call” system. Every serious applicant got, on average, 30-45 minutes of director-level time on the phone before any screening began." You seem to continue on from that point. What happens before first-call?
What surprised me about recruiting was how much more important qualification is than outreach. Without good qualification, it almost doesn't matter how good your outreach is, because you're filtering for the same highly visible easily accessible candidates every other firm is looking for. The best-priced talent is buried. What you need is ground-penetrating radar. Clever outreach schemes helped us, but what really let us make our best hires was a qualification process that allowed us to confidently ignore resumes.
We didn't just maintain our team's level of quality by ignoring resumes. We transcended it. We never could have run a resume-based process that would hire people who could use lattice math to break crypto. To hire those people, we needed to select for aptitude, not experience.
I haven't said it anywhere yet, but in case the subtext isn't clear: the person who can implement an attack for which edit/compile/debug takes 4-6 hours also does a pretty great job on every other facet of the software security job. There are aptitudes that are very good proxies for other aptitudes!
Maybe my premise of 100 candidates is incorrect based on how you did outreach?
How do we figure out which ones are serious? We talk to them, usually give them homework, and then give them work-sample tests. This is not particularly expensive for us, and gives us repeatable, relevant, apples-to-apples metrics on how qualified they are to do the work. This is, as Tom insists, surprisingly easy for us. (Not to say it's easy, but nothing about hiring is easy.)
Two questions for you.
What is your experience hiring fresh college grads? Is the process different? Since one of goals of the process seems to not arbitrarily adjust it for every candidate, but rather stay consistent, do you think it makes sense to adjust it for someone who just graduated. They might have been learning computer science type stuff, data structures, did a few group projects and have certainly been practicing the typical interview puzzles, fizbuzz and O(n) times of the all data structures.
How do you account (or do you account at all) for culture fit. That is a fuzzy and slightly dangerous area. Is there a score in that regard? There are some people who are brilliant but they do not work well in a team, or rather would not well in one particular team. This is course full of danger as "culture" fit often becomes an opaque discriminator for racial, sexual and other stereotypes. "He's just not a good culture fit" is such an overload euphemisms that I feel kind of sleazy just saying.
The lede I buried in this post is that we hired more-or-less resume-blind. I had final call on every hire and went through some effort not to look at resumes. On our first-calls, I'd ask some questions that would give me background hints about candidates, but that was primarily to tune my spiel (I didn't want to explain blind SQL injection to someone who'd spent 3 years as a pentester already, for instance).
If I had it to do over again at Matasano, I'd have done it formally resume-blind. Only Dina would have had access to resumes, and interviewers would be forbidden to ask resume questions. The process would have worked better if resumes were firewalled out completely.
There was no culture fit score; in fact, if you gave subjective feedback ("passion", "confidence", "work ethic") your feedback was likely to be discounted. We were proud of the culture diversity we managed to scrounge out of our candidate pool; to have team members with kids who needed to keep strict hours, and team members who'd roll in at 11 and still be there at 8 working on a pet problem. We had drinkers and non drinkers, college students and people with 15+ years of dev experience.
You could get booted from the process for being overtly an asshole (I, for instance, might not get hired), but that happened very rarely and with crystalline clarity.
Don't get me wrong, I understand and agree with your reasoning. I'm just curious how transparent it is to the other side.
So really "what did you study in college?" needs to come off the list of things you ask (at least as part of the filter).
on the reverse of that, i have sometimes avoided topics that the candidate was familiar with because i didnt see relevant experience on a resume. that was a major miss, and ive tried to avoid using resumes for shaping the interview topics since.. i see the resume now as more of a signal of what the person is interested in (since most of us carve huge swaths of our experiences out, and tailor them for the job we apply to anyways).
Good job coming up with something different. I hope it's really working out as well as you say.
Hmm, I once took it upon myself to do exactly that (mostly using non-blocking sockets, skipping hostname lookups, etc.), but what's so "tricky" about this question? Assuming the position involved network programming, I'd consider it a good way to test someone's networking knowledge. Is the "tricky" part knowing how routers often drop ICMP replies if they exceed some threshold rate?
That's nowhere near as bad as the interviewer who has the primary goal of demonstrating how smart he is (to the person he is interviewing), with the actual hiring process being secondary.
So, I'm not a fan of giving interviews. I wouldn't mind shadowing someone better than I, but my first inclination is that that would be a terrible idea since it would make candidates even more nervous.
This may not be an issue for you yet, but what about plagiarism? And can you expand on your hints about how you changed your pipeline at the front of the process? You don't interview every resume that comes your way, right?
I did a lot of calls where I knew a couple minutes in that we were unlikely to hire the person, but (a) I got surprised by outcomes enough not to shirk on those calls, and (b) those calls are a very small price to pay for finding buried talent.
1) People generally don't cheat. If you have a high percentage of cheaters its how you are filling your pipeline you need to address, not the filter.
2) What most people think of as "plagiarism" is actually very common in the real work of software developers. Very frequently you see/mimic other peoples work to solve problems. Instead of being freaked out about a skill that is central to the job, why not use it to evaluate the candidate. Did they "plagiarize" the right thing? Did they do it effectively. Did they do something backwards that a simple google search would have found a thing to copy?
3) If you are big enough for plagiarism to be a real problem and have addressed points 1 & 2, it is relatively easy to detect mechanically (and there is a surprising amount of research in the field as CS professors invariably write both an automated grader and then an automated cheater detector).
I was wondering if you could share some info on the salaries. Did you offer every candidate the same salary or adjust it (based on resume/previous salary/age/experience/...)? You mentioned 100% retention - so I assume no candidate ever left for another position with a higher salary?
Being a specialised consulting firm, I imagine Matasano could offer highly diverse, very interesting work long-term (because consulting), and higher-than-average salaries (because specialised), so maybe it wasn't a problem for you to offer higher salaries than the competition, or to retain candidates at equivalent salaries.
Great article. The prelude to the interview with books given to candidates is especially interesting. I'd love to see that reading list, and would further like to suggest posting it on your careers page. I would imagine that anyone who goes to the trouble of working through your suggested materials would be providing a very strong signal of quality and investment.
Also, what percentage of people who complete the work-sample task get an onsite interview?
I'm asking these questions because I think most software companies who filter out people based on resumes and phone screens do it mainly because there are too many applicants and they need to filter out many candidates at earlier stages. Do you think your hiring strategy resulted in more time spent on the process per successful candidate and per applicant?
As someone who already has a job, committing a large number of hours for the chance of an interview seems like a waste of time.
If every employer demands a 5-10 hour work sample test before they even talk to you, that really isn't a scalable solution from the viewpoint of a candidate. I could easily put in 100 hours into works-sample tests without getting a single decent interview, so now I just refuse them.
Everyone else is screening people based on resumes? Then, you can find good candidates by ignoring resumes, so you find good people with bad experience.
Nobody else hires people older than 40? Then you target people who are older than 40 but are still good workers.
Everyone wants a programmer with X experience but there's a shortage of people with X experience? Then hire good programmers without X experience, and give them a chance to learn X on the job.
Nobody else gives a work-sample test? Then giving a work-sample test is a benefit. You find candidates who are desperate and good enough to do the work-sample test. If everyone gives a work-sample test, then you get an advantage by NOT doing a work-sample test. (For example, a company with a work-sample test is now excluding me from their candidate pool, due the the large number of bad experiences I've had with them.)
One idea that seems workable is the paid work-sample test. But then you have to do some screening before giving people the test. (I.e., hire someone for a 10-20 hour mini-project before committing to hiring them full-time.)
With 100 hours of free time, I could get a personal project done and put it on the Internet somewhere. That seems like a much better use of my time than doing 10-20 work-sample tests for companies that aren't going to give me an interview even after I do it.
I will second the value of a work sample. It's so simple that it feels incredible that we didn't do them in the past. What my team __hasn't_ is issue the exact same work-sample problem; you make a compelling argument for doing that, so I think that we'll implement it.
While I agreed with many of your points, I could not stop thinking what a huge time burden of implementing something like what you propose, at scale. For a growing company with several work streams and projects hungry for talent, the interview approach you posit would never work.
Another thing that came to mind is the fact that educators and cognitive scientists have been working for decades in what constitutes a good and effective test. Here you seem to claim that somehow you and your folks have "cracked" the system and have come up with a test process that's guaranteed to yield good long-term and performant workers?
Finally, I feel this approach while intended to alleviate some of the most common pain points of the interview process, it tries to reduce it to a number. While it would seem a number is plain and objective, much like a test score, it neglects to "tell the story" and unless all you want are gifted and highly skilled human automatons, you need a way to gauge "soft skills", which one would argue are as important as their core technical competency.
[1] http://mhutter.org/papers/Mulder2014UsingBleichenbachersSolu...
Not sure if there is any difference in content.
Please please please apply for full-time when you graduate! The process is a lot smoother.
Couple questions for you. How do you go about choosing which metrics go into a grade when designing the work sample test? Should the process of grading it be completely automated, so as to eliminate bias? What's the ideal length of time for a work sample test, in your experience?
In other words - YMMV, what works for others may misfire miserably for you.
Right now at my job, we don't do a good enough job of this. I get candidates that walk through the door that can't code more than "Hello World!" without screwing up. I'm not happy about that.
This post, I think, enumerates a large number of things that you can do to determine worthiness before they even walk through the door -- the remote test is only a small part of it. If all you're doing is a remote test, then maybe you need to do more, yes?
There are the hired.com models where independent experts thoroughly vet the candidate, but that involves a lot of manual interaction.
What I really want as a hiring manager is for someone to show up and say "Here's Candidate A. We think they're strong because X, Y, Z, and here's the proof." Someone whose mission is to honestly find me the best candidate, who isn't just trying to place someone as fast as possible so they get their commission. Most recruiters I know are interested in placing a good candidate as quickly as possible, but there are a lot of problems with this. "Good" isn't "best", and often times the recruiters don't know the difference between Java and Javascript, and I wind up with junk resumes in my inbox.
So how do we fix this? I have no idea. But there's gotta be a way. I don't think that way is going to be scalable with a simple web application. I think you have to have a secret weapon that nobody else has, and maybe that's just a group of in-industry professionals that vet candidates on your behalf, or maybe it's some incredible machine learning algorithm, or maybe it's something else entirely.
As a job seeker, I want to get the best job I can, and as a boss, I want the best employees I can get. And I don't think those things are mutually exclusive at all.
Hired.com will turn away people they don't think of as good candidates, but usually after people manually review their profile.
I wonder if The Secretary Problem could be used to calculate data points for whether a candidate is good or not. http://en.wikipedia.org/wiki/Secretary_problem
I worked at a company that had a remote, 3 day test. It was pretty fun, and I still sometimes try out things on it to see how they work (it was a two-parter, with part 1 being to write an evaluation function, and part 2 solve a NP-hard search problem).
I strongly feel that the information I gained from looking at the code candidates wrote for that problem told me lots of really good info.
As far as I know, nobody cheated. (All submitted code as unique, and everybody could talk cogently about what they did. Some people did great things, some people terrible, and lots in the middle.)
In conclusion, I would strongly recommend a "take home" test. Maybe give a bit longer than 24 hours. And make sure it's the same problem all the time, so you have a point of comparison.
Thankfully, with application security, you can mitigate most kinds of cheating by presenting a custom-written black box and require a candidate to attack it in some way. Unless previous candidates are leaking or sharing info, there's not that much you can do to cheat on that, especially if you have some reasonable time limit (< 5 hours).
For a dev interview, taking something your team has built and scooping out some of its functionality seems like a test that is significantly harder to cheat on.
For my part: we ran this process for over two years and never discovered any plagiarism. Meanwhile, we drastically increased the size of our team and had total retention; from the time I took over recruiting to the time I left the firm, we lost a total of zero of those hires. All of them worked out.
I'd think better would be to have a few of your devs come up with a small app, and set of instructions, that you pass to others of your devs. They can then create a reasonable baseline you can start to measure against. You may need to be a little forgiving initially, if it's not a completely made up example, but it probably is more valid than something that was built for production, with months of domain knowledge behind it.
At the end of every trivia question I'm tempted to ask: so is this typical of what I will be doing here? Will I be implementing rand(7) given rand(5) once a week?
I've also had a bad experience with code challenges. These are given before the first phone interview. I'm happy to do a challenge taking < 1 hour. I once did a challenge that took me 3 hours. Got rejected because they didn't like my visual design (I was interviewing for a backend position).
I've realized that code challenges require no investment on the company's part, so I don't want to do one that requires significant investment on my part.
That sounds like a great question to ask!
Same.
I'm happy to do one after we've talked about the role and done the "warm up" phase. Just last week I got given a 150m challenge on an online platform with very little information about the company, role or code challenge.
It's all automated, which lowers costs for the company, but I think they're loosing out on candidates with this strategy. A little fine tuning and they can have low costs but with a better success rate.
I told them it was rude and I expected to know something about what I'm diving into before setting myself up for 150m.
It's obvious when people are working because you can see them context switch back to you.
I have a current position (full-time job) and really just don't have time for (with each company) 30 minutes of emailing, an hour phone call, a short coffee meeting, an offline coding challenge and then a 3 to 6 hour on-site interview. Additionally the weather in Boston has been dreadful, and getting to places just takes longer than normal.
Again, I already have a full-time position, and finding ways to schedule these into my work day just doesn't really work well. The interviews I've had have gone well, but I'm counting down the days until I find the perfect company that makes me an offer that I like, because the entire process is just eating into my life.
There's a pretty big plug coming next week, though not from me. What we're doing meshes well with the worldview in my post, but it isn't a productization of it. This is just what I really think about hiring.
"The company is called Starfighter. We'll have a little more to say about it next week."
:P
I interviewed at matasano before and regret not really getting it. I think I came close but I'm eager to try again.
I mean, this was me looking for my "job with low standards" so I was going through recruiters, right? That's how you get a job with low standards. You put your resume on dice and include a phone number. Answer the phone. Wade through all the bullshit jobs that have nothing to do with your skill-set. Find jobs you could reasonably do, and say yes to the interviews.
In my experience, this means that I'm competing, mostly, with people that don't have other options... people that can't get jobs through people they know, because they just aren't that good. For a lot of complex reasons, well, that's exactly what I needed at the time.
I negotiate with the recruiter and $150K is within their range. I mean, I'm... not 100%, right? lowered expectations. So I should be looking for less. But I've never had any problem asking for too much; at worst, they laugh and offer me something lower, right?
Anyhow, so I show up at this place and they grill me for the usual 8 hours. I mean, it's not too unpleasant; there's beer and food and everything you expect, and they seem to be really into me. The guy who is going to be my boss gives me his email and tells me to send him an email so he can get things started over the weekend.
The business guy is, well, he's a business guy, but everyone else seems like people I could work with; people I'd get along with, and the skill level was right in there; even in my reduced capacity, I wouldn't be holding the team back.
At the end of the interview, they ask what I'm looking for, money-wise. Now, like I said, I negotiated this with the recruiter, so I said $150K, but I noted I could probably be talked down some, and I mentioned a lower sum, one a friend who has worked for me for most of his career recently got.
The mood immediately changed. We said our goodbyes, and I left. I did email the boss, but total radio silence.
I never heard from these people again. I eventually got the recruiter to tell me they didn't want me, but...
Personally, I'd be less pissed I spent 20 minutes on a coding problem for them at home and then didn't hear about it.
My favorite method, really, on both ends of the fence? Just hire the person and start them working, for pay, with the understanding that this is still an evaluation type deal. If it doesn't work out? if I'm the one being evaluated, at least I got paid for the time. If I'm doing the evaluating, seeing some one actually work is a way better indicator of, well, how well they work than anything else.
The problem with this method is that it doesn't work if you are trying to hire someone that values stability away from a stable job. You are limited to people that are unemployed or that don't value stability.
Sure, that's a lot like what I was getting at with the whole valuing stability thing. It can also work if the person being hired is having a hard time finding a job.
>Also, ideally you have a specific small project for them to work on.
Actually giving them a project, you know, meeting the legal definition for "contractors" is super rare, at least on the middle where I work. I do see it happen, but it's usually those "10x" programmers who get those jobs. Or it's jobs working for poor companies that pay very little.
In the middle, you find people like me, usually working through corporations (or in my case, whole chains of corporations) that end up paying the contractor on a W2... the idea being that the irs is less likely to reclassify the kid if someone is paying payroll taxes somewhere - because the work does not make the legal definition of contractor.
As an aside, I think most of this shell game isn't about legal liability; it's about making it clear to the other employees who is a temp and who isn't. From a legal perspective, compared to what you pay a body shop, firing someone just isn't that expensive. My belief is that the major cost in firing someone that is performing okay is in morale of your remaining employees. firing your contractors first allows you to have the flexibility to fire someone without making your full time folk feel like they might be next.
>Finally, you have to be able to pay them the market rate. If all of that fits (rare) then it's a great choice.
See, it's not rare, and it happens at all pay grades. Different pay grades have different rituals and ways of arranging things, though. Right now I'm working as a "contractor" - I'm going through some shady body shop, but I actually sit at a desk in the office and eat the free food with the employees. They pay okay, and the expectation they set was that I'll be a contractor for a year, I mean, assuming things work out, and then if we like oneanother, they'll hire me full time at the end of the year.
That kind of arrangement is really common in this industry when the economy is good. They need people now, and I can be a person now, and if they like me, well, that's an easy hire. If they don't like me, or if market conditions change before the year is out, they can let me go without damaging employee morale as much as firing a full-timer. (similar arrangements, with less emphasis on you becoming a full-timer at the end are common when the economy is bad.)
I like it 'cause I'm going through a company that I mostly own, and it gets a bunch of money corp to corp which I can use to pay my people to see if my business can get off the ground, (it pays me, too, on a W2, and buys me mediocre health insurance) and if at the end of the year, it does take off, well, I quit, and hey I was a contractor, right? that was the deal. and if my business doesn't take off, I've got a foot in the door at a decent big company where I might want to actually work for a few years, you know, put something on my resume besides abject failure.
But, as far as I can tell, this arrangement is super common. Actually going through a corporation you own part of is less common, but not unheard of.
It sounds like you have come to the same conclusion psychologists are coming to. Under stress, the human mind uses mental shortcuts, or heuristics, while making important decisions, and act irrationally.
I don't often come across a post that takes on these biases so systematically, and translates them into one of the most important decisions a startup has to make, i.e. the hiring decision, and you even invent useful ways to mitigate their effects. Kudos for that!
Every company should be doing this. Overcoming bias should be a top priority at all HR departments.
-------
[0] http://en.wikipedia.org/wiki/Heuristics_in_judgment_and_deci...
One thing I wonder. Why are people being interviewed for technical positions in a google doc or whatever fake coding environment? Servers are cheap.
If you have to go the way of the technical phone interview, doesn't it make more sense to have both (interviewer and interviewee) parties SSH into a server and work together on some type of problem for an hour? The interviewer can even watch the candidate program or work in screen. This type of interview isn't perfect, but closed book "Programming Jeopardy" style interviews are just worse in every way.
I can't believe anyone would use Google Docs. Even if you don't care about your interviewees' experience, huge waste of dev time.
When they told me they passed they also told me that "some people get passed the first time and then study for the next 18 months and do amazing the next time!" and I just don't have the time for that. I already worked myself half to death in my twenties for two startups, I'm not going to study for a test as a second job for a year just to get a job at Google, although I still think it'd be fun and clearly challenging.
I recommend studying your ass off for algorithms (specifically figuring out what algorithm would be best for different real world problems, especially in regards to various Google products, including those you've never really used before) and practicing writing code on a whiteboard before you go. I studied for a solid week and a half beforehand, and it wasn't enough. I was prepared, just not prepared for what they asked me.
That's a common issue I have with these interviews, is that computer science is such a vast field that it's impossible to have everything in your head ready to shoot off in any interview, but interviewers somehow think that if you missed a question or two you're somehow not qualified to work for them, despite having years of direct experience at companies beforehand and being able to Google and refresh literally any topic you might encounter at your job in less than a minute.
Google seemed better than most at that, though. I still found the experience to be valuable and interesting.
Sounds like a nightmare.
What's "good for google" right?
@cableshaft how would you rate @tptacek s observations on hiring with what you observed at google?
I'm going through interviewer training at Google at the moment. (There's a few courses they like you to take before letting you loose on candidates.)
With all of that "Cracking the coding interview" book in my hand anyway! :D
We'll see what happens!
“bring the same level of rigor to people-decisions
that we do to engineering decisions.” [0]
[0] http://www.tlnt.com/2013/02/26/how-google-is-using-people-an... [1] Adam Bryant: 'In Head-Hunting, Big Data May Not Be Such a Big Deal' http://www.nytimes.com/2013/06/20/business/in-head-hunting-b...
Possibly, I don't see google going down, so the engineering is sound. That's missing the real issue though.
Google, the people who lead and work there fundamentally do not grok people or psychology. This will is problematic as they attempt to diversify their workforce. I'm sure this is a known-known at google, hence the training that @eru is receiving. This is why the @tptacek article is such a good read. A tech-company attempting to understand more about hiring humans who understand machines.
People are not machines.
cf: http://www.sfweekly.com/thesnitch/2015/03/07/former-google-e...
The first one taught me a bit about the company and team, and gave me a good impression of the interviewer (the hiring manager), which then led to me doing a work sample. (Without the phone screen I don't know if I would have sunk several hours into the work sample.) I ended up getting an offer but not taking it.
The second one also gave me a good impression of the company, team, and interviewer (a senior engineer) and led to me doing an full day of interviews. Which led to me accepting an offer. The one used Collabedit in addition to the phone.
I know they're not perfect, but it's much cheaper to talk on the phone for an hour than to fly someone in for face-to-face interviews. Given way more resumes than resources for in-person interviews, I'd rather narrow the field using phone screens than narrow the field based solely on what's in the resumes.
I saw the part in your article about work samples, and I agree that they're great, but they are significant work for the candidate. So I think it makes sense to do have some kind of phone call first, to see if there's enough mutual interest to make it worth the candidate's time.
A phone screen is a call whose purpose is to select out candidates. It's a call whose outcome can be "stop talking to this developer". They're very common, and I think uniformly evil.
I don't know how you avoid that. If you have 1000 candidates per open position, you probably can't afford to do in-person interviews with all of them, or even all the ones with decent-looking resumes. What's the alternative to phone screens? An automated remote work sample system, maybe?
This particular phone screen was probably the most nervous I've been during any interview ever. Predictably, it was enough to disqualify me.
"Newline. Tab. Tab. Tab. Tab."
A Lisp would be nasty, too. "... Close paren. Close paren. Close paren. Close paren. ..."
I ask people to do the best they can to get the syntax right, and I will ask for corrections if it's way off.
I tried to copy/paste in/out from vim and it was a disastrous process. Nearly I could not use my previously implemented bst implementation. (Which I was told to).
I screwed the test - of course it was not HackerRank's problem - and retried in vim after time ended. I could code it in a fraction of the time that I could in that ide. Time limits never work for the candidate.
I do that on all remote interviews. Code a simple-ish thing that doesn't require that you've read about it before and show me how you'd do it. Complete with the build & run command.
That said, my wife happens to be a teacher. The process from what I can tell in private schools is pretty similar. Usually a recruiter ("placement agency") sends a bunch of resumes (CVs) to a hiring manager (department lead or school head). They pick some based on how they feel that day and conduct phone screens. Successful applicants then come in to do the equivalent of a whiteboard interview -- they are evaluated on their ability to teach a classroom of students they've never met a lesson plan they've probably not seen more then a day or two in advance (although at least they get that much warning!)
edit On reflection I hope that teachers are at least a little more capable of objectively evaluating candidates then your average sysadmin/software engineer (how often are you asked to assess someones ability outside a handful of interviews?), but believe me the sample lesson causes as much stress on the candidate as a whiteboard interview. It also stresses the teacher whose class is being "donated" to candidates and depending on the length of the process can derail what's usually a carefully planned out quarter/semester/etc.
In other words, a work sample test.
With many other industries, you can say "Do this" in a shorter, observable time frame and have it more aligned with what they'll be facing on the job. Contrast this to dumb interview coding challenges, for instance.
I myself am a pretty mediocre developer. But fortunately for me, I realized that tech interviews are complete and total BS.
So what I did was spend a whole bunch of time studying whiteboard algo questions, and I became really good at interviewing. I've gotten a couple awesome jobs that I had no business getting (and usually get fired after about 6 month, because I'm a shitty dev, but whatever, on to the next job).
You saw the flaw in the system and studied so that you could exploit it improve your own situation. Some people may be mad about that, but I'm honestly not. I'm sad that you put effort into practicing interviewing and sound like you're not putting effort into practicing your actual job. You say "I'm a shitty dev", which is a fixed mindset type of thing to say. You identify yourself as your current capabilities, as opposed to a growth mindset where you may say "I'm working on improving my dev skills".
It sounds like touchy-feely pop-psych, but think about it: you saw that getting better at interviews would land you better jobs, so you worked on interview skills, and you got better jobs. Now if you took your current job (and your next one) as a chance to work on your dev skills, maybe you won't get fired after 6 months. Maybe you'll grow in to absolutely deserving the awesome job, and then grow beyond that.
Please think about it. Paraphrasing an Adventure Time character, being a shitty dev is the first step to being a kind of good dev. And we need more good devs and fewer firings.
Now, tptacek:
What can I do to make this better? How can I contribute?
How would you compensate for candidates that are great developers but are new to Rails, or whatever specific language/framework your company uses?
1) If you think that the language/framework is intimately important to the hiring process (I don't but I'm not judging) then you don't compensate for that. It's an important data point.
2) If you don't think its important, you don't worry about matching "idiomatic" practices. You don't judge on obvious library mismatches, etc. If you don't feel confident doing that. Don't have the challenge interact with any language specific code. Let them do it in any language you feel confident of judging.
The less flippant answer is, you give them a deadline but it is both A) flexible in the face of their "real life" (ie if they tell you they are going on vacation next week deal with it) and B) it deals with realistic time horizons (ie a week, not 4 hours).
Courage
Determination
Unselfishness
Cheerfulness in the face of adversity
These sure come in handy in some programming environemnts :-)
After going through the march of those two big companies I found the process incredibly annoying and not useful at all; nothing of substance was really ever discussed it was almost always academic or some sort of trick question where I get to hear the interviewer talk for half an hour how they came up with a superior technique.
I've interviewed a lot of people. Interviewing software developers is such a pain in the ass.
The hope that as more companies figure this out, there will be more and more companies with interesting, challenging and rewarding work as opposed to companies that are only a paycheck, and as soul-sucking one at that.
The dread is that I would never make the grade at the former kind of company.
I think that dread cannot be decoupled from hope. And that this is especially true for folks with higher developmental potentials.
It seems to be working out so far. I don't have a lot of data to go on, but I haven't had to completely throw out code yet.
Major citation needed. I don't have any real data either way, but I find this extremely hard to believe.
Yeah, sure, I can buzz your fizz with my eyes closed in brainfuck. But that's just measuring language facility.
The real tests are more topical, and more complex: is there a way to make this code run faster?
There are two types of good candidates that will get that question:
1) the kind who have run into the problem before and know the answer stone cold, and tell you instantly
2) the kind that have no experience in that particular area (say, search algorithms) but are google-proficient, such that if you give them a day or two will talk your ear off about them.
In a work situation, the two are really equivalent. After a day, 2) is indistinguishable from 1).
But when interviewing?
If a candidate needs a day to learn up about something trivial they should be able to derive in minutes, then that kind of thing might compound and end up producing a huge difference in actual productivity ("all other things being equal").
1. Those who had seen the question before, decided not to tell you they had, and faked figuring it out.
2. Those who had never seen the question before, but were able to figure it out.
You can, of course, distinguish them from the following, but you can't distinguish the following from each other.
3. Those who have never seen the problem before, could figure it out, but do poorly in the pressures of an interview.
4. Those who have never seen the problem before, and could never figure it out.
And it is -possible- you get the honest candidate -
5. They've seen the problem before, they -tell- you they've seen the problem before, and go on to solve it easily.
#1 looks the best, and you've learned nothing about them (and they have nothing in their favor other than having seen the problem before). #2 looks okay, but pales in comparison to #1, despite having actually demonstrated ability. #3 looks poor, but you have actually learned nothing about them, and they may in fact be absolutely amazing, except not good with interview pressures. #4 looks poor, and is indistinguishable from #3 (unless they do so badly that it's clear they have no idea what they're doing, rather than just being off in the weeds somewhere). #5 you now know is honest, and that they've boned up for your trivia challenge, but nothing else.
You haven't measured anything you set out to measure. You wanted coding ability and/or ability to reason out a problem; all you got was whether off the top of their head they were able to solve this particular problem. If that was what the job entailed, parroting back answers from Cracking the Coding Interview and the like, then you'd have a good test, but that's probably not what you need in a developer.
Actually I think this is the worst kind of question to ask, because it measures: can the candidate code under pressure? Not "we need to get this done by launch/before client meeting" pressure, but right now pressure.
If a company wants literal coding ninjas who can reason about computation while in the middle of lightsaber battles, sure. But in the much more likely event that they're hiring people to sit for days on end and work on a large project, then this is an unnecessary obstacle.
I'm probably on the wrong side of reality since plenty of people make lots of usable stuff while using vastly suboptimal approaches. OTOH if someone doesn't known very basic stuff, it seems unlikely they're gonna Google stuff and become wise. We're not talking about anything advanced; these are basic algorithms and data structures. Some people just aren't going to be able to handle the concept of memory layout so might as well figure that out sooner than later, right?
> Not only are you putting the candidate under pressure, where thinking is difficult, now you expect them to walk you through their thought process and narrate what they're thinking, disrupting the thought process.
I expect a capable developer to be able to explain to themselves why they are choosing (or avoiding) an approach. If they cannot to that, would I trust them to be able to explain their methods or WIP problems to other developers?
When I do interviews, I care less about the solution the candidate comes up with, and a lot more how they ended up with the particular solution they used. The process of iteration and dead-end elimination is fascinating. This means that the best questions have N+1 different solutions. Some will be less optimal, but that's life. I also find discussing the effects of different solutions fruitful, on the less common occasions when the candidate ends up tossing one approach and settling on something altogether different.
And by the way, I really dislike the idea that you're supposed to do an interview in less than 1h. For anything remotely realistic, that doesn't even let you scratch the surface.
I don't have a strong or well informed opinion about it, but I would not be shocked if the answer is no.
People were saying similar things 20 years ago, too. Others will likely corroborate that from earlier.
It makes me sad that I will probably never see a single one of your suggestions implemented by any potential employer that might want to hire me, specifically.
This is because there is one more major problem that you can't fix from where you sit. The people creating the interview protocols do not solve their problems like software professionals. They don't collect and analyze data rationally. They don't control the variables. It never occurs to them that there are a dozen flaws in their system that they just haven't discovered yet. They don't read HN.
They don't realize that software developers can write the kind of software that runs on people, too.
The only reason I ever designed a hiring pipeline is because I was asked to be one of the people in the interview pool. Me (and others in the pool) decided we were doing a bad job and started iterating on the problem (much like we would for software). Management was thrilled with both the initiative and the results.
That was years ago and working on hiring pipelines has become something I just do. I've never had an employer push back or make that hard. Quite the opposite, they are usually happy to have help.
Try fixing the problem in your current job and see. You might be surprised.
I also encounter the problems endemic in the hiring process far more often at other companies than I do in my own. The reason why I start interviewing elsewhere is often because my current company has stopped hiring (or started firing). I have occasionally tried giving those other companies feedback, but the response has always been, without a single exception, "We know what we're doing; don't tell me how to do my job."
I have neither lever nor fulcrum for this problem. My frustration is not without reasonable cause.
I was implying that you could see this at your current/next employer by implementing it there. If you fix it there before the time comes to move on you personally may not reap the benefit but someone else might and the lessons learned by you and your colleagues get spread further out. If nothing else, you learn how to maneuver hiring pipelines quite well by designing them.
> The people creating the interview protocols do not solve their problems like software professionals.
As soon as you are asked to become part of the hiring pipeline whether it be resume sorting, phone screening or interviewing, you become one of the people creating the interview protocols. You can apply whatever tools in your arsenal to fixing it. I have found using standard software development methodologies to be very compelling and useful in this context.
> The reason why I start interviewing elsewhere is often because my current company has stopped hiring (or started firing).
This is a problem that is more worrisome than involving yourself or not in designing hiring pipelines. It implies a reactive approach to your career, and that is likely to lead to suboptimal results, and can be actively detrimental to your job prospects in bad job markets. I would look to the root cause of that behavior and see if it lends any insights.
Second, work-sample type of questions are also expensive to answer for candidates so you can probably just ask one question as part of whole interview and that's about it. So any conclusion is drawn from sample of 1 as opposed to sample of 5.
Third, work-sample type of questions requires knowledge of engineering skills that candidate may or may not have developed well at point in time. At many companies, emphasis is how can we solve problem X from algorithms perspective and engineering is just something you learn on the way if you haven't already. Key is ability to figure out computer science part rather than language and tooling part.
Overall, I think many companies blindly try to follow the model of Google, Facebook etc and fall on their face. If you expect your developer to write CRUD apps or do mostly plumbing work, there is no point in asking questions like find common ancestor in binary tree. At other companies, developers are expected to solve computational problems and engineering/plumbing/CRUD is small part of the solution pie. There it makes a LOT of sense to ask candidate to solve problems that requires deep computer science. Also asking as many of as possible so that your sample space is large and outcome has more confidence.
http://www.matasano.com/careers
This sounds onerous, but it is less onerous than the normal dev hiring process, which involves an onsite interview that eats the whole day. We did on-site interviews, but they took just 2.5 hours to complete; candidates would be out at lunch. Essentially, we were shifting some of the time candidates would spend sweating in a conference room to their couch instead.
There are bad work-sample tests. "Bad" comes in a variety of flavors. I think bad work sample tests will out-predict the median interview. But there's a lot of room to optimize and improve this process, sure.
This is somewhat alleviated by the knowledge that you do a phone screen with each person prior to that stage, but it still leaves me with something of a bad feeling.
Uh, no. Have you ever been paid for your time at an interview? For most people, it costs them at least 1 day worth of pay (holiday or day rate).
Most importantly though I like programming, but I don't like speaking up in front of a group of strangers, so 6 hours of programming is much more enjoyable to me then even just 1 hour of interviewing.
The process is already badly asymmetric in addition to being artificial. At least the work sample ideal addresses one of those.
I remember talking with a manager after an interview, and listening to him go on about how his company was empirical and data driven. Later, he mentions how this one interview activity was valuable because it resembles something in the job, and I asked him what his empirical data was, and he had to admit that he was caught out.
1) start with basic textbook questions like what is authenticity and authentication or XSS
2) catch what the interviewee said and build questions (e.g. I said something about private key so interviewer asked me about pro and cons of asymmetric and symmetric encryption). Oh yeah - know your shit because they are going to catch you! It's okay to say "I don't know." Being straightforward earns respect. My interviewers didn't penalize me much (well I just graduated from college...).
3) the next couple interviews again starts with introduction, then deep dive into what the team does, what the team is building at a high level, then proceed to ask me my interest. Here i would talk about my ideal projects, show them high level how I would go about implementing my idea, challenges I face (and also why I have to build one; are there any existing solution and are they not adequate). Take caution of your words - know the things you say aloud.
Somewhere in those 4-6 interviews, add a programming sessions if you haven't done so (for me I skip that and went to onsite because of internal referral).
I didn't get an offer probably because I didn't quite know what I really want to build. My idea was too generic and probably too "child play." It was a really intense and yet fun interview. This interview process allows interviewer and interviewee to see if they are a match or not quickly and pleasantly. I always look back at this interview and believe that the rejection is just and great for me and for the team.
The net result was a win-win: you get bargain candidates and they get 4-page CVs.
1) Hire based on current ability, not potential. Hiring based on potential is a minefield that our whole process is designed to avoid. ("Enthusiasm," "cultural fit," talking a good game without being able to walk the walk, etc.)
2) Be aggressively fair in giving out promotions and raises. Keeping someone's salary low because they're bad at asking for raises is not a good long-term strategy, especially in consulting.
Seems to be working out.
> Keeping someone's salary low because they're bad at asking for raises is not a good long-term strategy, especially in consulting.
Yes, in consulting the distance between money made and each individual employee's efforts seems shorter than in a big tech company. So keeping salaries in line with impact of contributions should be easier.
There's a common set of skills that most any software engineer should have. I think those skills can probably be checked via a web tool, and in higher fidelity than you can get over the phone.
As a candidate applying to N jobs, I'd much rather take a web screen once instead of doing N technical phone interviews. And as an interviewer, I'd much rather not have to ask FizzBuzz again.
One stage of the pipeline is left intentionally out. The candidate then is required to build an actual service. They can use any language they want. Even any cloud host they choose. All that's provided is the entry points, exit points, wire protocol, data encoding, and the spec to be implemented. After passing that, advanced topics like logging, telemetry and performance provide grounds for further discussion to assess development "philosophy".
But the real genius would be selling this work-sample platform-as-a-service to enterprise customers. Who could be prompted into building a catalog of microservices based upon their APIs and real data. And out of that hopefully some innovative, hackathon-esque mini-products could arise ;)
I definitely never thought I would become an "HR guy". But I really like this idea. Will include a contact email in my profile if anyone is interested in discussing it further off HN as well...
1) The main idea was to have the results of unit tests visible live. I've done a few tests where I found out I didn't get through even though the test rig says it works. It compiles, it gets the examples right, but there's some hidden test that failed. You'll never know why, even though it's probably not something surprising.
So just make it explicit. A bunch of lights for each test, a description of what the test is, and there you go.
2) That solves little problems like "given some set of numbers, how do I find some ridiculous property of those numbers?" type questions.
What you really need is something that tells you how people deal with complexity. I have a large problem, how do I solve it? There's always more than one right answer.
Here you might be better off doing some kind of subjective voting. People who are looking for work might find it comforting that other people in their situation are judging them. Or gratifying to be able to judge other people's skills. Perhaps there's some incentive structure that cleverly aligns this.
[1] http://www.launchfestival.com/theinterviewclub [2] http://www.launchfestival.com/prehire
Minor nitpick from a grad student here: interviews are nowhere near as hostile as academic peer reviews for papers. In an interview, the interviewers aren't actively trying to find any holes in your logic (no matter how small). To me, technical interviews are a walk in the park compared to conference submissions.
Hiring isn't broken because people use the wrong interview questions. It's broken because firms are engaging in a zero-sum battle for talent. But if what you're doing is worth a damn, it should be worth a damn when ordinary people do it. If what distinguishes your company is that you've managed to hire smarter people, maybe your company isn't so great.[2][3]
The proper hiring criterion is pretty simple: do the people who will be working with the candidate like her? In Blink, Gladwell presented good evidence that formal interview process isn't any more objective than this. And when you think about it, what would "objective" even mean?
There's also an issue with the labor market -- in particular, the difficulty of firing. You're much better off hiring liberally and firing candidates that don't work out. If all firms did this, the labor market would be more efficient and firing wouldn't be so bad.
Apple is a good example. Their profit per employee is second only to Netflix among large tech companies, and this includes a significant fraction of retail personnel. Apple's profit per engineer is probably higher than Netflix's, and that's remarkable considering they're over twenty times bigger.
Does Apple achieve this with some optimized interview format? Higher salaries? Onsite massage? No, they do it with better products (and a command and control governance model...).
It means that the software industry isn't nearly as innovative as it thinks it is. This is actually so obvious it's the subject of popular jokes ('like airbnb for pet food' etc). It's also obvious when you consider the ratio of effort spent building tools to products. All the latest frameworks but... Where are the apps? Where are the apps that are even half as feature-rich as a typical Windows application from the '90s? Google Plus still can't keep which things I've +1'd straight.
[1] It reminds me of the situation with essays claiming to explain why privacy is important even if you 'have nothing to hide'. See https://gist.github.com/clumma/414f0c9a8fedf9ba9db5
[2] https://twitter.com/clumma/status/139082596587012096
[3] Or worse, that you've managed to deprive your competition of smart people. Google has done this -- hired people without having work for them to do, on the reasoning that something might come up and at least they won't be working for the competition.
It's just a toxic environment.
Where do you think the compelling products come from?
There are plenty of companies that take the view that you seem to be advocating - that there are employees who "create" and other employees who just turn the wheels and implement those ideas. Your average fortune 500 company is just like that. The engineers working there pretty consistently complain that the software dev team isn't respected, and isn't sees as producing things of value, or providing a competitive advantage. Those companies believe that thevakue isn't in the oroduct/idea and the execution is easy and ought to be cheap. Which is fine, as long as you never need to do anything that requires significant technical skills. Because such environments force out anyone who thinks that they gave more to offer than simply churning out cookie-cutter, mediocre code.
Their are ways to build successful teams that don't require that everyone is top 10%, but the notion that you just need compelling product ideas and a bunch of average developers to implement them is pretty common place, and rarely produces exceptional products.
Top universities, which for the most part feed the tech industry, face a similar problem at an earlier stage in the pipeline. It's been noted that less prestigious universities tend to draw their faculty from a very narrow selection of elite universities.
The education and labor markets are linked, and inefficient in ways which perhaps reflect the same underlying cause: scarcity of people with apparent ability, and self-interest wanting to profit from oligopolistic control of this incalculably valuable limited resource.
[1] Should be noted that it isn't the entire group, since some will choose to work for organizations outside this group because the work is more interesting or fulfilling.
In relative terms, the bar for, say, the NBA is much higher than for top universities and tech companies. They set the bar high enough so that, for all practical purposes, minor league teams cannot come close to matching their combined level of play. Also, if you don't compete regularly against NBA caliber players, with NBA quality coaches and trainers, it would be hard to reach the same level of skill even with comparable native ability. I don't think the league is that concerned with missing out on marginal players, as long as on balance it gets those correct. The disaster would be to miss out on the Michael Jordans or Lebrons, Bird and Magic. So they scout and pay through the nose even for the slim chance of that kind of greatness, with the tolerable outcome being a good role player.
At some level, we're all looking for greatness. Nobody wants to see a league entirely made up of role players doing somewhat above average things.
The analogy breaks down in that it's a lot easier to accurately measure (though perhaps not predict) athletic ability. I do think that investors in tech tend to heavily favor obviously stellar performers over "good but not great" ones.
I'm sure that's intentional. Joel Spolsky wrote an influential guide to interviewing in 2000 that lays out this philosophy; the latest version (updated in 2006) is here:
http://www.joelonsoftware.com/articles/GuerrillaInterviewing...
The gist of a large part of the article is that interviewers should have a preference to saying "no hire", because bad hires are toxic and hard to get rid of.
But then he writes this:
----
Of course, it’s important to seek out good candidates. But once you’re actually interviewing someone, pretend that you’ve got 900 more people lined up outside the door. Don’t lower your standards no matter how hard it seems to find those great candidates.
----
Which is terrible advice if your standards are, in fact, unreasonable.
I hear you. One reason for this is scalability. You can build richer web apps if you aren't shooting for scale.
I worked on two startups with conventional LAMP stacks and heavy user sessions before joining Yahoo. At Yahoo there was really no session object[1], due to scalability. A few cookies could be set to customize behavior, but adding new cookies was seriously frowned upon. This made me appreciate sessions as a means to adding in nice little features.
In principle you can solve that by storing the session in a scalable key-value store, providing the latency is low enough.
[1]Generalization only valid for the properties I was exposed to at Yahoo.
Do you think you've learned lessons that could be applied by individual contributors in large orgs who are called on to interview? If so, it'd be great if you added that to your blog backlog. :)
pernicious - Having a harmful effect, especially in a gradual or subtle way.
New word for me. :)
Don't get me wrong, I can code, I'll write you anything you want, AND IT WILL WORK! and I'll still know there may be a better way but this is the way that works and I still think I don't have enough years to live to argue with social incompetents who think that fighting over tabs or spaces is a good idea and a pocket protector makes you sexy.
When we started doing work sample hiring, we put a process in place to handle many applicants, make sure we were responsive, and work people through the funnel as efficiently as we could. I don't think it's possible to take advantage of it any other way.
We got complimented several times on how reasonable the process was (by people we turned down!). This both made me feel good, and caused me existential "the world is broken" sadness. It's not a high bar to be decent to people.
Still, a company could have a great, responsive process in place. But my experience, and that of others, is that it's not worth my time and effort to do work for free. Let's be clear: this is what is being asked of the candidates. It therefore starts to undermine any company using such a process even though I don't think it's an inherently bad process.
How they're implemented has varied wildly between companies, but most of them were used as an initial screener rather than a major deciding factor, despite being one of the most time consuming parts of the process for the candidate.
Like you I was able to go through with most of them, but I felt like they were a massive waste of my free time and companies were often very slow to follow up in it. I think a lot of the time it just isn't worth jumping through these hoops for a company I don't know anything about.
My best test I took was around an hour and involved building a really basic CRUD app in Django as part of a larger face to face interview, followed by a short discussion. It worked well because they set up an environment for me, the data model was fixed, and there was a list of short requirements I could complete in order, so I was able to just dive into the work, and spend time on the kind of things that were relevant.
Another one basically asked me to take a large dataset, build a web application around it, and deploy it, which would probably have taken me a week to complete but "should" have taken around 3 hours. They tried to present it a really interesting problem I would enjoy working on and then asked me to keep all my work confidential.
Most of the tests I've done are still closer to general problem solving/comp sci tests, which is still pretty far removed from the actual work, but at least they're not a huge time sink when implemented properly.
If a kick-ass code sample could have turned a junior-level engineer position at 0.01% into a Director-equivalent (I like being a full-time coder and don't need reports, but I generally prefer that management see me as an equal, so I can get my job done, because a 10x engineer disempowered is a -3x engineer) with above-market compensation and legitimate equity, then I'd say... yeah, these things can spot early talent that nothing else does. But, as far as my experience has led me to think, code samples just provide another reason to reject someone. ("This guy used the TestEase framework instead of EasyTest. What, does he think it's 2013??")
I'm a good technical interviewer. I'll say it: I'm very good at it. The first thing I do is that I explain my methodology, rather than hiding what I'm after, so the candidate is at ease and not surprised by difficult questions. I tend to prefer a "rapid fire" approach to collect as much data as possible, so as soon as she's demonstrated that she knows what she's talking about, I'll move on to another question or topic... and I disclose that this isn't to be rude but so I can get the best read possible. I tell the candidate that, unless she is a 0.1% outlier, I will ask questions where she doesn't know the answers, and that's OK. It's not a percentage-based test where you need to get 80% right or it's a no-go, but it's a binary search on the competence spectrum. Generally I'll take something on her CV that I'm familiar with, research it a little bit, and start with a mid-level question that I'd expect 75% of the people I'd want to hire to get. (Statistically, you get your best results by adaptively asking questions at a level of difficulty where there's a 50% chance of success. But I aim for ~75% because most candidates aren't used to difficult interviews, and I don't want anyone to freak out.) That means that there's a 25% false-negative among good candidates, so if she doesn't know the answer (and it's OK to say "I don't know"; I'm in the top couple percent of my field and still have numerous blind spots) then I'll give her another question, also of moderate-high difficulty. (I don't start giving easy questions unless I've decided on a "No", and that's usually to put the candidate at ease, because I might still be wrong, and I want the next interviewer to get a clean read.) A good interviewing process provides multiple paths to success, not multiple ways to trip up and fail. I prefer to ask hard questions rather than easy ones because, as far as I'm concerned, 5 hard questions is multiple paths to success (if you get 1 or 2, you're solid; 3-4 means you're excellent) whereas 5 easy questions is multiple paths to failure.
I know I come off as an asshole on Hacker News (because it's turning into this festering pit of pro-corporate Wrongness on the Internet, and someone has to do something about it) but I try to be as nice as possible during interviews. I want to be tough, intellectually, but I want the person at ease as much as possible (which is why I explain, ahead of time, that I've been a notoriously difficult interviewer at every company where I've worked) so I don't get a false negative. I generally give a series of hard questions (not all related to each other) that seem relevant to her experience. I might give four, and a candidate who can competently answer one of them, I would generally say is worth considering. If she gets 2-3 out of 4 hard questions right, then she's probably a good hire and I'll recommend her.
I don't think hiring engineers is as hard as people make it out to be. I do like to see code, but I can usually tell the socially adept non-coders and charlatans from people who can actually program, even without looking at a line of code. I tend to prefer to hire for general intelligence than a specialized "hole" that someone is trying to fill, and the main question in my mind isn't, "Did she get all of my questions right?" but "Would I want to work with this person?" and "Would she add something to the team that's not already there?" I won't remember, 90 minutes later, whether she answered every single question right, and if she screwed up and said "Lasso" when she meant "elastic net", that's not terribly important... probably a mistake, and knowing the ideas is more important than the vocabulary. Besides, I care more about hiring the person who has the curiosity and drive to learn new things than hiring the one who already knows what we're "looking for", because the latter changes in most companies by the quarter, while a candidate's general ability doesn't.
Bad hires tend to come from two sources. One is when non-technical people override the technical interviewers, either formally or informally. It's best to have an environment where people can speak freely and, preferably, independently (i.e. each interviewer is on the spot to give feedback before discussing the candidate with others, but also feels safe giving an honest read). The worst thing that can happen is when an executive says, "This person is amazing", before any feedback has been given. In many companies, that means that the decision is already made and the engineers are just there to ratify it. That produces a lot of underqualified hires. The second is stack-ranking, which encourages teams to keep an "insurance incompetent" on hand, so that the middling and top players can safely focus on work, knowing that they won't end up in the bottom pool. In companies with stack-ranking, however, I would generally avoid interviewing as much as possible. (In fact, this may be one reason why companies that use stack-ranking tend to decline.) If stack-ranking is in play, you need to focus solely on (1) your main project, (2) getting credit for what you do (the politics of performance, which is more important than performance itself), and (3) being perceived as a top performer without actually trying to outrun the bear (outrun the other guy, and he gets eaten; outrunning the bear is impossible). If you're in a stack-rank company, then interviewing is that class of work that's good for the company but won't improve your Perf score, so you should avoid doing it as much as possible.
When the author says selecting for: "people who have the social skills to actively listen to someone else’s technical points, to guide a discussion with questions of their own, or to spot opportunities to redirect a tough question back to familiar territory." is a mistake, because some of them can't code. I know how to turn someone who can't code into someone who can. I don't know how to turn someone who fails at dealing with humans into someone who succeeds. And, at least in my job, engineers spend a whole lot more time dealing with people than with code.
If you can do that reliably, you should be making zillions of dollars.
My experience is the opposite: if someone is very impressive in an interview but a zero on the team, they're an intractable management problem.
In any case: there is a difference between being cripplingly antisocial and being able to deftly handle an interview. The social skills required to flip the script on an interview are distinctive, and they aren't the "team player" skills. In fact, as I re-edited that paragraph and came up with "controlling the conversation", it occurred to me that they might signal a candidate who has problems on teams.
I disagree to a degree; I've met many people who were simply fantastic with figuring out problems but didn't know much software development and they ended up being awesome software developers.
It's certainly doable and I don't think it's so difficult that someone who can do it should be making an obscene amount of money but they are not cheap either; that's usually a really good development lead.
There are people from all sorts of backgrounds that can become very good developers, but their common feature seems to be that they were already the sort of person who could become a good developer, not any methodology used to train them.
For example, I'm confident most talented physics Ph.Ds[1] can make decent developers of numerical code, but that says more about them than it does about my skill at training.
Given a random person, hell given a random person who is already decent developer in a different domain, my confidence in their ever becoming very good at that domain is much lower.
[1] the problem then being, that pool maybe smaller than the one you are trying to populate....
I don't think anyone was saying a random person but someone wanting to get into the field or is already in the field but perhaps not in the right spot when you pick them up.
I can't imagine anyone would mean a random person here, that wouldn't make sense as development is a skilled position. not everyone can do it and OP was talking about interviewing developers not random people.
It's a better representation of the job than writing code on a whiteboard.
The question shouldn't be "is it perfect?" The question should be "is it better than what we have now?"
I'd say something is wrong then. If your business depends on shipping code why are your developers talking to people most of the time?
What kind of job do you have?
You judge someone based on their skill level. You have to know what require('fs') is in Node, but you also have to understand truths like the concept of technical debt and so forth.
Not everything is broken or "needs a new paradigm". Know what programming is. What high skill means. If you're any kind of leader - TEACH someone younger or less experienced. When you need to judge, for hiring or otherwise, measure against those truths about what good software development is.