We’re Bad at Interviewing Developers – Interview with Kerri Miller
blog.fogcreek.com
blog.fogcreek.com
I have done a number of code challenges and while I don't mind doing the challenges (I look at them as training) and I can always create a working solution there is nothing worse than receiving a simple 'we won't be continuing with your application' email with no other feedback. If you are going to take up a candidates time with a code challenge the least you can do is help the candidate become a better developer by providing constructive feedback. You owe it to the applicant and also to the community at large.
I have had about a 50/50 experience in this regard. Of all the companies that reject me with good feedback I am left feeling content that I have improved as a candidate throughout the process. The others who don't provide feedback leave me feeling like I've wasted some time that would be better spent reading about best practices and techniques or developing my own software.
Still, as an individual, I am resistant to the current tend where employers look for "someone we'd like to hang out with". I have a life, I don't share the same interests of a recent grad. When I was a recent grad I didn't like the collegiate attitude anyway. So, it's off-putting, but it's a big filter nowadays. It's let's have lunch with the team so you can get to know them....
Beyond the logistics of communicating to dozens of candidates for a single position, legally speaking, telling people why you rejected them gives you ammunition in court to say the process was discriminatory, and a violation of one of the many civil rights act. Simply saying 'we decided to go with a different candidate' gives you a courteous brush off, without giving you reasonable cause to go fishing with during discovery.
The purpose of an interview is not to get better at interviewing. The place for this is mock interviews. Honest feedback is expected to be given without the risk of employment discrimination lawsuits. I figure to the extent that constructive feedback is owed, mock interviewing is the way to provide it.
If you get rejected you were simply not the right fit.
Much for the same reason, competent HR professionals appreciate if you give them a reason for leaving a recruitment process or a job, it helps them identify areas of improvement.
I interviewed for a jr dev position and was not asked one technical question. I am sure that the reason why I didn't get an offer was because I did not wear socks. Seriously. I made sure I always wore socks after that.
Having reviewed code samples I can say that it can be exhausting trying to write constructive feedback to the person who wrote it. It is much easier to give a more dispassionate summary to the recruiter than one that would be appropriate to give to the person who wrote the code.
If I run a web development studio, it is not my job to assist in the personal development of people I've decided (for whatever reason) are not a good fit. Yes, logistically that's a pain in the ass if I send 1 offer letter and 12 explanations about various issues ranging from employment history to experience to skill set to education to interview skills to "I got the distinct impression you would rob us blind the first time you were in the office alone." But most importantly (for me as someone who assists but does not make final determination on new hires), my job is to find the best candidate for a given position and it's only a hindrance to spend time with those who are not.
It seems like a fairly common complaint from rejected candidates that they want the company to continue investing in them after a decision has been made not to move forward. And in the rare instances when I've offered this advice or help after one or more requests, it has been invariably met with an attitude either that I am wrong/stupid for coming to such a conclusion (e.g. not enough education, or no experience with something we want someone to have experience in) or a flat out request to give them another shot.
Yeah, I've hit this and it really is annoying to deal with. I have one candidate we rejected 4 or 5 years ago because his resume was questionable, and once a year he still emails me asking for a position (and I'm not a hiring manager!).
I have had it go the other way, too, though. I had a candidate who didn't meet some criteria. He asked why, and I told him. A year later he had done a bunch of work to improve himself and we hired him.
So it can work both ways. But I definitely agree that the more common case is for the candidate to rationalize away why the company is wrong.
And I think companies that got feedback from candidates would do the same thing: "He said we didn't offer a big enough salary, but our pay is exactly the median of other companies around here. He's just a prima donna!"
Show them your github, credits in some os software, side projects, dont be another no name code monkey.
I completely understand why. Two very good reasons have been identified in the comments here: 1) it opens the company to litigation, and 2) the company's goal isn't to provide feedback, it's to find good candidates.
Here's the problem, though - none of this makes things any better for a candidate who spent weeks or even months studying, took hours of in person exams, and is brushed off with a one line "not a match at this time". And after a few of these experiences, the candidate starts to give up on trying. Tech interview exams may deter people from interviewing, and may play a role in why developers with potential give up on the field entirely.
http://www.fastcompany.com/3043082/most-creative-people/why-...
High tech employers almost universally claim that there is a severe shortage of candidates, and yet here we have a process that may be deterring developers from interviewing or even remaining in the field. There are solutions out there, but we're not going to get to them by saying "oh well, it would open us up to litigation, and that's not my job anyway". Well, true, you don't owe anyone a job or feedback, but the world sure doesn't owe high tech employers all the developers they want to hire either.
So far, I really don't get the sense that the high tech industry has come to terms with the depth of its own role in why they are having so much trouble hiring.
I'm terrible at operating under pressure and generally need to take my time with any problem I'm given. Unless you actually work in an environment where it's critical find the optimal solution to a large constraint search problem in 30 minutes or the tower blows up I don't see what the difference is to taking a couple of hours to find the solution. The only way I'd be able to do that is if I'd seen the exact problem before a couple of days ago and can recall the solution from memory. For problems I haven't encountered before I need a pad of paper and make a lot of mistakes before getting to the right answer.
I'm just slow, deliberate and not well equipped to win at Jeopardy. It's not my bag.
But if you asked me to teach you something I could go on for hours on procedural generation algorithms, pseudo-random number algorithms, neat tricks from old languages, compilers, macros... yadda yadda.
Not to mention, if this scenario is frequent enough that you really need to screen for it in interviews there is probably something fundamentally wrong with the way you are running your business.
The reason I like the suggestion is that it asks the candidate to show you how something works which seems like it would demonstrate their ability to understand a topic, how they approach it, and importantly whether they can communicate what they know in a way that other people can understand.
I'm more interested in whether a candidate can write a decent bug report and form a coherent patch than how many puzzle solutions they've memorized to slip through my filters.
That is, they may treat you like a rockstar when they sign you, but then if you complain because you got a crappy Dell laptop that was a hand-me-down from a salesman who couldn't sell, that your build process takes 40 minutes and that this could cut that to 20 minutes. Of course they have the "sick system" excuse that there is no money, but you get blamed for complaining about the 40 minute build, and then you get blamed that it takes forever to do anything because you are always waiting for that 40 minute build to finish and the system is such a house of cards that if you work on anything else in parallel whatever you would have learned from that 40 minute build is invalidated.
What exactly do you want that you don't already have?
Every company I've interviewed at definitely seemed to make developer happiness a priority (especially the smallest ones). Even the smallest of startups (ie. one guy with a tiny seed round) understand that buying your developers great gear is an essential investment.
Different problems happen in companies that are not "startups". For instance, a startup won't have a 40 minute build process because it hasn't accumulated enough code. A company that has been around 10 years however and hasn't deleted anything is a different story...
Then we put them, a bunch of mostly type-A software developers, to work doing the same boring crap everyone else was. Not surprising, after their one year commitment (for all the extra perks they received, like a housing allowance) was up, many of them jumped ship immediately.
Besides just hiring good people, companies need to put real effort into growing them as effective contributors. However, that process gets a lot of discussion but little real effort is put into it.
In the UK the contract rates are typically significantly higher, so even if the person isn't offered a job at the end they aren't bothered. Everyone wins
In my particular case, the company I did opt for ended up being a bust, to the point that I resigned without having another offer in place. It ended up working well—I'm in a job now with tons of job security, good pay, a good work environment, and interesting work—but I ended up in the exact situation I was hoping to avoid when I rejected the CtH position.
I hear this position often when people talking about contract-to-perm, and I just don't see it. If you're good at what you do, and people enjoy working with you, why wouldn't you take a contract to perm position? The only reasons I can see are that you are afraid you don't get the job (well, then you wouldn't have been a good fit if they hired you) or you end up not liking the team/company as much (same as if they had hired you.)
So, why wouldn't you? And I don't buy the 'I have a good job' line, because you wouldn't even be entertaining offers if you weren't interested in leaving your current role.
Or, you rubbed the wrong, politically-connected person the wrong way. Yes, this is always a danger, but a contract employee is more expendable and easier to get rid of.
Or, they weren't really looking for a permanent hire and this was just a means of tricking someone into a contract who wouldn't otherwise take one.
Or, the company hits a rough patch and decides to freeze hiring to avoid layoffs.
There are a whole host of things that could go wrong in a contract situation that are either tempered or nonexistent when you are permanent.
Not in my field, but perhaps the CtH position doesn't actually want quite what I am good at, and we didn't figure that out beforehand.
> Why are you interviewing if you're employed at a good company?
Because I've been there a long time and I'm bored, or because my boss isn't bad but isn't the best, or because opportunities for advancement seem slim, or because I want a shorter commute, or because the prospective company has a great reputation, or because I think I could work with better engineers somewhere else, or because...
I wouldn't take the chance at a CtH job because there's a significant probability that I end up totally out of a job, rather than employed in an OK (but not great) job.
Of course it's possible that I would accept a FT position and it would turn out just as poorly as a CtH that didn't work out; I would just expect that the probability of it not working out is much higher in a CtH scenario because the company I'm working for has invested far less in me than one that has hired me full time.
However you know what you are getting so you could argue it's worth it...
Normally you need some experience, ideally in a specific skill, but shorter contracts may be available for people with less experience. If you are interested, it's best to find a decent recruiter who knows what they are talking about (easy to tell from a quick phone call).
For whom?
It's bad for the worker because you're going to work for a company that is admitting by its behavior that it has little confidence in its abiliity to assess talent. Where else is it weak? (My experience shows that those who can't assess talent usually can't do a whole lot of other important things, too.)
It's bad for the company because it limits its pool of candidates. Good luck recruiting desirable high achievers by telling them you can't tell if they're good enough to hire.
Assessing talent and determining long term resource requirements is like anything else in I.T.: it's fundamental and must be mastered to become any good. Contract to perm is a red flag that something is wrong.
You seem to think that willingness to admit that assessing fit for a particular job is unreliable without actually experience of the candidate working in the real conditions is correlated with the relative degree to which the company is unable to assess such skills relative other companies (rather than being driven by the degree to which the company is aware of difficulties affecting all companies hiring for similar jobs, or differences in the features of the specific job which make assessment difficult.)
There is no justification that I can see for this assumption.
I have hired hundreds of programmers and have never been disappointed. The recruitment process lasted as long as it had to to know that it was a good fit. Contract to perm was never needed.
Alternatively, its a predictable scientific process which is more accurate when actual time in seat is devoted to the assessment, and good companies know this and bad companies just delude themselves into settling with a less reliable process.
Or, more accurately, assessing fit for job is a task that varies by the specific job (including, among other things, how much room the role, size, and mission of the organization leaves for customizing the job around the individual) for which suitability is being assessed, and good companies (in this regard, at least are those that understand the mechanisms which are most appropriate for assessing fit for the jobs they actually are hiring for and choose the right methods (which may or may not include contract-to-perm for particular jobs), and bad companies are those that don't (which may include picking contract-to-perm for jobs where its not the best choice -- or not choosing it for jobs where it is.)
I do agree with you, this doesn't work, but that's because once the candidate is of the quality that they can easily grab contracts, they have no incentive to come work for you as a permie.
That said, only high end contractors are really that more expensive. You need to remember, your employees are likely getting sick days, holidays, and pensions. You need to pay employers NI and probably offer other benefits. In addition to this after a while getting rid of them becomes difficult.
When you think of it like that, a often contractor isn't a bad deal. It's a shame a lot of companies feel like they need ownership of their staff.
Typically here in London these people are already contracting, so all you need to do is convert the best ones.
If it doesn't work out, it looks worse on your CV to be a "failed" contractor scurrying back to a permie role, than if you'd been a permie who'd quit after a few months.
Maybe some people do go down this route - but a company whose policy is to hire this way, won't be able to scale it.
They brought me in for a total of ~20 hours of interview, and in that time I discussed work-related topics with every single C-level. I went through a standard interview with questions, and then a longer skill test that included every current engineer at the company.
For the last few hours they paired me up with a current engineer to work on some database issues they'd been having.
I've been very happily employed here now for around 7 months, and after adding this company to my resume I've had quite a lot of offers on LinkedIn. I feel nothing other than "Oh, this company has offered me an interview, that's really cool!" towards these offers.
I think it's one of the best things in the world to have that feeling, and I wish everyone who was hired in an engineer position felt the same way.
I don't understand what she's talking about here. Every time I see a job opening, the software hiring manager and the team peers know there's a pain point that needs to be solved. E.g. "We need to an embedded C programmer to make this firmware talk to our USB protocol". "We need a developer to port the Java code to Go." "We need to expose our mainframe transactions as a REST API", etc.
Look at the previous "Who's Hiring" thread. Do those posters look like they post the job slots without a clear intent?
https://news.ycombinator.com/item?id=10152809
>just automatically do, and we don’t think about, “Well, what questions are we trying to answer by asking a candidate to solve a problem?” Are we dinging people for trivia questions, for not remembering, “Oh, I need this third option flag, or an obscure method from a core library.”
I'm not sure where her experience with whiteboarding comes from. In my experience, companies use whiteboarding to sketch out algorithmic thinking. Whiteboarding is the opposite of reciting trivia such as the "3rd parameter to an obscure library function" Interviewers don't care about perfect syntax or missing semicolons.
>Instead, I really want to focus on questions that are asking about decisions that they’ve made, what choices have they made, and what choices would they make again in the future? Are they reflective about mistakes that they’ve made? Are candidates looking for opportunities to improve, and how do they actually go about it? Do they make plans for themselves, like how they would improve a certain skill set, whether that be a technical skill set or a more soft skill set, for example, management, or project shepherding for example. Those are the kinds of questions that I think really get you at the heart of not necessarily what somebody knows, but what they’re capable of.
Those questions are fine but it's wrong to prioritize them over concrete programming questions. Companies are trying to evaluate if the candidate can actually program and those "Emotional Quotient" questions she favors are easy to bs about. Instead, companies need to assess real programming skills and they can start with FizzBuzz and then move on to more comprehensive evaluations -- whether it's the take home programming problem or the onsite whiteboard algorithms discussion.
Where I'm being told, "Hire more developers! We need more developers!" I respond with, "I don't need any more on my team right now. We can't handle more with our processes and the head count on our other teams."
That's not important though. To the people I was working with, more developers meant you could do more work, even if we didn't have a plan for what that work would be yet. Incredibly frustrating since more devs could actually contribute to gumming up the works if not brought in appropriately.
I also have seen whiteboard coding interviews where interviewers are inclined to ding people for not remembering small details, so it's a legitimate concern.
Not necessarily here, but there are also a lot of buzzword soup and horribly vague reqs floating around out there. Just browse Dice or Indeed.
> I'm not sure where her experience with whiteboarding comes from. In my experience, companies use whiteboarding to sketch out algorithmic thinking. Whiteboarding is the opposite of reciting trivia such as the "3rd parameter to an obscure library function" Interviewers don't care about perfect syntax or missing semicolons.
My experience has been that significant numbers of interviewers want working code on the whiteboard, Amazon in particular. If whiteboard interviews were what you describe, I and large numbers of other people would not hate them so much.
It isn't just whiteboard interviews, either. I have had phone screens where my Linux programming skills were judged based on how well I had memorized POSIX manpages. Maybe I have just had bad luck in drawing interviews, but then maybe you have had good luck. I don't know. I can dig up anecdotes from around the web to support both our experiences.
The point of a permanent software development teammember is to add to the capabilities of your engineering staff. What is it you want them to add?
What I think she means by 'hiring with intent' is this - identify what you need to add to the team to create the kind of team you need. That could mean someone with more follow-through, or someone who will explore and innovate more. Someone with big-picture systemic thinking skills, or someone detail-oriented who will sweat the little things.
My examples are being interpreted too literally. They are not tasks that must literally go into the job description. Instead, they are bullet points of internal thinking as to what the additional programmer will do. He/she will convert the Java code; and can also refactor the batch data load, and add 2-factor to customer logins, etc. A laundry list of "things to do" gets built up and it justifies another full-time employee. However, sometimes one item such as "make website scale to 1000x users" can be so large in scope that it will require many FTEs just to work on that one objective.
I've just not seen a situation where a team would hire a new developer and he/she just sit there for days and weeks twiddling their thumbs because the company doesn't know what to do with them. Yes, the new programmer may do nothing because of bureaucracy (IT hasn't created userids yet, repository access not granted, etc). Other than that, the hiring manager usually has a an idea of how the new programmer is supposed to add value.
It's rare that I see a req that interests me, because most are extremely situational and based on very specific tasks. Again, I recognize these sorts of jobs will always exist in established companies, and I thank the writer for making that clear, but a list of 8 technologies (who am I kidding, it is usually 15 with 10 more 'nice to haves') from a start up that will probably be pivoting in 3 months anyway? Doesn't make a lot of sense to me. "Come work for us and absolutely stall your career as you grind out tasks that you mastered 3 years ago" is not an appealing sales pitch, and not one that will bring in the talent you need to respond to new challenges.
I'm not saying you meant any of that in your post (you might of, but I'm not sure), your post was just a convenient one to respond to.
I understand. There are 2 sides on how to describe the existence of a job: (1) for the executive manager approving of the requisition and (2) for the candidates
In other words, when the hiring manager goes to the vice president asking for another FTE, the exec is going to ask the justification. The hiring manager could say, "well, we can teach him the latest cutting edge functional programming techniques, and also let him build up some AWS architecture skills which will boost his resume and make him more valuable to other companies, and he can play in our foosball ping pong table area, and eat our catered food, and take 4 weeks of vacation." Obviously, the vice president would think those are silly reasons for paying someone. Instead, he needs to know what value-added tasks will justify the job position.
(1) hiring manager perspective: the real work that needs to be done is highlighted, the employee's benefits are implied
(2) candidate perspective: the employee's benefits are highlighted, the "real work that needs to be done" is implied
The thread was about Kerri Miller who was talking about job positions from the company's perspective instead of the employee's and that's what guided my examples.
Behind every job ad that says "change the world and come work for us at our beautiful campus!" from YC, Google, or Apple, etc is a list of tasks that need to get done.
I'm glad this one has the soundcloud link. Some of the others I've enjoyed don't and it's much easier to just listen.
Why? Because physics is about making a conceptual model. i.e. telling a story about a problem. And then solving it.
For interviewing, communications skills are a must. People will be working in a team, and must be able to explain themselves. And get along with others.
Another thing rarely checked for is the flip side... understanding explanations. If you have to explain something four times to a candidate, you'll probably have to explain things 1000 times on the job.
That's pretty counter-intuitive. Can you give a source?
> The study by researchers at Harvard University and the University of Virginia (UVA) found the best predictors of success in college science courses to be high school classes that foster mathematical fluency, value depth over breadth, and feature certain types of laboratory work.
The study appears to be https://www.cfa.harvard.edu/smg/ficss/research/articles/Scie... . Table 4, model D, page 14 shows that science grades, followed closely by math grades, are much better than the English grade at predicting "College Science Grade". (It does not specifically pull out physics.)
Likewise, the best predictor of on job performance is on job performance.
If this isn't a recent grad, interviewing on the resume will tell you more any proxy for on job performance. Sure, people may be seeking to step up from a drudgery type job, but I think that usually comes out (citation needed, I admit). "I wanted to replace the terrible X with the wonderful Y, but I was over ruled".
"question about why this was rational".
"rational explanation ensues"
"want a job here?"
I admit it ain't perfect, people lie on resumes and take credit for other people's work, but we all know the whiteboard tests and other things are just voodoo. Google's hiring data proves it out - only one person in their company was able to give better than chance ratings based on interviews.
And, who really cares about resume inflation? I'm scrupulously honest in my resume, but if someone can describe all the trade offs, costs, and benefits of an architecture from a previous job, and answer speculative questions about it ("what would you change if you needed to scale X"), then heck, they can probably do the job. If I understand engineering and am an active contributor, what more do you want? No, "doesn't get nervous during a highly artificial situation of whiteboarding" is not a rational want, since it will not add 1 dollar to your bottom line.
What frustrates me is that, as an interviewer, you can catch out all but the pathological liars if you know what you are doing. The problem is, very few interviewers in this industry know what they are doing. From the transcript intro: "We typically don’t get trained on interviewing". That is a glaring, fundamental problem.
Worse are those who treat interviewing like a distraction from their "real" job. Those people do not belong anywhere near an interview table.
<sigh> And how do you predict university physics scores for people who've never been to university?
It's pretty obvious that for someone already doing a job, that the best predictor of future performance is past performance. So your comment is unhelpful and condescending.
Her suggestion for "explain something to me" makes me a little nervous; I'd be worried that the interviewee would have a hard time coming up with a good topic on the spot. But last I interview, one person asked me to pull up one of my open source projects and explain the code to them. That struck me as a very fair test.
[1] Described here: https://www.quora.com/What-are-the-best-programming-intervie...
As we move more and more towards working in a distributed fashion, using tools that don't require us to even meet someone in person for months at a time, it's a little strange that there's so much emphasis placed on in-person interviews (which is inherently a fairly stressful process). How someone communicates in person is fine, but that's sometimes only part of the picture.
On top of that, some people are much more comfortable communicating over the internet, and weighing in-person interviews so heavily ends up discounting their real talents. Ideally you find people have skills with both asynchronous and synchronous communication... but you need to be keeping an eye out for both qualities for that.
It sounds like a great way to test for communication skills while also learning something about the person's interests. Plus it could make the process a lot more fun for the interviewer.
However I expect it would depend as much on the personality of the interviewer as on the interviewee. And at least in the US, it could have legal implications if you decide not to hire someone because they didn't have anything "interesting" to teach you.
So this exercise would be artificial and forced. And, yes, for sure, I can put on a show, but it's not me and I'd be doing it because that the game or expectation and further, I think it's rather beside the point.
On the other hand, I think the model you suggest would also be problematic because on a project of any size you need to glue back together a lot of very thin slices. It adds the problem that no one person is going to be amazing at all those things, so a fair bit of important work will get done poorly.
I think the best model is instead cross-functional, closely collaborating teams where the decision about who does what is extremely fluid. That gets the benefit of lots of people doing (and understanding) different sorts of work. But it lets key work be done by people with strong expertise.
Real-world examples of this include IDEO's pursuit of "T-shaped" people:
http://chiefexecutive.net/ideo-ceo-tim-brown-t-shaped-stars-...
And Atomic Object's use of "poly-skilled, co-located teams of generalists":
http://spin.atomicobject.com/2011/09/16/poly-skilled-teams-d...