We analyzed thousands of interviews on everything from language to code style
blog.interviewing.io
blog.interviewing.io
My interpretation of this is that interviewees who can communicate clearly about code (whether they wrote it or not) correlate with high technical ability. Does this suggest that rather than having the interviewee write code on the spot, one could give them some new code they've never seen before and ask them to reason about it aloud for 30 minutes, then gauge their technical ability based on their ability to communicate clearly about the code?
In other words, could you replace live-coding with "here's some code, tell me about it"?
Have them delve through it. Reason aloud, ask questions about it. See how they gain context, and how they troubleshoot. The biggest down side was they sometimes need to reference external projects, or documentation. I would often have to search on my work laptop and then let them should surf the docs.
That sounds like a trick question. I've done a lot of code optimization. Here are a few generalizations I've drawn:
(a) You rarely know what part of a program is the bottleneck until you profile it on an appropriate workload.
(b) You often don't know why it's slow until you dig into performance metrics like cache-miss ratios, branch-mispredicts, etc. Often at the disassembly level.
Now if you responded in the form "Of course in the real world we don't look at code until after a profiler shows the need. I see X which is suspicious, let me look deeper". In this case the comment is perfect: it shows you trust, but know to verify that trust. It also has the side effect of giving you time to think.
"It doesn't make sense to hire smart people and tell them what to do; we hire smart people so they can tell us what to do." - S. Jobs
Refusing to do something because it hasn't been profiled is not analysis, it is refusing to do analysis. You have been given a problem and been asked to analyze it. That refusal makes you not a team player.
Once again, this is an interview. Noting that this scenario (a single function that is known to be too slow for reasons that can be fixed in 10 minutes) never happens in the real world is good - it shows you understand that real world analysis includes elements not included here. Stating you won't do this because you don't have profile data is not being a team player.
I wasn't saying anything about how one ought to handle such a situation during an interview.
I totally agree that a response akin to "Your question makes wrong assumptions. Let's move on to the next question." would be completely inappropriate for an interview, or if the same issue arose in a normal work situation.
So they're getting rejected for a reason other than technical skill.
[1] "Okay, we're going to talk like I've already fixed problems A, B, and C, but not X, Y, and Z, because we obviously don't have time for a full interaction that is open-ended enough to allow for all of those to be an issue, and you're being judged on how quickly you focus on and solve the X/Y/Z."
Anyone can repeat someone on StackOverflow who says that function X really is faster than function Y. That's hardly an accomplishment. It's like asking someone to perform a basic non-trivial calculation out of his head -- why bother?
I don't know how what was said implies this in anyway.
This is one of the biggest problems with interviews. Every interviewer has some dorky idea that x means y is a terrible developer while another interviewer thinks x means y is a great developer.
"I want an opportunity to learn" - Oh great. We want someone whose interested in learning more and isn't cocky.
"I want an opportunity to learn" - Oh no, that just tells me you don't know much and aren't interested in contributing
The chance that the interviewer expects some weird thing like that is random, so there's no way to know whether you're getting it "right" or not. Select the wrong answer and you get viewed as an ignoramus or a smartass. Just have to take the gamble you like the most and see if it aligns with the interviewer's preferences.
Once I was asked what kind of software publications I like to read. As a habitual HN user, this should've been the easiest question ever. However, when asked, I guess I was feeling strict that day and said "Well, I don't really read a lot of software-specific stuff, it tends to the more businessey side of things rather than being strictly software related", thinking of highly technical blogs like Lambda the Ultimate. The interviewer said "What about things like Joel on Software?" and I said "Yeah, I've read most of Joel's stuff, but I don't really consider it very 'softwarey'".
I could tell that cost me the interview, but what can you do? Someone else in a strict mood that day may've been delighted by such an answer, happy that I didn't hold delusions and could tell the difference between comedic rants about the business of managing projects and hiring developers and serious, borderline-academic publications that include equations and ponder theoretical dilemmas.
The best thing is just to take the whole process casually without getting over-committed or holding a grudge. False negatives cost companies much less than false positives and they're evaluating you in the context of their other applicants, who are also essentially random, so there's no way to really know anything. It's just about whomever seems least risky and most useful, and as an interviewee, there's no real way to know what that means from the POV of the interviewer, so it's hard to optimize your responses.
The real way to win interviews is to recognize them not as technical processes, but social processes. Admitting this makes engineers nervous, but it's important they understand that humans, not compilers, will be performing the evaluation, and they must learn to speak human to get satisfactory results.
I'd rather see a question like "Please modify this piece of code to add X functionality."
Basically, give them less than ten lines of code, ask them what it does, where are a couple bugs, ask what would you name the function, etc. Then we talk about how to improve it. I'd say, less than 20% of people I interview pass. It's actually amazingly low how many people can find a bug and communicate it.
Half the people don't even tell me what they are thinking. And no matter how many times I try to work with them, act like their buddy, or w.e. they just kind of shut down. They think in their head, don't work through the problem at all.
I'm left wondering why this would be a problem.
If they sat there for a bit thinking about the problem and then provided a solution, I'm sure that would be fine. But that's not the situation he was describing.
I don't have any source but from my experience mathematical/computer minded people are more likely to be introverted/talk less then arts people.
I think you are right about that, but that just means that if they want to be effective members of a team in an organization, they need to work on their communication skills. Ability to communicate is something you can learn to do better.
I know this for a fact. I used to teach public speaking, and students came out far better at it than they came in. The same is true for many other communication skills. See for instance Marshall Rosenberg's book Nonviolent Communication, which is great for interpersonal conflict situations.
In fact, I think a lot of introversion is at least in part due to poor communication skills that could be improved with training and practice.
As for programmers - same thing. If they are any good they are more than able to talk the talk.
Effective communication doesn't necessarily mean you talk a lot or are extroverted. It means that you say things that matter when they matter, and that is a crucial skill to have in almost any work environment.
In my work experience I don't remember ever having to do that. I'd look at code & investigate a problem independently, then talk to someone about it afterward. I think that should be considered at valid approach to the interview.
In other words, from my perspective, you're not actually in disagreement with their method. They weren't saying "narrate in real time as you're first having [thoughts]". Practically nobody does that ever, or is expected to do it, except in other practices like mindfulness or therapy and what have you. They were saying they ask them to understand it, explain it, and then talk about improving it. OP's problem was with the candidates whom couldn't understand the code, and weren't even willing to talk about the state of their understanding so that the interviewer might help them walk through things further...
(a) discussing an open problem with a colleague, and
(b) explaining what's going on in my mind as I'm attempting to solve the problem single-handed.
I'm very effective at (a), and it's a skill that I've used many times as a software developer and/or grad student.
But (b) is more typical in psychoanalysis sessions.
I often have to explain a problem to my coworker and vice versa. It's also really helpful when pair programming. Just curious about your experience.
There's no back and forth -- you're not actually trying to help them solve anything -- it's someone who already knows the answer judging how much of it I can reason out in half-an-hour. I can't take 5 minutes to think it through, I can't stop talking for any significant amount of time without being told to narrate what I'm thinking about, etc. That's nothing like a workplace situation, where they'd be expect to hold up half the conversation, be providing input and thoughts on the problem, and I could just tell them I'd be with them in 15/this afternoon/tomorrow/etc.
It would be more realistic if the interview didn't get to see the question before the interview, either. (And why I think design questions work better than coding questions -- you can do something they're not expecting in the design, ask more kinds of questions about the spec, etc which cause an honest back-and-forth.)
I've often had the pain of dealing with people who try to do it all in their head. Sure, one or two are good enough where it's not a problem. The rest of them? They keep getting confused and making mistakes. For things that are not very complex. If they had only written it out or drawn on paper/board, they would not get confused.
It's most obvious when I try explaining some code's algorithm to them. They'll begin with understanding, and then 70% of the way through they'll keep getting confused because they couldn't hold it all in their head.
I've often had to tell these people "Don't try to do it in your head!"
You don't have to just accept doing your mutual conversation exactly as he likes it.
I had a lot of success with asking him if I could think about the problem and we could schedule a meeting in half an hour or so. He was much happier with that result because I had thought through the problem and was now able to give him the back and forth he needed. I was also much happier because I was no longer expected to come up with insight on problems I had no time to chew on.
When I am asking someone else to be the rubber duck I strive to highlight any assumptions I am making at each step of defining the problem. If I don't do this the other person will usually struggle to keep up.
If that doesn't work, get him a rubber ducky [] and tell him to go talk to it if he's not willing to compromise on communication styles.
Maybe they are introverted or simply need to think before they talk. Around 50% of people are introverted (less in usa). Many people are like that and simultaneously quite skilled. Moreover, some environments punish errors, so people who worked/studied there tend to be conditioned to think before talking.
"And no matter how many times I try to work with them, act like their buddy, or w.e. they just kind of shut down."
You are not buddies, you are interviewer about to decide whether they get hired. Many people shutting down might mean that they are not comfortable juggling "buddy" social role and expectations and "serious job interview" social expectations simultaneously.
Very good point.
Oh I do love the interwebs.
So my bet is on error avoidance.
"Tell me what you are thinking about right now"
"What is jumping out at you for you to be stalling?"
"Did you see something?" --> "Why?" --> "You seem to be quiet and deep in thought, what are you thinking about?"
etc...
In a normal work environment where he or she is left to think freely the the same candidate might excel.
The whole point is to make them comfortable and that takes the highest priority. Being able to read the individual and react to them is key, not everyone is the same, hence why there are a multitude of responses, and the ones given are just a starting point.
I.e. if stuck at something, even if seeming trivial, it might pay (to go back or have a chat around it, even guide him/her a bit so as, etc) to get the candidate loosened up and feeling safe which should then yield better responses after.
There is no one-size-fits-all interview technique; if there was, it would be used all over the place. Good interviewing techniques are about playing the numbers, not leaving no programmer behind.
If the candidate would like some quiet time to think about the solution, I expect them to respond with "I think I'm forming a solution, just give me a couple of minutes to think about it before I present my case".
If they can't even do that, then I'm sorry, but they are no good to anyone. You can be the smartest and best developer in the world, but if you're completely incapable of representing yourself and your ideas; to have a dialog about your work, you are effectively worthless as an employee.
Some companies might have a place for that special someone, who you can lock in a room for a month and he will later emerge with an amazing new piece of code that will solve your problems, shielded from all the problems of the outside world, but companies where that is possible are very rare.
Also, being able to develop like that is very rare because in the real world, requirements change or are vague and need to be clarified, or your software has to interoperate or integrate with other software. There is simply no way to avoid regular communication and still produce something that works how it needs to. Problems where you can shut yourself away for a long period of time certainly exist, but in the grand scheme of software jobs, they're few and far between.
The point is that with this kind of make or break style of candidate vetting you're probably leaving a lot of great talent by the wayside.
Even if I solve the problem I'm walking out of that interview with a very negative opinion of your company.
I also don't think he was advocating nagging either, simply asking questions to get the candidate talking. If you "go dark" for 10 minutes, you're pretty much definitely stuck. Talking to the interviewer might get you unstuck. Staring at the whiteboard quietly probably will not.
This also isn't your day to day job so you have to keep in mind that these interviews are generally held to an hour length and you can't really give them 30 minutes for a single problem that wouldn't give you much insight into the person.
It also heavily depends on the culture of the company as well. So being able to communicate effectively and act under pressure is something that goes with pair-programming and code reviews.
To note, I have never had anyone ever leave an interview feeling bad about the company or our process, the interview is about the individual.
I still think it was pretty cool way of interviewing.
You're not bothering people while they're taking an expected amount of time to look at it. You're stopping a several minute silence after handing someone a basic loop and if statement.
For most positions we're talking about, this should not be a major problem. I found when doing something similar people either looked at me like I was crazy for giving them such a basic problem and after being reassured there was no catch answered simply and clearly, or just couldn't do anything. Followup questions about the code then went terribly (e.g. a JS contractor asking for £500+/day who didn't know what the difference between global and local variables was, why you should put var in front of things, etc).
These things are often good to weed out the incredible number of people who seem to have little to no coding ability at all for jobs where that's an obvious pre-requisite. I've also had people fail to solve a basic problem (roman numeral generator or reader) at home in whatever language they wanted, not even getting vaguely close. One person even sent in their broken version pasted into a word document.
I've had my own team members tell me they aren't engaged, don't speak up in meetings, etc. because they are an introvert. Like it's a condition we must accommodate. Bullshit.
Communication is a skill. We value clear communication in our company. If you join our company, you may not be an expert in communication, but you will grow that skill.
An interview is a different scenario, but still -- you need to be able to communicate with me. If you can't, we will pass on you as a candidate.
There is value in silence and limited communication.
Replace "communication" with "data modeling" or "project management" or "reading". There are plenty who don't necessarily possess those skills.
Should we expect that others shouldn't develop skills in those areas (assuming they are important for their role) in the name of "diversity"?
People should speak in meetings when they have something to say. Not just so they speak - that just waste everybody time.
The more important point is that if 80% of people shut down while they are talking to someone, maybe has more to do with that someone then 80% of population being uncommunicative.
So let's clarify -- "speak up in a meeting" means when a senior person doesn't provide input at a time that would provide better direction, identify issues earlier in a process, etc. If you're a senior engineer, I need your input to ensure we're doing the right thing(s).
The forum of a group meeting isn't always in the wheelhouse of someone who freezes up when speaking in front of others. It's understandable -- I used to be one of those people. It can be out of your comfort zone, and doing so feels super-risky as well as plain frightening. It is still just a skill to be learned, no different than building muscles through exercise.
In our group, we talk a lot about trust and support. Everything we do is in the spirit of making each other stronger. Everyone has strengths and weaknesses -- share your strengths with the group, and build your weaknesses from the strength of others. It sounds pie-in-the-sky, but as the head of my group, I make sure we take this seriously.
So we don't have 80% of our group shutting down -- nobody talks over the top of anyone else, and we don't allow it. We need the input of the best of those, and sometimes that involves speaking up. If you're on my team and that notion makes you nervous, I understand. But as I've always explained to my team, we're going to help you build skills, and communicating is one of them.
The fact that someone takes time to put together ideas before opening their mouth is not bad sign. Bad sign would be if they don't share result once they have done thinking.
Also, could you possibly just do an initial phone/Skype interview to save time in that case?
And their is an initial phone interview, but it is done by HR or our boss, neither of whom are technically proficient.
And "you haven't passed this interview" is less suspect than "I ended this interview early"
Indeed, it's amazing how quickly people reveal themselves in this way.
No whiteboards, no quiz shows, no grilling of any kind needed. Just: "tell me your story".
Those are independent clauses. Just because the individual is not talking out loud, should not be indicative that they aren't working through the problem. Personally, I am a very visual thinker and layout visual models in my head for problems with high (3-5d) dimensionality that would take longer to draw clearly in 2D space while explaining all the traversals I'm mentally making. I would then reduce a solution, and explain that, or I apologize for zoning out.
It became obvious to my office mates when I was working through and reducing a problem space, because I would "hang" on the thought. Sometimes I would be down, stuck in thought, for 15-30 seconds and according to them they could tell because my face loses emotion and my eyes flutter about.
As it happens most of my "aha!" code-related moments come when I'm either under the shower or when I'm washing the dishes. Very rarely did it happen for me to look at a piece code while in front of my computer and then immediately be stuck by what that code does or by some hidden bugs in it. It's all sort of mechanical, at least while I'm sitting at my desk. I'd say it would be very hard for people like me to reproduce that "aha!" moment during an interview, in front of some other people who expect me to have those moments of enlightenment right there and at that precise moment.
I'm a product manager and I suspect I would do pretty well in this type of interview unless the snippet is especially complicated and/or esoteric.
In that case you might be screening for people who can talk but not actually code or people who can code but might not be the best communicator.
Interestingly enough, if youre optimizing for specific outcomes, you may not need to screen out the latter.
Engineers who struggle to communicate but are productive can be extremely successful given the right environment - including having someone on the team who knows who to work with various personalities and is technical enough.
Just some rambling, but it's worth pointing out given the scarcity of engineering talent. Not everyone is going to be Paul Bucheit, ace product manager and engineer all wrapped into one.
From TFA and literally the thing that started this whole thread:
> Furthermore, no matter what, poor technical ability seems highly correlated with poor communication ability – regardless of language, it’s relatively rare for candidates to perform well technically but not effectively communicate what they’re doing (or vice versa).
This is after all an advertorial piece of data spelunking not a study.
Many are taking the article at face value, and are assuming there are no methodological flaws and that this study has been repeated enough times to invalidate any comments to contrary. It also didn't turn this topic into a "case closed" matter, as it's too big of an unsolved problem for that. They also just analyzed interviews, but did they monitor job performance after the interviews and for how long?
Additionally, differentiating between being a "good communicator" and a "good interviewee" is frequently overlooked. People too frequently rationalize their own interview approach as not the problem. Most candidates accept that they could improve and perform better at interviews, but a lot of people conducting the interviews don't think their process could do better.
You could even have HR do it. When they arrive, HR hands them the code and instructions. Then, they read it for 15 minutes, until you walk in.
You can act buddy all you want, but people aren't stupid. They know if they take a few extra seconds too long, you're already thinking "this guy sucks"
I like your idea though.
After hiring him, he was unable to even SSH to a box. As in he didn't know how to give us his private key, or how to configure his SSH client.
Maybe one guy takes interviews under other people's names then lets them mess up the work.
My brain simply froze during the live coding test. I looked like an idiot, even though the coding challenge was one I could have normally handled in my sleep.
My guess is that it was due to anxiety about doing well on the interview. But to this day I'm gun-shy regarding live coding tests.
People normally perform better by verbalizing the process than they would by sitting at their computer and performing it, since a lot of the concern in an interview can be whether you're getting your code/commands syntax-perfect or not.
The only way I can really see this going badly would be if the interviewer was only asking generalized questions that someone like a non-coder HN reader would be able to pick up from the zeitgeist without actually having ever implemented. Instead of asking about trends or fads, it's probably better to ask detailed questions about implementation processes.
If they really have internalized all the applicable concepts but can't express them in computer language, well, it shouldn't be hard for them to get up to speed on the syntactical fineries. Make sure you're testing for grokkiness of the core concepts needed, not how well they're following the trends or the news.
Personally I use a very small code test and put the rest of the energy into a detailed conversation.
The old saying "If you can't explain it simply, you don't understand it well enough" applies here I think. :)
Then for the next round of hiring, advance the employees up the education pyramid. The trainees become trainers, and the previous trainers mentor the new trainers by reviewing the newly trained employees.[1] It was one of the things I think we did we really well and that I am really proud of (even though the business went belly-up).
When training is done is this pattern, hopefully a virtuous cycle of education is established throughout your organization.
1: Note that this was at a quickly growing company, with recruits that had little to no relevant actual know-how to do what we did.
Based on other responses to your post, it's heartening to see that others do this kind of interview, so maybe in the future I might encounter it.
Some engineers spend a lot of effort trying to project a specific image of themselves at the expense of bringing up the morale of others in their team.
On a few occasions, I've seen an engineer tear apart the work of another engineer at a team meeting in order to make themselves look good and more senior in front of their boss - Often, the critique is delivered with such vigour that it sounds rehearsed. I've seen this behaviour in startups mostly.
When I'm nervous, I don't code well.
I don't know about other people, but for me personally, when you measure me under a high-stress situation, all you are measuring is just my level of nervousness.
In exams I don't get as nervous because I know that if I know the material then I will pass, if I don't then I won't. You can't bullshit your way past the professor/TA grading your exam.
But in interviews I know that people who are far better than me have failed[0], and that people less competent than me could get the job. So it's not my coding ability or my experience that's being measured here, it's something else. And I don't know whether I have that something else or not. Maybe I do, maybe I just don't.
[0] https://twitter.com/mxcl/status/608682016205344768?lang=en
The canned response I got was "We're sorry, but we can't currently recruit you. Please try again in half a year or so, maybe we will have some new opportunities." If you press on, you just get evasive answers.
I suppose if you went to a great school and then to some shitty interviews, you'd like the school more than the interviews, and vice versa. The difference however is that a teacher is only motivated by their altruism to help you, whereas a company has actual financial motivation to find good people during the interview.
I'm in the profession to write and debug computer programs, not debug people who like to keep me in the dark. This is frustrating.
Engineers who are capable can clearly explain their coding choices and their style is evident. If they've borrowed too liberally or had someone else write it for them, it becomes apparent pretty quickly because they can't clearly explain the code they supposedly wrote.
This is a fairly new approach for us, but so far, the candidates have appreciated it and the team reports that this approach gives them a good sense of the candidates ability.
I do give candidates the option of the take home vs. white boarding. So far (it's admittedly still a pretty small sample) they've all opted for the take home. However, we'll take a look at this approach as well.
I'm not aware of any solid data around take home tests. All I can give is yet more anecdotal data, which is that they take a lot of time on both ends. So much time, in fact, that I've had several interviews where the interviewer never looked at the test code I was required to send in. Whether they work or not is irrelevant if nobody ever looks at the code.
No thanks.
Someone says "I am a 9/10 in JS" I give them something that uses the animation frame api, the dom api, and a few more javascript APIs that I personally am not fully comfortable saying that I am a 9/10 on (been working with JS for over ten years) and see how they decipher what it is doing, what jumps out, and how they would review it / give feedback.
9/10 times they don't actually understand Javascript.
But, its quite possible for people who are not naturally able to communicate with a wide range of people to be able to communicate with a smaller subset (say people they are more comfortable with, or people working on similar problems to them).
The more interesting question is, could those people who communicate less broadly, in a situation where they do communicate well, perform technically at the same level (or better) as people who are more natural communicators?
Another way to say this is, evaluating interviews is fairly prone to bias towards people who interview well. As someone who is hiring I'd be much more enthused about their results if it correlated to job performance, not hiring results. Its almost gospel at this point that there are people in software who interview well who do not make good hires, but the methodology of this article doesn't talk about this at all.
All told, nothing about this research is persuasive to me on my opinion of "interviews are largely worthless in determining who will be a good hire".
The unsuitability should be extremely obvious, no "spot the missing semicolon". Maybe it has a severe performance problem, maybe an obvious security flaw, maybe a logic error. The point is to prompt a discussion of better alternative ways to do what the code is trying to do.
One of the biggest keys to doing well on technical interviews is to completely separate the problem solving from the coding. The strongest interviewers will discuss the problem and solve it at an abstract level using diagrams. Once satisfied with the solution, they'll code the entire thing making few mistakes.
I think this is what drives most of those metrics. Strong interviewers submit code later, and have a higher chance of it being correct because they take the time to problem solve upfront. Their thought process seems more clear because there isn't the iteration of "this should work, let me code it, oh no wait, that's wrong, let me erase this now..."
I think Einstein's the one that said: "If I had an hour to solve a problem I'd spend 55 minutes thinking about the problem and 5 minutes thinking about solutions."
"Fix" "Real fix" "Fix the fix because of fix"
Then you know you don't want those on your team. It really bites on tight deadlines when you have to put something to production but poor dude needs to push that really, really last fix.
I agree. In general I'm mediocre at coding interviews, but I do best when I have a chance to whiteboard and draw the problem. On the other hand I do absolutely terribly on phone screens with a shared document and no where for the interviewer to see my drawing.
The "average" is too sensitive to outliers and should not be used for such a comparison...
[Edit] Being bored I calculated the Kolmogorov-Smirnov statistic based on the chart. It is between 10%-10.5%. The number of defined funtions seems to be a significant but weak indicator.
What's wrong with this, I think, is that a (journalistic) title should give an ultra-condensed summary of the main point of the article. This title suggests that the authors gathered a lot of data but didn't find much.
(I find myself quite intrigued by clickbaity titles somehow, sorry for that.)
Even more galling when you have a healthy GitHub portfolio that they refuse to even look at in favor of a quiz (this has happened recently).
It gives a common baseline to judge. Each candidate does the same thing and we have a good idea of what we are looking for, the rest of it tells us how you think, how you approach your work, organize your work, and best of all? You can compare that to how others do it.
Now, not to say that's everyone, that is what we use it for.
---
Before, as an Engineer / Manager, I hated doing "live coding" tests when it wasn't relevant. For example doing "algorithms" or "palindrome" or "sliding window dns" or "O(n)" examples when you're doing front-end or a management position screams to me that the people doing the interviewing don't know what they actually want.
Instead quizzes or live coding that are relevant like "tell me how to access all the elements in this particular element and traverse the children to apply some styling" is much more relevant and will show me the thought process, their ability to retain information, and their recall. It also shows communication ability when they get stuck and ask for help or use me as a sounding board.
It's not always about your implementation, but how you handle the situation and communicate
And the thought that quizzes provide a fair point of comparison comes across to me as putting process ahead of substance. Interviewing isn't meant to be fair to everyone - only one person gets the job, after all - so it's not like handing out cookies and stickers in middle school. It's meant to see, in part, whether the person is capable of generating working code. If you have a person who can provide samples to prove it, requiring an artificial quiz really is a slap in the face to a lot of good candidates.
What's important to us is the person, their ability to communicate, learn, interests, then their skill set and what we have is a good fit for them and us.
Focusing on a singular part of the interview without taking a step back and reviewing the whole isn't too beneficial.
That all said, the correlation between great whiteboard / coding tricks and them being a successful part of a team isn't so great. It's not their job to code short programs under pressure of being watched. It's a proxy. There are better ways.
Yep. Articulate attention is the name of the game (where "communication ability" sounds a little nebulous).
If you can't organize your thoughts, bring them to the forefront of your attention, name them, you're likely bad at handling abstractions. And abstractions are at the core of "technical ability" -- the ability to name things, find the appropriate abstraction boundaries, chisel structure out of chaos.
Articulate speech is the greatest human invention for a reason.
Testing for that (plus conscientiousness -- can you pay attention to details and get shit done?) during interviews makes perfect sense.
- liked the person
- rated the questions 3 or 4 stars
- gave the interviewer 3 or 4 stars for being helpful
Do the trends still hold?
How are those trends compared to only looking at interviews with:
- disliked the person
- rated the questions 1 or 2 stars
- gave the interviewer 1 or 2 stars for being helpful
Looks to me that interview length is correlated with success rate. If your interviewer stops before 60 minutes, there's a bias towards successful interviews. It seems like the interviews that end up being "no"s tend to get hard-stopped right at the 1-hour mark.
But that's reversed. It is in fact fairly difficult for a high-level language programmer to pick up C++, and facility with C++ (or at least C) is a common, accepted goal for C++ hiring shops. A C++ shop that hired candidates without regard for their aptitude at C++ would have real problems.
Interesting that this effect does not show up for Java programmers.
Why isn't it the same in Java? I'm not sure, perhaps its because Java has less gotchas as a language (certainly a lot less undefined behaviour and weird memory-related gotchas, no templates, no multiple inheritance etc) and C++ has this "its a difficult language" prestige which Java doesn't have.
Edit - it looks way better if I disable Open Sans. It might just be a font issue with Chrome or Windows.
Clicking on the graph to go to plot.ly and viewing its "data" tab, it looks like there's a blank X value for:
text y x
0 bucketed_success_rate: -0.02<br>pct: 0.1<br>`Interviewer Would Hire`: False 0.0987654320987654Maybe that is a tad bit too harsh, but surely the use of "big difference" and of "significant" seems like not being justified by the actual data:
>On average, successful interviews had final interview code that was on average 2045 characters long, whereas unsuccessful ones were, on average, 1760 characters long. That’s a big difference! This finding is statistically significant and probably not very surprising.
An average of 1760 vs an average of 2045 indicates a general average-average of around 1900 lines, so that would be 1900+/- 7%, and anyway the difference in ranges is so little that anything could cause it.
To have more or less 200 characters, merely calling variables a, b, c, etc. vs. FirstUserChoice, DefaultArrayIndexingField, you know what I mean, would be enough.
Same goes for:
>On average, successful candidates’ code ran successfully (didn’t result in errors) 64% of the time, whereas unsuccessful candidates’ attempts to compile code ran successfully 60% of the time, and this difference was indeed significant.
As I see it 60% or 64% as an average are almost exactly the same number, and bear very little significance. Maybe it is just me missing some sensibility ...