On Interviewing Software Engineers
zdfs.com
zdfs.com
Good programmers really enjoy talking about the details of the technology and how they solved challenging problems. You'll be able to see them geek out a little bit and see that they really enjoyed getting to do it.
Outside of baseline, core competence, that is the litmus test for hiring good programmers because it shows level of interest. Level of interest leads to continuous learning, which is essential to this field. Unless you have some ego-driven reason why you want to believe that YOUR stuff is too complicated for them to learn of course...but that's usually more of a reflection on you than them.
I realized this about 5 years ago and I've not made a single bad hire with the approach since. 100% success rate at this time.
I've had this at several companies. I think I've always gotten past this litmus test in one of the interviews, be it on the phone, via Skype, or in person. The problem, for me personally, is the inevitable followup interview consisting of "go to the board and scribble out a function that does X", and then of course "after you're done, I'll add important details and I expect you to be able to modify the function to accommodate as if you were using a text editor".
I admit it, I fail at the "get in front of the board" problems. I'm downright terrible at them. I have no problem with the conversational interviews. I'm not bad at presenting tech to my peers at meetings and book clubs, but in an interview environment I'll freeze up and, at best, come up with a very inefficient implementation. I hate it and it has probably cost me opportunities at places where the rest of the interview felt like it went great.
Yes I know what the Big O is, I know what the Big O of the optimal implementation would be. I can talk all about hashes, final, static, or whatever you think you'll trip me up on. But when I'm up there writing code, every little mistake psyches me out more and more, especially I can't just hit "backspace" to fix it.
At my previous job, after the phone interview they sent me a programming project to implement in a day or two. After submitting it, they had me go to an interview and talk about the solution with a group of engineers, followed by a short conversational interview. That project was my first time writing ruby and I got the position and have great recommendations since leaving the place.
Makes me wish that recommendations held more sway, or that more companies would do more of this "here's a task, send us the solution when you're done with it" type of thing. Dishonest candidates would easily be exposed when they're forced to discuss the solution in depth.
Of course, upon walking out of the building, all the answers magically pop into my head...
The result: the correlation between passion and competence is not 1.
Initially I wanted to give them the benefit of the doubt and hire primarily on talk alone, however I found that some people are passionate and interested yet can't code very well. After having to fire folks because of it, I decided to actually make them write code during the interview. Sure enough, I discovered a few candidates who could talk the talk, but when faced with an actual problem* couldn't actually solve it.
* When I give interview problems I like to actually give problems that I have faced in my job.
I shouldn't have represented my original comment as the only thing necessary. Getting a mainframe guy to geek out on me for an Android position is probably not going to be all that beneficial.
I also don't find testing for actual coding skills to be a problem as most candidates don't make it to the interview if they don't have some sort of portfolio of code samples.
This is a valid point. A good work/life balance is critical. But to counter it, don't you want to look at candidates who _want_ to get hired, so they've spent a little extra time writing something for potential employers to look at?
A developer who gives up at "I don't have code samples" isn't someone I would want to hire.
Would you hire a graphic artist, a writer, a painter, etc. without previewing their work first.
Prove they are generally smart, do a work sample test representing the work they'd actually be doing, for a kicker test their integrity. You'll have > 50% wins.
http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%...
I don't really enjoy talking about anything and usually do so as a necessity. I guess I don't fit in with the extrovert engineers people are looking for. I also contend that not being able to talk well about how you did something doesn't mean that you aren't good at it. Talking about doing something you are good at is a different skill. Being be able to communicate is important sure. Is it the number one thing you're looking for? And candidates have to be happy about the ordeal too?
Code is easy, being able to communicate is not. We engineers love to loathe non-tech people who talk too much while producing nothing tangible. Even more though, effective engineers loathe their peers that don't talk enough.
Bad programmers also enjoy talking about that. They can make it sound like it was an awesome project.
You can't tell until you actually look at their code and realize it's all a spaghetti mess.
I've found the problem is sometimes missing good candidates because that particular one found you intimidating in some way. You can try to mitigate this some by having multiple interviewers to try to find someone on your team they can relate to. But really even this tends to work in your favor, because if they don't click with anyone on your team they probably won't do well even if they are good.
Tell me about a project, any project large or small, that was 100% you. You thought of it, you coded it, you wrote the tests, you documented it. The only other requirement is that the project has to have at least 10 users who installed, used, and continue to use the project without any help from you.
Lots of people don't have an answer. Some you need to coax a little, because a small project is fine. For example one that I did close to 30 years ago was move/copy utilities that used regexp. You could do
move =.fortran =.f
and it did what you wanted (first = is .* and second is $1 and they could repeat).
Anyone with basic C skills could have done this (or perl, I think someone else did a perl version and that won). So people would think a project like that doesn't count, but it does. Small is fine, you just had to do the whole thing and people needed to be able to use it without asking for help.
This covers a lot of ground and my experience is if you get lucky enough to find someone who has done projects like this, and there is at least some overlap with what we are doing, yeah, we got lucky.
* Talking about something you've done? You're selecting for marketing, not programming.
* Letters of recommendation? You're selecting for ability to network, not programming.
* Academic qualifications? You're selecting for persistence, not programming.
* Offline tests? OK right now because they're a minority, but I guarantee if they became the standard, you'd get an astounding amount of cheating.
I'm not a marketer and I really resent how in academia to have any chance you have to "sell sell sell". I appreciate anyone who gives me a chance to just objectively demonstrate my abilities, whether I do well or whether I do poorly.
EXPLAINING something you've done.
In an actual interview, when they are talking about something they've done and explaining it, the interviewer will be able to ask more questions and hopefully (as an interviewer) have enough of a technical background to separate the BS from the skills.
If two chemists are talking to each other about chemistry, one's not going to be able to easily just make stuff up that the other blindly accepts.
You have to do this everywhere. There are limited resources. You are competing for them with others. You need to convince a gatekeeper that you deserve those resources. In academia, you are competing for grant money, lab space, etc. on the basis that your work is of more import than that of someone else. In enterprise software sales, you are attempting to convince a customer that your product is a better solution to their business problem than of some competitor's. In an interview, you need to demonstrate that you can accomplish a function better than one of your peers who is also vying for the job, and probably has an almost identical skills profile to you.
I've had it happen twice, and man it ain't fun. The second was like an out-of-body experience. I forgot the question, I forgot my train of thought, I knew I wasn't responding, I knew that I knew that the interview knew that I was just sitting there, saying "ummmmmmmmmm" for about 147 seconds now. Yikes! Yet in day to day, these would have been easy questions. And TBH I didn't even want to job at that point.
It's critical that you as an interviewee learn to monitor and CONTROL your physiological response to stress. And I've learned that as an interviewer, I try to pay close attention to physical stress responses from the candidate -- flushed cheeks, signs of sweat, sudden lock-up -- when I see these signs, I back off and let them come back to normal first.
See, most interviews are along the lines of: Q: Are you smart and good looking? A: Yes!
This is why I don't really intervew all. When I take the "best" ranked applicants I give them actual work; small slices from our actual code base. I provide them a complete, functional dev environment with one restrciton: cannot commit to git. I can see their work directly, I work with them directly and get real-time data on what it's like to work with them, how they handle a project, how they communicate blockages, how they figure out our existing systems.
In four to eight hours of work I know who is the right fit for our team. My cost to find and filter is very very low and rather than waste 2-4 hours per interview for 10 candidates I pay for 16-24 hours of work for 2 or 3 candidates. The hard costs are the same, it consumes less of my (or my engineers) time and the candidate fit is very good.
See, most interviews are along the lines of: Q: Are you smart and good looking? A: Yes!
But a lot of these have "Communication Skills: 9/10" on them. Whether this is added by the candidate or the contracting company, I don't know.
- Admission that arbitrary coding challenges that are unrealistic to the applied job are ridiculous
- Recognition that you need to understand the engineer and approach things from their point of view, not from your own selfish view as the interviewer
- Urging that being polite and humble is important. Even if you don't like the candidate, there is no need to attack or diminish them.
One question I have: What is wrong with copying and pasting solutions to issues that have been solved before by others? I tend to do this all the time myself ( albeit I often end up rewriting the solutions to tailor them more to the problem )
Overall decent suggestions for interviewing.
However, I have a lot of respect for companies that do interviewing well (testing skill and experience in a less contrived and awkward manner), and the ones that have made the process as painless as possible tend to be well-run companies.
You can learn a lot from the process with each company, especially at an on-site. Dick-swinging contests definitely happen, which to me means that the company values rock star coders over programmers who are skilled, pleasant to work with and good at communication (and moves to the "won't accept offer" pile).
Sounds like a bad excuse to me.
If you get nervous on simple coding questions, how will you perform on your daily work?
Unless you're applying for a very junior position, chances are your work will involve plenty of technical discussions.
Technical discussions at work aren't so high-stakes as ones in an interview. If you do badly in an interview, you've wasted an entire day for nothing. If you fail to speak in a technical discussion at work, best case no one notices, worst case they think "Hm. Looks like Brian didn't get much sleep last night". Just the knowledge of these stakes can affect your performance.
I'm interviewing right now and all except one have lead to offers, so no, I'm actually good at my job. I'm just a better engineer than I am an interviewee.
I've had some issues with the whiteboard in the past. After I blew one interview (partially based on my whiteboard performance), I spent a lot of time coding solutions with a pen and paper, working on my penmanship (so that I could read my own writing) and just getting used to solving problems away from a screen. It really helped me get over a lot of my issues with that particular situation.
It puts people on their home turf, where they are most confident and comfortable. They can show me what they are proud of. I can ask lots of questions about it and see their thinking without having to be a total dick asking for an esoteric algorithm on a white board. There's also a lot to be said for how well someone know show to use their tools. Are they navigating proficiently, or are they poking around the IDE like they just started out.
This sounds like a softball interview, but you would be amazed the number of people it weeds out. You probably need something more rigorous beyond that, but don't start with the hard problems when an easy filter can really give you a lot of insight.
(Note that generally you can't share your current job's code, but people are fairly comfortable opening their own laptop and poking around in the code. I never ask for people to send me that kind of code).
This is of course a double edged sword - on the one hand you only have a generic sense of what you are looking for and on the other you have no vendetta for hiring someone who is sub-par to fill your urgent needs.
On desktop found this app to adjust contrast per-website:
https://chrome.google.com/webstore/detail/high-contrast/djcf...
You should read it again. He's talking about kindness and giving people space to come out their shell (amongst other things). That's not him bragging, that's him helping the interview go well (if it can).
Shrug. I thought it was a good article.
For me, nothing else comes close to evaluating talent.
I prefer making my candidate comfortable and writing something simple. 15 minutes, privacy, a drink, and pencil & paper. I hate white boards for this.
I come back & see what they've done. It doesn't have to be perfect. It doesn't even have to be right. I am less concerned about their code than our discussion about their code...
How did you approach the problem?
Why did you name your variables like this?
What is that supposed to be doing?
What happens if...?
What would you change if I added <x> to the problem?
How else could you have done this part?
Why did you choose your way?
Fizzbuzz is a great problem for this. I'm still amazed by how many "experienced" programmers can't do this.
It bears essentially no relation to my work day-to-day.
If you won't code for me in an interview, why would I possibly be interested in you?
Thanks for saving me so much time.
I think I might walk out too, because it's clear that power will be exerted on any candidates who choose to subject themselves to it - it says a lot about the type of work environment that is fostered.
I bombed the whiteboard/paper tech interview when I was first invited to interview at my current job (lots of ex co-workers here), and was declined as a candidate.
Months later I was invited again, this time just to start, skipping the interview. I'm finishing up my last week here and have nothing but praise and recommendations from my higher-ups, yet I would have never had a shot if people I worked with in the past didn't work here and reach out to me.
Actually at one of the places I recently interviewed, I got past the phone screens, cultural interviews, and even a google hangouts/screen-share session where I coded live while talking about the solution -- but I still had a 'get on the whiteboard and write some twist on academic sort functions' and of course froze up more and more with each time I used the eraser. I'm confident I would have done better with a computer.
I don't code on paper and whiteboards. Most of us aren't trained to, and state-dependent learning is a real thing. You cannot expect most people to perform as well in a drastically different environment from how they typically do their job.
Ask for references, read his open source code and talk to him over lunch about some of the stuff he's worked on. Someone who doesn't know how to develop software will be easy to spot.
2. If you think coding = dancing like a monkey, maybe you should do something else.
3. I've done everything you suggest, but the interview test is much more effective. You can bullshit through your suggestions (references are a joke), but not mine.
(2500 technical interviews)
I never know how to approach this question, because often the answer is, "the problem is so simple and general to the point that all variable names are either academic (solved computer science problem with technical name for each concept involved) or hopelessly generic ("counter").
When I code things that actually do things, I just name things what they should be called. I think functional programming makes this glib prescription much more tenable, as many of the things you write are abstract of the implementation, so you never find yourself naming something "screen_refresh_counter_for_process_x."
I don't give a shit about what you wrote on the page, I care what you have to say about it.
Your comment tells me more about you than you imagine.
I'll flatter myself and assume you liked what you saw in what I wrote. But if that's the case, why code at all? Why not just sit down and have a short conversation?
I feel like there's something about a rigorous, test-of-endurance interview process that feels professional. "We can't afford to hire anyone but the best."
So experienced hackers with good instincts end up ignoring them in favor of a gauntlet they know is not representative of actual work.
... can't do this in your incredibly artificial, unrealistic context.
I hate live coding in an interview, especially on paper or whiteboard. I know you say you don't care about syntax and little stuff, but that's not what happens in my brain. I suddenly forget the most basic things. "Wait, do I use semicolons at the end of lines in Ruby? Aaaah, I've been using Ruby for 5 years. How can I not know this. I'm going to bomb this interview. I should just leave now."
I know it sounds ridiculous to you, but something very similar actually happened to me. Give me a block of time and a sample problem to work on at a computer, fine. Ask me to refactor code that already exists, whatevs. I'll happily write and discuss code. Just don't give me that blank piece of paper. My brain goes crazy.
And the funny part is that I've been told one of my best qualities as a developer is my calm, thoughtful performance under pressure. When somebody accidentally drops that table in production or both load balancers go down, I'm your guy. I'll get it fixed quickly, without a lot of panic and fussing about. Just don't give me that piece of paper in an interview.
I can't explain it, but it's true, for me. Maybe it's just that I deal with real crises often enough to have developed a mental muscle for it, but I interview so rarely that I don't have the patterns yet. Either way, I think you're missing out on great candidates with this approach.
It doesn't have to compile or even be right. But you should at least be able to pseudo code your way through it, explain what you did, and discuss it.
For a professional programmer, anything less is just an excuse or an attitude problem.
> For a professional programmer, anything less is just an excuse or an attitude problem.
You're entitled to your opinion. I disagree. I think you're projecting your personal experience of the world onto other people, which is your prerogative as the interviewer. However, speaking from personal experience, I know some people who would otherwise make excellent teammates, top contributors, etc., will fail this test, and many who pass this test will make terrible employees.
As I said elsewhere, you will get many false positives and some false negatives. Thus, I think it's a bad test and a waste of time.
... can't do this in your incredibly artificial, unrealistic context
Programming profession is hard and full of rough edges, unfortunately.
Citation needed.
That's the part I don't get. It's so arbitrary and artificial. All you can be sure of is that I can solve fizzbuzz in 15 minutes.
> Programming profession is hard and full of rough edges, unfortunately.
I agree completely, but I think paper programming exercises produce false positives AND false negatives.
The flip side equivalent: many modern writers cannot use computers to be creative: http://mashable.com/2014/02/15/modern-writers-technology/
Here is a list of (mostly manual) typewriters that authors use: http://site.xavier.edu/polt/typewriters/typers.html
"What was your approach to the problem?" = I pretend to think for a few seconds, then answer. You're basically just figuring out if I've encountered your almost exact interview previously and I've got the canned answer ready.
Currently we have a timed programming exercise we let people do at home, then review and ask questions like this over email as a filter, it seems to work fairly well.
In person, I just like to ask people about their work experience, especially painful failures and learning experiences they've encountered.
Things like variable names are meaningless, imho. Every company tends to have an arbitrary set of rules they like to follow, unless someone is using things like 1a a1 11a1 a1a1 altogether where I can't read it, I've gotten over caring.
(I think a longer work sample test on the candidates own time is probably still better, but any approximation is better than nothing)
Then "how about without the modulo operator". The solution is so obvious - using the modulo operator, I had a mental block. It was fine once I got nudged in the right direction, but under normal circumstances I wouldn't have needed the nudge.
He kinda touched on it at the end with #1: "Know what you're looking for".
Kahneman is more explicit in Thinking Fast and Slow: come up with a simple rubric of 5 or 6 traits you're looking for in your new hire and ask questions that allow you to score each candidate on a scale of 1 to 5 for each. When you're done, choose the candidate with the highest score.
Maybe the technical ability score gets double-weighted for developers. But it seems like a lot of the interview process (and discussion of it) fixates on this. Well this and, of course, vague impressions of the candidate's personality. In my experience, this misses other key things that make up a competent programmer and worthwhile employee.
In the spirit of Thinking Fast and Slow, here's a question. As an interviewer, what are the 5 most important traits or qualities you look for in a candidate?
Experience can be similar too. Many of the best see themselves "merely" as an important contributor to a team effort and remember the mistakes they made, etc. Many of the worst honestly believe they did a great job, led the effort, etc.
That would be the Dunning-Kruger effect in action https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
It really makes me wonder if our tools are broken, if we're all trying to solve the wrong problems, or if we simply are hiring too many people for whatever the project is.
I disagree a bit with the center pane, though (with the dog typing loops and the caption "I have no idea what I'm doing"). It is, in my opinion, very impressive to be able to find the largest sum of all square matrices in an NxN matrix at a whiteboard in 45 minutes. It's impressive enough (especially not knowing the exact question in advance) to be a positive indicator of the ability to do what is in pane 3, especially considering how many variables and logical basket weaving is often involved in moving some text a few pixels to the right. As long as you are willing to tolerate a very high false negative rate, this could be a good way to go.
Unfortunately, it does mean that most developers looking for jobs will need to intensely study data structures and algorithms, and walk around with them loaded in "exam ready" memory" in a way that few people would need to simply to do their jobs. Personally, I'd say most people would retain this mental state anywhere from 1-3 weeks after a college algorithms course, provided they were a good student who understood the material. And it also means that lots of developers who could absolutely do this if they were willing to drop everything to study 50 or so hours and train at a white board will not be hired because they were more interested in just doing their jobs (which, ironically, is what they'd be doing at the new job). So we are going through a very inefficient re-taking of exams every time we interview. So I'm not surprised that some people have decided to stop interviewing, or just give up on finding a job that requires this sort of interview.
It's up to a company to decide if they want to give up on hiring these developers, or to tolerate this kind of false negative rate. However I have close to zero interest in hearing about how hard it is to hire good developers if this is their approach, since it suggests to me that the aren't going to make any effort to find a way to avoid these false negatives.
http://kelukelu.me/interview/powerpose.html
Also, I'm embarrassed by how bad my art is. I was learning to use a brand new tablet.
I fail to see what is the difficulty on that, looks like very straightforward and easy question to do, even over the phone.
So, why is it such a stretch to ask people to solve Fibonacci? I don't understand this mentality. Reminds me of "why should I learn calculus? I'll never use it in real life!" from college. You're supposed to be a professional software developer, and should be able to handle an unfamiliar problem.
If you're a professional musician auditioning to play Mozart, you wouldn't complain that your audition piece is Beethoven.
The problem is, the vast majority of software development jobs are not this. That was the point of the comic in the post. There is a whole lot of maintenance, API gluing, and spec work that needs doing. Even at the "elite" companies like Google.
The current trend in hiring seems to be at least partially born of Google's ill-advised[1] strategy of hiring generic software engineers that could theoretically be slotted into any role at the company. Hire for a position. If that position involves challenging, novel problems then throw things like that at the candidates in the interview. If it doesn't, don't.
[1] After they reached a certain point as a company