Ask HN: How do you conduct coding interviews? | Hacker News Reader

Ask HN: How do you conduct coding interviews?

53 pointsmrburton61 comments

Recently, I've seen a massive increase in interview questions that seem to mirror leetcode.com and similar websites.

A lot of my colleagues, mostly senior, feel these types of interview questions are wasteful because anyone who spends a week reading over leetcode.com can memorize most of the questions/answers.

What's your perspective? Are you a big DSA (Data Structures and Algorithms) Interviewer or are you more of "Ask various questions to see how naturally intelligent are they?"

View on HN

We've spent a lot of time honing our interview process, and I think we're at the very least on the right track.

Our Java phone screen is really short:

1. Explain the difference between `public`, `private`, and `protected`

2. What does `static` mean?

3. What's the difference between a `List`, `Set`, and `Map`?

This takes about ten minutes. Anyone that can answer these three questions and doesn't present as a raging asshole gets an in-person (or skype) interview.

The in-person is a live coding session. The candidate can use their own laptop, or we will provide them one. The test is contained in a github repo, and consists of a couple dozen unit-tested challenges. We pick and chose which challenges to give them based on their experience.

The challenges range from "Write `Hello, World!` to the console" all the way to "Implement the QuickSort algorithm". The candidate is encouraged to use Google, and we link them to the Wikipedia page for any algorithm we're asking them to implement. This part of the interview takes one to two hours.

This process has essentially eliminated bad hires; nobody that we've hired using this process has turned out to be a dud or a jerk. I still wonder if we're passing on candidates that would be good if tested in another manner, but right now I have no idea how to correct for that, or even what I would be correcting for.

This is great. I especially like these parts:

> The candidate can use their own laptop, or we will provide them one. The test is contained in a github repo, and consists of a couple dozen unit-tested challenges.

> The candidate is encouraged to use Google, and we link them to the Wikipedia page for any algorithm we're asking them to implement.

The only other thing I'd be looking for as a candidate (and that I make sure to do when I'm hiring) is to provide reasonable estimates as to when the candidate can expect to receive a response or decision. And then meet them.

This is almost exactly how I like to do it. If you say your have X years of Java development (or whatever language) I will give 10 or so trivia questions like this. I don't expect 10/10 correct, but you should probably get 6-8 and you should be able to answer those in 2 seconds. I also give that instruction up front so they don't hem and haw on the 2 or 3 things they've never seen before.

This helps you when everyone and their mother is asking those same three question in their interview?

Those three questions aren't to pick out good candidates, they're to pick out hopeless candidates. I used to be almost offended when I would interview for a position as a senior developer and get asked entry-level questions about my programming language of choice. Then I started sitting the other side of the table and realised that someone applying with "5 years of experience using C++" actually might still not know when to use public, protected and private. It's shocking how often something like that happens, and if you don't pick it up until an in-person interview, you're just wasting a lot of everyone's time.

>Then I started sitting the other side of the table...

I think everyone should get a chance to participate in their company's interview process, even if it's just shadowing while others conduct the interviews. It's one thing to hear stories about people with 10 years experience not being able to write basic code. Seeing it in person really drives the point home.

Seeing it in person really drives the point home.

It does. The only thing I've seen that drives it home more is being part of the process to fire someone who was a bad hire, particularly if you were also part of the hiring process that gave the wrong answer. If that individual is basically a decent person and just doesn't turn out to be very good at the job then it's never pleasant to know that you're taking their source of income away and especially to be the one breaking that news to them, however much it may be necessary objectively and professionally to do those things.

Here's a general style of problem that I've asked a few different ways and have been happy with:

1.) I provide an API (some function signatures and documentation) that will be used for the problem. In one case it was some functions for iterating through search results from a search service. Ideally the API is unintuitive in a few places (as legacy code often is), but I make the documentation unambiguous. Ideally it's inspired by something I actually did at work.

2.) I provide some problematic code that tries to use the API. It has correctness issues, style issues, and performance issues. I ask the candidate to point out any problems they find and fix them as they see them. We keep going until all of the issues are solved, and we have a short discussion of the new code at the end.

3.) If that goes well, I provide a small extension to the API and ask the candidate to write some new code against the API that solves a particular task. Ideally it uses the code from the previous part as a helper function.

The format is nice in that it simulates a lot of the nitty-gritty things you do in software engineering (code review, bug fixes, refactoring, dealing with legacy code, reading documentation), and it's always specific enough that they definitely haven't seen it before. It's also standardized enough that it's easier to compare across candidates without bias.

One downside is you need to prepare it for each language in advance (assuming you're willing to allow multiple languages).

I'm starting to wonder if software engineering as a discipline needs to become one of the professions.

All of the big companies have roughly the same interview process, and ask roughly the same sort of questions. It seems like a huge waste of resources to me.

I'm wondering if it would be better if everyone got tested on the same set of tests to eventually achieve a qualification that is universally recognised. So like "SE level 1" would do your standard data structures and algorithm puzzles from Cracking the Code Interview, "SE level 2" would be like system design, maybe doing a project etc.

That way the interviewer just needs to test the candidate on the soft skills, would they be a good fit for the team?

I know software engineering is multi faceted, and people might specialise on certain areas, so they could focus on those bits if the job needs those requirements, but I think having the baselines stuff in there would at least indicate to the employer that this person has passed the reverse a binary tree stuff and so on.

This would probably involve Google/Amazon/Facebook etc working together to organise such a qualification path, but surely this would be beneficial to them in the long run?

I feel that software engineering in its current state is more of an art than a science (or it's the art of using specific sciences?). It's not understood well enough to say "this is the way to do this thing". Thus all the bike shedding.

Until we do, we should treat it like art and have portfolio be a standard thing (github account or whatever). Yes, that means having side projects. If you don't have time for it, you're in the same spot a graphic artist who doesn't have time to draw is. Hopefully you have something to show from a previous job that's public or something.

Then people can just have a behavioral interview and a review of the portfolio and be done with it.

> It's not understood well enough to say "this is the way to do this thing". Thus all the bike shedding.

Most of the "reverse a binary tree" and "write a linked list" and other problems that use common data structures/algorithms (e.g. depth first search) are pretty bounded sets of problems, surely you could test on those?

I agree that the rest of software engineering is quite loose, my idea was more to set a "baseline" so that when companies interview someone, they can skip the "hey can you reverse a binary tree?" test and jump straight to the most interesting stuff like soft skills and so on.

> Most of the "reverse a binary tree" and "write a linked list" and other problems that use common data structures/algorithms (e.g. depth first search) are pretty bounded sets of problems, surely you could test on those?

That's like testing a home renovation contractor by asking them to pierce a hole with a drill.

They should 100% absolutely know how to do it and its critical to their job, but it's not exactly what makes them qualified for it.

> but it's not exactly what makes them qualified for it.

No, which is why the interview should be testing on more interesting stuff like system design, team based skills, other technical minutae that might be relevant to the job.

I just feel like compaonies waste a lot of time and resources on "reverse a binary tree" and "do the travelling salesman" type problems. I understand why they have to, because there's nothing out there to say "X candidate knows how to reverse a binary tree", so they have to test on it to make sure at least the baseline stuff is there.

And they keep retesting. If a candidate moves from company to company they get the same test, regardless of levels of experience. It just seems like something that could be nipped in the bud with a certificate/qualification

Fair enough. Though we have college diplomas and certifications. People just cheat a lot.

Other professions generally have significant checks and balance (eg: inspections after renovation, all the regulations around health care, etc). And even then lots of bad things happen.

I guess we could start keeping software engineers legally accountable for stuff...but that doesn't sound super fun to me.

> I feel that software engineering in its current state is more of an art than a science

It's a craft. What makes you say it's an art? What exactly is "artistic" in programming?

How subjective it is. How 2 groups of equally qualified programmers can look at 10 line of code and completely disagree on how good it is.

The software I write controls expensive hardware. I can't easily show things like that on github.

Besides that, it is just plain wrong to require us to have side projects on github. I love what I do and I'd much rather spend time tweaking my work code to perform better.

Other professions with similar subjective requirements need similar things to get a job (hell, some need it to even be able to get in school). Some you might even be working with on a daily basis and they make a third of your salary.

I'd dare say the salary the average software engineer command these days (especially in North America) has that baked in the compensation.

If you're not in the US and make a salary more in line with professions that have similar level of efforts, fair enough, you might have a point.

Personally I just filled up my github account with 1-2 projects in 2-3 weeks last time I was looking for a job. A few thousand lines of code is enough to get the point across.

Just because you'd rather spend your time on work code doesn't mean it's 'just wrong'. That said, if you can show what you've worked on in some capacity then I don't see why having extra code on just GitHub would be necessary.

Many of us are in jobs with contracts that don’t allow us to share side project code, even if written outside of work.

Depending what state you're in, those are often not even enforceable.

Really? I didn't know this was a thing. Money must be good.

This would definitely help people with interview anxiety and put those people on a more level playing field.

I think it's really, really important to point out that the criteria you're selecting for can vary greatly depending on the company, industry and role. A lot of companies have adopted Google/Amazon/Microsoft-style interviews for roles that are absolutely nothing like what those companies need.

In my work experience, knowledge of DSAs is really a luxury and has relatively little predictive power for a successful employee. I want to know how you approach engineering problems, what your experience has taught you, how you work with a team, what kind of work do you find interesting. A lot of people successful at my company would not be successful at Google and vice-versa.

I try to ask some general questions about what they like, what part of the stack they find themselves most proficient in. If they claim to be full stack still find out what they're most proficient in. Then take them through some of our code that needs refactoring, from obvious to deeper refactoring, have them ask me questions about it, make suggestions and so forth. But I haven't done that for nearly a year because I'm just consulting now.

I was at a thing last night waiting in line, and someone behind me was talking about an interview they had given. They had the guy explain what quicksort was and do some whiteboard coding etc. etc. and he did everything perfect and he had an outrageous cv and they were wondering why he wanted to work there ( this was half overheard so I didn't get everything) I was surprised because I've never actually been anywhere were people ask me to explain quicksort...

I was at one really good interview one time where I had to suggest refactors to a little game in a language I didn't know and the suggestion should have been binary search - but I had a brain problem at the moment and couldn't remember the name of the algorithm so I hesitated (uh how do I say what to do here when I can't remember the name of that big array algorithm) after I had gotten the code down from taking half an hour to run to taking 8 minutes (saving different states so I could just look them up when needed) and the interviewer got bored, did the solution and showed me out (politely)

It turns out, the answer to whether someone can do the job you'll hire them for is to do a little of the job you'll hire them for. (Carpenters don't whiteboard bridle joint solutions for cabinetmaking, prospective chefs don't solve biochemistry problems, etc).

I work for Pivotal. For engineering hiring is done in two parts.

The first part is "Rob's Pairing Interview". This is a simple technical screen -- essentially, if you have some programming nous, you will pass. Takes about 30-60 minutes.

The second part is to bring you in for a pairing day.

In the morning you pair with an engineer on one team. In the afternoon, on a different team. A different group of folk take you to lunch (we try to have a mix of disciplines and divisions for the lunch).

When you're pairing, you will work on real code, for a real problem. We usually try to put you into two teams with different stacks or problem spaces, ideally matching one of them to something you cite on your résumé and the other one to something you don't.

I think it works pretty well (obvious bias: I did well enough to get hired). Folk who really don't want to pair self-select out, usually even before applying. Everyone I work with is capable and personable, which goes a long way to making it a great place to work.

So if you have a way you actually work, see if you can just try that.

So, you/Pivotal are saying you require an entire day after 1st interview to deduce whether an applicant can work for you? Seems you're initially screening by who can devote that amount of time before anything else. I also assume the shortage/difficulty of finding tech staff doesn't affect the company, given the insistence on this.

Do you declare this time requirement in your job ads? I imagine candidates would appreciate knowing this so they wouldn't bother applying in the 1st place.

> Do you declare this time requirement in your job ads?

I checked, we don't. You're right that we ought to at least flag it as a possibility.

It's worth bearing in mind that it's the last stage. By the time you reach it we've done a résumé review and a recruiter has rung you to describe the RPI and pairing day.

There's no up-front time commitment beyond submitting your CV.

I imagine lots of people say no at this stage. That's fine. It's actually as it should be, if you think about it.

> I also assume the shortage/difficulty of finding tech staff doesn't affect the company, given the insistence on this.

It affects us sharply because of this. We do it because we would rather choose colleagues carefully than quickly. Hiring is not a sprint.

Every hiring decision is strategic. There are no shortcuts. We pay a heavy price in lost opportunities and we pay it gladly.

The problem is that you're also filtering out the candidates who are not interested enough in the company yet to take a day off from their current job. And most of the good programmers probably already have a job that's comfortable enough.

This is one of the two most common objections I see raised when I describe what we do, the other being the pairing thing. Of the two, the fact that we preach and practice TDD and pairing probably shrinks our applicant pool more than any other factor.

For people who can't get the time to do a pairing day, it sucks. We miss out, they miss out. There's no dressing things up: we know it costs us good applicants.

But for people who just aren't feeling it, it's actually a good thing. As you note, it's a seller's market. Why would you apply for a job you feel only "meh" about? Why would we want you to?

> Why would you apply for a job you feel only "meh" about?

Because I don't know yet. You're not that famous that I would know work conditions at your place from hearing your name only. You need to convince me that you're a good company. (Too many people forget that interviews are not only for company to assess the candidate, but also for candidate to assess the company.)

> Why would we want you to [apply for a job I feel only "meh" about]?

An employee doesn't need to be all cheery and enthusiastic about the field to do a really good job there. For instance, I couldn't care less about TV, I don't watch it for a long time, but since I started my current job in IPTV, our infrastructure got much more robust, easier to manage and monitor, and easier to extend.

And why would you want me to apply? Because I'm a good craftsman, I bring to the table a perspective unusual among the programmers, I like to share my knowledge with colleagues, and I like writing (technical) documentation.

It's a ridiculous thing to expect a candidate to be enthusiastic about you before he even got a chance to talk to you. This idea needs to die out.

I feel like we are talking past each other; that you're getting the impression that we're looking for cheerleading rah-rah types.

That's not what I intended to convey. What I wanted to convey is that we want people who want to work the way we work: TDD, pairing, sustainable pace, balanced teams, mutual trust. People who hear "Do what works. Do the right thing. Always be kind" and think it shouldn't need to be mottoised in the first place.

> Too many people forget that interviews are not only for company to assess the candidate, but also for candidate to assess the company.

Absolutely. I've known folk who did great, enjoyed their time, but got the chance to taste the actual job and decide that it wasn't for them.

Expanding on your point: wouldn't you want to know what the job is actually like? Because it's not as though trivia contests are a daily part of engineering life. What you see is exactly what you'll get.

I got an offer in October of 2013 and I started in February of 2014 (immigration, yay). I spent the first few weeks waiting for the shoe to drop. "This is just like the interviews", I thought. "When does the real job start?"

It took a while to realise: this is the job. This is the actual job. There were no surprises. I came in, I interviewed, I had a fantastic time and I've wound up in the best job I've ever had.

> And why would you want me to apply? Because I'm a good craftsman, I bring to the table a perspective unusual among the programmers, I like to share my knowledge with colleagues, and I like writing (technical) documentation.

Again, I think I gave the wrong impression. It may seem that I am standing in judgement of people who don't want to apply, or prefer not to work the way we prefer, or who think the process is ridiculous. That is not the signal I mean to convey.

Let's go back a step. The discussion here is: "how do you conduct coding interviews?", with the assumption that we've reached the coding interview in the first place.

Pivotal does that with pairing and frankly it works really, really well for us. Getting to that point is difficult, but we wouldn't do it if we didn't think the net results were terrible.

"When you're pairing, you will work on real code, for a real problem."

Do you pay them at least minimum wage for this real work that benefits your company, as required by federal law?

For coding interviews or take-home challenges, I like to ask applicants to solve questions that are a microcosm of the type of work they would be doing for us daily.

The questions have nothing to do with brainteasers or puzzles or even algorithms, and the coding challenge isn't even particularly difficult. A baseline solution requires only 5th grade math and a day's experience with the framework we use - but is flexible enough that the developer we're looking for will solve it differently than the one we're not.

That's because the work we do isn't groundbreaking technologically and doesn't require technical rock stars, just basic technical competence.

We're more interested in how they think about a problem that's similar but not identical to those they've solved before; how they use the time allotted; how they structure their code; whether and how they communicate about their decisions in comments or a README. Those traits are much more indicative of whether they'll be successful on our team.

And the coding challenge isn't just about testing their skills. It's also a jumping off point for discussions in the interview about how they think and communicate about code and problem solving.

Yes, it is a waste and really a bad practice but this is the only way to get into these big cos unless until you are famous speaker or github\twitter\youtube rockstar.

I know a lot of interviewers who go through cracking the code interview book before taking an interview so what is the point? It is like school system. You read from a book and in the exam you will get questions from that book. It is really a pathetic and sad approach. Some people even spend 3-6 months practicing all of this. I would rather spend all this time in building something new.

We code in our zone and peace of mind. While interviewing, you are supposed to keep talking what you are doing and thinking. How is it possible? I think it is extremely hard to come up with a solution on the spot unless until you have spend long time practicing these puzzles then you get accustomed to these tricks.

If you are still keen on trying then there is an easy way - telephonic rounds are mostly about arrays and string so you can focus on that. For in-person rounds, you can focus on graphs and trees. It may or may not happen for you.

I know some very talented software engineers who would not have gotten their jobs if not for studying leetcode. It's now expected that successful candidates are able to solve these puzzles without much trouble so even experienced engineers will treat leetcode as standard interview prep.

I think the best questions to ask are those that tests someones ability to take a well defined process, and ask them to simply represent it in code. DSA questions are often more about problem solving, and I don't think problems are solved in 20 minutes of a stressed induced whiteboard interview. So its often a matter of luck if the person finds the solution in time or not. Luck beeing if they've recently practiced and reviewed similar problems and their solutions.

If you can find a question where the algorithm and technique is given, basically you give the solution but in prose, and they have to translate the solution in code. Then you ask questions that has the candidate reason about various aspects of the code he wrote like: How fast do you think it'll be, how much memory would it consume, how easy is it to extend, how can it be tested, how easy is it for others to read, where is it most likely to fail, how could it be scaled, etc.

You could ask say, I've got a bunch of fruits, there's apples and bananas and pears. They need to be labeled with their price, and put into distinct baskets. The price is calculated based on their weight and which kind of fruit they are. Banana is weight * 0.3. Apple is ..., etc.

There's no trick to this question, no problem to solve. All that's needed is can you model this process in code in a way that's maintainable, testable, efficient, extendable, scalable and do that quickly enough. This question doesn't require any practice or reviewing of theory prior to the interview. Questions of that sort have worked the best for me.

Caveat: This works when you need a good generalist that's well rounded. If you need a specialist, you expect them to know and be well practiced and versed in the specialty. But you almost never need a specialist.

Disclaimer: I've currently conducted around 40 interviews, a mix of on-site and phone, for both hiring junior and senior software engineers. So that's the sample size that lead me to believe this works well.

Three things I look for at the outset, fairly unrelated to coding ability: (1) humility, (2) entrepreneurial spirit and (3) passion. If they have those things, I feel I'll enjoy working with them. Then I'll spend 30 minutes talking through an actual problem I'm trying to solve, to see if they can add value. I ask the interviewee to be at ease, and to treat this like a taster consultation they might give to a prospective client. We cover all the bases, from design to data structures for the implementation. I generally don't ask them to write code, as the ongoing discussion easily reveals who can and can't write code well. I might check out their github repos if I'm unsure about code quality, but often the conversation is enough to know if they're worth hiring.

Leetcode has 695 public questions. If you can memorize so many questions and answers over a week, you deserve to be hired.

Most of their questions are trivial anyway.

https://leetcode.com/problemset/all/

I wouldn’t say the optimal solutions for most questions are trivial on leetcode. And to do all of them in a week is impossible, your need a few months.

My favorite question to ask is one I've iteratively improved on over the last 5 years or so.

To start, I always phrase the question as a real world example - "I have a server, and we currently have some functionality X. I want to augment it in way Y, and I want you to help me do that". I structure it like a real problem rather than a toy "write an algorithm that does this, and demonstrate that you know this one arbitrary data structure I have in mind" type of question. The end task is the same, but it gets the candidate in the mindset of doing actual work and thinking about the implications of their design decisions.

The question starts simply enough, with most candidates at the level I'm interviewing for able to complete it in ~10-15 minutes, focusing on basic language structure common to most languages, and simple DS/A.

Then we get to have a little fun. Based on the role that the candidate is being considered for, how they did on the first part, and the feedback on other interviewers on their strengths and weaknesses, I can steer the question in several directions:

  - Let's talk about concurrency (also plays nicely into the "this is in a server we have" setup, since thread safety is not just a nice-to-have, but a real requirement)
  - I'd like to augment it even further with new functionality (which involves much more deep knowledge of various data structures and their nuances)
  - I'd like to build this into a whole separate service to handle this functionality.  What would that look like?

I come with a laptop and some skeleton code in a handful of common languages, and give the candidate the choice of white boarding or typing, as well as allow them to make use of the internet, to look up libraries, APIs, etc. This all also plays into the general theme of "structure the interview like the work they'll actually be doing".

If the candidate is not doing terribly well, the question can also stop after the first part (which even the weakest candidates can generally get to a decent solution on after the length of a full interview). This avoids the candidate from feeling like they did terribly, and leaving the interview with a negative impression.

At CharlieHR we have two parts:

1.a remote tech test. Limited to two hours (though not timed - honour system only). Any language, any libraries.

2. in house pairing exercise. This is 90 minutes pairing with me implementing a simple thing. Any language, and libraries, all the online docs you want. They do most of the coding and should talk a lot about what we’re doing.

The important thing is the nature of the tasks. We use the same two every time and their characteristics are:

  * not logic problems
  * not algorithms or data structure problems
  * a simple task (like – but not – telling if a given noughts and crosses grid has been won by either side)
  * CLEAR requirements: there is no mystery as to how they should work, and there is no logical leap required at any point
  * related to what we build at CharlieHR

I’ve found the pairing interview to be very good for cutting through the bullshit and really getting a handle on their thinking and ability. Can they interrogate requirements and turn them into working code?

It’s interesting because in theory a junior dev and a senior dev could both do very well on these problems, but it is usually evident when someone is senior thanks to the level of additional insight they provide into different ways of approaching the problem.

Brief telephone screen, mainly to confirm the role is a match to their expectations. We tell them about the role, and give them the chance to opt out if the role doesn't fit their requirements.

Then an in person interview. The usual discussion about work history and experience.

We do a technical piece. This consists of a spec, a laptop with appropriate tools/compilers installed and about half an hour to implement it. Followed by time afterwards to discuss their implementation, what else they would do that they didn't have time for, choices they made, etc. We are around to answer questions, but leave them in peace if not. We allow full access to internet resources as you would expect in the real world. The specs are usually fairly contrived, they have to be to fit in the time aloted, but we try to make them something that any decent programmer would be able to implement even if they have never see the concepts before. Where the best stand out is in their follow up discussion afterwards.

This blog post has been quite influential in our process - http://www.lihaoyi.com/post/HowtoconductagoodProgrammingInte...

I just wrote up some thoughts on the topic:

https://www.exratione.com/2017/10/in-search-of-the-spark-whe...

Essentially, the question asked is just the setup to watch someone approach a novel experience. If the candidate is not in a natural environment, i.e. working with a comfortable IDE and with access to Stack Overflow, it is kind of pointless. You want to see what they do in a situation fairly close to the mundane daily example of "how do I do this new thing."

So don't ask standard related to business challenges that have answers online: find novel exercises. Write a sort function that sorts everything except for at least two items. Write a linked list node that sabotages the user in the most subtle way possible. And so on. Fortunately these are easy to create, as there is an infinite space of engineering exercises that no-one would carry out in reality as they are self-defeating, or pointlessly destructive, or otherwise non-productive. But they still serve to illustrate how someone thinks their way around a new concept, and that is the important thing.

I'm a fan of using open-ended discussions rather than testing "book knowledge". Some simple textbook questions to gauge a candidate's approximate level of skill and experience are fine, but that's just the introduction. For the main event, I might give scenarios that have a lot of potential depth for them to implement, or I might ask them to review existing code for such scenarios that is relatively crude in its implementation and perhaps contains some errors or oversights. In each scenario, I'm more interested in the way a candidate is thinking and what sort of level they are working at than any specific details.

For example, assuming the candidate can give me a correct implementation of some specified behaviour at all, do they just take a brute force approach? Do they demonstrate some basic awareness of different data structures and algorithms that might help them, and make reasonable choices? Do they immediately start looking at the required data and access patterns carefully, and automatically consider whether the standard library provides the required tools, whether other well-known libraries would be better, or whether something completely new will be best? What sorts of constraints do they take into account in making these decisions? Do they consider issues like how we might ensure the correctness of the code or measure its performance? What tools do they use? What sources of information do they automatically turn to for reference?

I would always prefer to use a natural environment for this sort of discussion if possible, so the scenarios would be realistic and relevant, and within reason I'm OK with the developer using whatever tools or information sources they normally would for any programming involved.

If a candidate passes the basic phone interview, we bring them in for a technical chat. I'll ask the candidate to send some code of their choice before the interview and use that as a jumping off point. I tend to keep a few other questions in hand in case the conversation comes to a dead end, but I find it useful to let things go where they naturally go, since the candidate can show more about themselves and what they know. We only ask candidates to perform a test if we're really uncertain that they can code at a basic level, and don't bother with puzzle questions at all, since they've never told us whether the candidate is competent or not, just if they've done a bunch of puzzle questions in preparation. It's a waste of our time and theirs.

Some candidates are great at blagging, which can be hard to identify, but it's usually unearthed if you start asking about a subject on their CV, away from their day-to-day duties, in any depth (depth being more than 30 minutes of study, since this is all most blaggers have really looked into things).

At a previous job, we used to do a short phone screen, and then send the candidate a small homework assignment they could do on their own time.

The assignment was to build a simple client-server (in C) that solved a particular task, and the short spec (4-5 sentences) was intentionally non-specific about a number of things (that we did not care too much about anyway) but quite specific about some others. The task was similar to a small component of our embedded product, so it was relevant for the role.

The candidate would then spend half an evening or a weekend or whatever time they needed, and send us their hopefully working code. It normally took about 2 hours for someone familiar with this sort of thing, maybe 4 or 5 if they had no idea. We didn't impose any deadline, but most got back to us within a few days or after the following weekend. On a couple of occasions we never heard back.

We would then verify that the requirements were met, and we would take note of some of the design decisions where there was flexibility and things didn't matter so much. If they had at least made a good attempt we then had lots to discuss at the technical interview.

We had them walk us through the code, describing the flow and their design decisions along the way, so we could tell who actually understood what they had done. This was particularly interesting for certain sections where there were 2-3 potential ways to solve a problem, we would ask what the others were and how they had chosen to solve in this particular way, e.g. what tradeoffs they had considered. We would ask how much testing they had done (most said "thorough"), and whether they felt it met all parts of the specification (it didn't always, despite the "thorough" testing).

In all, most candidates told us they thought it was a good task without being daunting or time consuming, and we felt that we got some good insight into our candidates before we made a hiring decision. No puzzles, no grammar nazi questions!

First start with work that they've written. Ask for a Github project or even schoolwork. Get them to describe the steps that they took. Approach them humbly, since they might not be great at communicating under pressure and may not properly express a consideration (like a prof mandating that they use ZMQ, say).

The code doesn't need to be perfect, but well written and hopefully tested, especially if it's in a dynamic language.

After that, do some basic hash map vs sorted array questions, some "how does the internet work" questions, and maybe get them to code up some solutions on a laptop that you get them to bring ahead of time (provide one as a backup though, since some people only develop on a desktop at home).

We send out a simple homework. Should take you around 3 hours at most, creating a simple rest service.

Then we would go through the code. Apart from that, we would go through the cv/talk about past projects.

I have interviewed for a ton of companies in the bay. If I can't convince the company to offer contract to hire positions. I pick a realish problems adapted for interview from our day to day. I then try to "work with them" to see how much I would like to work with them. I prefer 1.5 hour interview slots but I usually only get 45m.

I find companies generally hamstring themselves by having people breaking up the interview slots into too small chunks. Basically limiting us from doing real interview problems.

I recently heard/read something which I actually can't really remember whether was aimed at tech hires or other, but it just resonated with me.

Remember that the process you choose to screen your hires defines the people you end up with. Understood as; if you present hard complicated theoretical assignments, then what you get is people who are good at these and may overcomplicate implementation. You may want these - but in a lot of cases what you probably needed was somewhat more pragmatic programmers.

And on the flip side, if you're the interviewee and you don't pass an interview, try not to beat yourself up. It just means the interviewer / interviewing company is looking for something different.

At a previous job we had a nice test: show them a listing with some glaring bugs and ask to find bugs from the listing (buffer overflows, obvious memory leaks, printf format mismatch) Surprisingly this used to be a very hard filter, people are trained to write new code and do not know how to review code of others (or even their own stuff)

I sit down with them and have them write code. I give them a problem, a test suite, and an editor of their choice.

Their challenge is to get as many tests as possible to pass.

It's not really intended to get all of them - they branch into different areas, and the candidate can choose the direction they would like to move in.

I just went through a round of interviews and came away with a better opinion of these "toy" interviews than I started with[1], so I'll play devil's advocate here: these questions have a place. The main reason to use these questions is that they're studyable.

Google and Facebook both put out content that walk you through how to solve problems like this. When you get an interview, you can delay it as long as you want, they can tell you the sort of problems to study: go read "Cracking the Coding Interview". Facebook sends you a link to it and a video of the author working a problem. Google sends you stuff like https://www.youtube.com/watch?v=XKu_SEDAykw. There are thousands of toy questions, and it's not feasible to memorize them all. The correct _and expected_ thing to do is study up for the category.

Now that your candidates have in theory studied, you'll get a studied, "best" performance for each candidate, and interview outcomes can be compared in a very low-bias, apples-to-apples way. I can't imagine the success criteria for a pairing being clearer than for toys. I think the potential for toy question interviews to be managed in a low-bias way is the main reason they're used.

In comparison, how do you ensure that your exercise in go is exactly as hard as your exercise in Java? Or do they need to know go, or some other language? Or that they didn't spend longer on the take home than you asked? Or 100 other factors of your particular exercise? It's really hard to let every candidate shine on a level playing field using "real world" stuff. [2]

That said, there's good reasons to do pairing, or a bug squash, or a take home, or juggling, or _anything different_. The first is that candidates with 15 years of experience don't show up better than college grads on "toy" questions, even if they've studied. You can only do so well on them, and experienced candidates tend to have options and won't put a lot of time in to studying. And the corollary to that is that the biggest tech employers are leaving some great candidates behind, because they interview better doing "real world" stuff. And your company can go get them, just because you bothered to do something different. Most companies won't outbid Google and Facebook, so why would your interview process try to find the same candidates they do?

[1]: And to be clear: I did not get any offers from my interviews based entirely on this sort of question. I'm not the worst but far from the best.

[2]: Triplebyte's long-ish video chat interview is the best thing I've seen at "standardizing" this.

ask about their last project, favorite project, challenging project, what do they want to do in the future, why do they leave their previous job. that's all.

I wrote up what I think is a good hiring process for technical positions on my blog - https://blog.urth.org/2017/07/14/a-technical-hiring-process/

DSA questions make absolutely no sense in 99% of the cases because

1) Most modern programming languages have ready-made implementations (by way of libraries) for almost every imaginable algorithm, written in the most efficient way, and probably all open source, so you can modify it if you feel otherwise.

and

2) It's very easy for a not-so-capable-engineer to "hack" the coding interview by -- as you yourself have mentioned -- memorizing the DSA questions and answers from various coding interview Q&A sites.

The 1% where they would make sense is if you are hiring specifically for such a role. Example: Amazon or Fedex trying to optimize their transportation model.

I've interviewed over 30 different candidates for 3 different companies I worked at. Here's what I do, and it works for us, YMMV.

1) Before the interview, I go over what the position is, the job requirements, ex: Front-end Web Developer, or Back-end Engineer, or Full-stack Engineer.

2) I go through the code-base of a couple of projects we are currently working on, and based on the job requirements, I create 5 to 10 code-related questions. Ex: We were once hiring a C# and ASP.Net engineer and had an enhancement in our website code base, to show open-close hours of bank branches on the website. Most people miss the importance of tracking timezones of the branches and pegging the open-close hours to some timezone , ex: UTC. So I created a question to the effect of "We need to store and display bank branch hours on our website, how would you go about designing a data store, and displaying it on the website." If the interviewee does not mention anything about timezones, then you know that they are not thinking through this correctly. These type of questions are a majority of the questions I come up with.

3) I throw in a couple of simple 1 line-answer type questions that will expose any lack of understanding of the fundamentals of programming, print it, and review the question with them and give them time to read it and answer. Ex: What's the difference between "==" and "===" in javascript. Or variable scopes in a sample method. etc.

This way, you are examining the candidate practically for what he/she will be working on, if hired and he/she is likely to be productive from day 1.

Disclaimer: I've done more than 60 interviews at Amazon. My views are my own.

When we ask DSA questions we don't care about the data structures or algorithms. We care about the logical reasoning and problem solving skills that enable you to solve those problems. We contrive particular questions to give you chances to showcase those underlying skills.

If candidates can get through an interview by memorizing problems from Leetcode or HackerRank then the interview process isn't working. Most companies miss the mark on this.[1]

Sometimes I ask a question and the candidate recites an optimal solution in 2 minutes. Great, they got an optimal answer. But it's useless for me — I didn't get any data to inform my hiring decision. Maybe they created the solution from scratch using strong logical reasoning and problem solving skills. Or maybe they saw that exact problem the day before when prepping for the interview. Without digging deeper it's impossible to know. In these situations it's the interviewer's job to gather more data.

Sidenote: If you've seen the same question recently it's better to tell the interviewer. If you pretend that you're solving it from scratch your interviewer can probably tell. And if they don't notice you'll just get harder questions later in the interview that you'll probably strike out on

[1] Gayle McDowell has a lot of insight on how to conduct effective coding interviews if you're looking to improve your company's process.

> 1) Most modern programming languages have ready-made implementations (by way of libraries) for almost every imaginable algorithm,

I really don't think this is true of any programming language, modern or not.

> written in the most efficient way,

Elementary but nontrivial problems like string matching and integer sorting often have poor standard library solutions, not to mention that "the most efficient way" is context-dependent.

> and probably all open source, so you can modify it if you feel otherwise.

Intel's MKL, which has a ready-made matrix multiplication that probably is about as fast as possible on their hardware, isn't AFAIK open-source. (Regardless, if you modify the standard library in your own environment, you risk creating serious interoperability headaches for yourself later on.)

>> 1) Most modern programming languages have ready-made implementations (by way of libraries) for almost every imaginable algorithm,

> I really don't think this is true of any programming language, modern or not.

Moreover, for this attitude (overreliance on external libraries) you get another left-pad.

Reply on news.ycombinator.com