My favourite interview question (2006)
weblog.raganwald.com
weblog.raganwald.com
1) determine the system the candidate will design. It should be an internal system or an internal oriented problem, not an external thing like Monopoly or URL shortening services.
2) stack your interview team with a mixture of people who can contribute different thoughts to this problem. Engineering, UI, marketing, management, etc.
3) first interviewer of the day introduces the topic and helps the candidate understand some of the internal lingo and bootstrapping questions they'll have right off the bat. The first interviewer is pretty crucial because the candidate has to be prepared to write things down and carry information across each interview.
4) all subsequent interviewers come in and ask two questions:
* What problem were you given to solve?
* I'm an expert in technology / business area X. How can I help?
By the end of the day when they're having their likely optional interviews with more senior people, they should look disheveled but pretty happy, because now they're pitching to senior management an idea they collaborated on with a wide variety of team members.
They should have a whiteboard full of stuff, stacks of paper, whatever. You should get a good idea of how they organize themselves under pressure.
The interview team should have a good feel for how the candidate leveraged them for answers. The interviewers should have done a lot of talking! There should also be clear evidence of the candidate getting better towards the end.
At the conclusion they're tasked with taking a step back to the higher level to explain the idea to management - regardless as to whether management is technical or not, they should be expected to walk away understanding the big picture. Having non-technical "business" people contribute to the interview is critical, since they help the candidate understand some of the human requirements.
I could go on and on, I'm pretty excited about how this technique has worked so far. The key element is deliberately having the candidate carry knowledge between interview teams. I'm sure I'm not the first person to think of this :)
EDIT: styling
One question: Do you let candidates know ahead of time what the interview process will consist of? That sounds like an important step that many companies leave out - whatever interview process they use.
I think if I knew before arriving that this how it would work, I'd be a lot more excited about the interview and better prepared too.
Because it also requires a lot of mental commitment from the interview team as we'll as the candidate, I'd recommend using this technique as the final hurdle. I don't think you want to phone screen someone, then just toss them into this process. You need to have them cross some basic technical gate (like FizzBuzz, a take-home coding project, whatever), and have them meet a handful of people first, just to make sure they pass those early smoke tests.
In fact I think this technique works best when you want to seal the deal with a very strong candidate who is entertaining several options. Interviews should always be a two-way process, and if a candidate walks away from your interviews thinking "I had real trouble getting my point across to those people, I would never work there" then that represents success for the interview process.
The strongest candidates tend to be the ones least influenced by money and more interested in the problems, the people they will be working with, the culture and the environment of the organization. So having your interview process structured around revealing that gives you the best possible chance of matching your team with people excited to be there.
Conversely if you're a candidate, even if the company you're applying with doesn't seem to have thought too deeply about this stuff, don't despair. Hiring is one of the hardest things to do and also tends to be done by the seat of the pants (Amazon seems like a notable exception) You can impose your own structure by asking the right questions and making special requests of the recruiter. Tell them you want a chance to write code with someone! Or that you want to mock up a UI that solves a problem they have. Or you want to see what their strongest engineer thinks of your design for this service you've put together.
It doesn't matter if you're desperate for the job, or whether you wish they'd stop calling you ... this kind of stuff makes you stand out in the crowd as proactive and motivated by the right reasons.
Basically, interviewing is a clusterfuck. :-)
Having said that, there are ways to offload. There's a biotech startup (I'd have to dig for their name, I remember they did all their coding in mathematica) that described their hiring process to me. Hopefully I'm not messing up the details too badly.
I recall one guy who went through it said it was way more stressful than his final year university exams. They tested candidates, all candidates, on physics, math, chemistry, biology, computer science, and more. Candidates were given projects in each area, and a week to complete all of them.
It was deliberately structured so that there was no possible way that anyone could humanly complete the entire set of tests, much less while also working a full time job. This was deliberate, so that they could observe how candidates chose to prioritize. They also wanted to give people more than they could do to stimulate a "by any means necessary" approach they knew would be required in the actual job.
Finally, it also caused a lot of people to say "erm, no thanks. That's a ridiculous amount of work", which they rationalized as being a good test for candidates who saw a mountain of work and then baulked.
Not sure if I'd use a technique like this myself, but I found the idea interesting!
http://www.abandonia.com/en/games/895/Monopoly+Deluxe.html
http://www.textmodegames.com/download/monopoly.html
And here complete list:
Whether or not questions like these are effective, the point of them is to engage with someone in a way that illustrates how they tackle complex, loosely defined issues: the majority of engineering problems in a nutshell. In most modern tech companies, good communication and not being afraid to ask questions are absolutely essential. Someone who will take an incomplete list of requirements and draw up an entirely incomplete product without questions or discussion is as much a liability as someone who takes the requirements and does nothing, paralyzed by indecision and not knowing what to do. And oftentimes, candidates aren't perfect but you're going to hire them anyway. Going through this process can illustrate how you would expect to interact with the candidate and where the initial rough spots will be as the candidate begins working with the team.
From another point of view, candidates are usually more than prepared to talk about anything on their resume. In some cases, this is equivalent to listening to someone give a prepared speech. Asking them free-form questions like the above can be a good way to understand how the candidate reacts in situations where they aren't fully prepared. For some positions this is irrelevant, but for others it's incredibly important to have some measure of poise and thoughtfulness when engaging with someone who doesn't necessarily agree with everything you say and is questioning your thought process.
In short, these kinds of questions aren't primarily testing your design and programming abilities, but your ability to communicate and reason with another person in a technical context.
Case and point:
Tell me about a design pattern you used on X project?
A singleton? Ok, why did you use that?
and a long discussion about testability and architecture ensues with questions from both sides. That is WAY more like the communication that happens on the job than a contrived problem which in many cases is disconnected from real life constraints.
Nowadays, very very few companies innovate in recruiting.
I've interviewed with most "hot" companies, and I can find most interview questions (>90%) from https://oj.leetcode.com/problems/, exact same question.
True-est, and it's the perfect filter.
I think my boss (CEO of a mid-sized public company) took it as a challenge and gave him a very large offer -- which was politely refused.
This question is a great positive indicator, but a terrible negative indicator.
That is to say, a person who works on a lot of stuff in their spare time is very likely to be someone I'd like to hire. The reverse isn't true, which is why there are other questions in the deck. I realize that there are great developers who don't have a well populated github profile and dozens of side projects.
If I had a candidate that had trouble with the question, I'd expect them to say so. I'm a pretty reasonable person. I would not expect them to try and exit the interview early with little to no indication why.
If you're interviewing an parallel universe's patio11-equivalent, and they say, "I have been running FlashCardCreator.com since 2001...", and goes on to explain neat things about what she's learned from it, it sounds like that should be a big green flag. Not only is such a person likely to have lots of cross-domain insight, but they clearly have the mojo to deliver working code.
If you're worried that they might decide that their work-outside-yours is going to cannibalize their time, they probably have an answer for why you don't need to worry much about that. (After all, they __did__ apply.) I would be surprised if you had to worry about them leaving any more than you would any other developer.
I want to see if your eyes light up when you talk about it.
I couldn't imagine working that often and not enjoying it.
> Ask me that question and I'll probably
> find a way to wind up politely and leave.
For me, as a potential employer, that would be a good reason not to want to hire you. It makes you sound like you would not be an easy person to work with. It sounds like if someone came to you with an unreasonable software requirement you would simply shut them down without trying to find a way to deliver something that finds a good balance between what they've asked for, and what can be done.
But perhaps that's not the impression you intended. If not, could you expand further?
However, as an employee, such questions might be highly correlated with past companies that are looking for people who do not have family commitments, or who are willing to do unpaid extra work at home "for fun". As an interviewee, it makes me wonder, even if only fleetingly, "Will I be discriminated against for not being (young+single)?". (Are we even allowed to ask about marital status or age?)
Even if such a thing was never your intention, the interviewee might have had prior bad experiences with those who ask such a question, and view cutting the interview short as a way to ensure they don't get mistreated later.
But I understand the point point of view, I'm just trying to give some balance, and point out that the black/white attitude this suggests is sometimes a clear indication that you are not cut out for a particular position.
Regardless, if it makes me not want to work for you, and you don't want to hire me because of my reaction to the question, then my likely response would be win/win, right? :)
Then again, maybe you would not be a good fit. My comment is simply that based on just one question you are actively avoiding finding out.
I'm always confused when I see this sentiment. I only ever see it on HN, and only when someone feels they've been asked a question below their station or that is inherently insulting.
I've never used this as an only question, but I have got better results from people who have used their skills to solve some problem for themselves, but that isn't unique to programming.
Heuristics are, unfortunately, quite valuable in hiring scenarios, especially for start-ups.
It's also just much more fun to work with people who love programming, and an easy way to tell (even if it's susceptible to a few false negatives like your friend) is to find out if they program in their own time.
The only exception to this was a guy who did amazing work on side projects, but the work he did for his paying job suffered significantly...
The thing is though, you can love programming and find that 40 hours a week of it is more than enough. I think what you'd find with this heuristic is that it tends to favor young people. It's a lot easier to be enthusiastic about some side project when you're 20 than when you're 40 and have a lot of responsibilities.
So I'm not sure why the person without personal projects needs to demonstrate that they're keeping tabs on the industry any more than someone who does.
Seems like you want to ensure X by testing for Y and mistaking correlation with causation.
I disagree. If passion is an important quality to you, then, as the interviewer, you need to ask questions that reveal whether or not the candidate has that quality. It's absurd to ask about personal projects as a proxy for asking "are you passionate about programming" and then to expect the candidate to guess your true intentions and answer accordingly.
First, I have a family now, and spending time with them is more important to me than any job. (And any company that has a problem with this is not a good fit.)
Second, I've been lucky enough to find jobs that are interesting and intellectually stimulating. My first programming jobs were kinda boring, so I had extra creative energy and learning and building things was an outlet.
Third, owning a home always involves upkeep and fixit projects. These can be fun and are generally easier get my kids to involved in than teaching them to program (going back to #1).
I'm in the office 40-45 hours a week typically, I spend 20-30% of my time just thinking about the problems I am solving, or stuff like this, or following github projects, or other technology reading. I probably do about 4-5 hours of actual programming a day at work. My output during that time tends to match or exceed most of my peers in both quality and quantity.
The problem is getting people to understand more hours isn't the same as more productive output. Spending a couple hours thinking through a problem (or even prototyping) will usually save you more time down the road. This is true from simple data modeling to working through how to fit a system together.
As to the original question.. it is far from the only indicator of a person's own skill and drive, but it is an indicator. Especially in more junior level candidates. There's also the fact that the technology in use changes rapidly... and do you take the time to learn it... either out of your in-office day, or outside.
It is not uncommon for people to feel this about a number of professions such as chefs, bakers, graphic artists, etc.
However, in my experience I prefer to judge an applicant on professional credentials rather than their free time because while it may tell me quite a bit about them it tells me little about their capabilities that I couldn't find out in better ways.
In both cases, the point isn't to get the right answer, but (allegedly) to see how the person thinks. With the estimation question, the trick is to make up some plausible-ish numbers and then multiply and/or add them to get a plausible-ish result. If someone's seen one of those before, they'll nail it for sure. If not, it's a crapshoot. In theory, you're measuring whether or not someone's able to reason well enough to spontaneously estimate something off the top of their head, but comparing the number of people who've seen those question before vs. the number of people who can answer that type of question without having ever seen it before, what you're really filtering for is people who have heard of or seen Fermi questions.
A while back, someone's interviewing experience got posted to HN and they mentioned that they got asked to design a URL shortening service maybe five times. They failed it the first couple of times and got progressively better each time. I doubt the interviewers meant to measure whether or not this person had done enough interviews to catch on to the style of questions that are currently trendy, but that's what they actually measured.
I think the most common objection to this is that the point of these questions is to drill down and figure out how the person really thinks, but in empirical studies on interviewing, they find the actual evaluation of chatty questions like this is heavily influenced by all kinds of biases. Techniques that have more clear-cut evaluation criteria, like work sample tests, and even completely non-technical interviews like behavioral interviews end up being better filters. Even then, the filtering isn't "good" (IIRC, the last time I read one of those studies, work sample tests score the best, and had a correlation of around .5 with an ideal filter), but it's better.
For these kinds of questions, even if you haven't seen the specific question before, there's all sorts of interview gamesmanship that helps tremendously. One thing in the blog post, not spending an hour on requirements gathering, is an example of that. In real life, if you're going to write an application from scratch, spending more than an hour on requirements gathering is perfectly reasonable for a lot of problem domains. But if you're playing the interview game, you know that you have, at most, an hour to sketch out the entire problem, so you have to cut the requirements gathering phase short (but not too short). The post mentions that this gets at real skills. That's true. It does. The problem is that everyone who's seen this kind of question five times is going to be good enough at the interview gamesmanship that they don't need to have good real skills to breeze through the "don't spend too long talking about requirements" sub-filter.
I am not saying that you shouldn't test the coding abilities of your candidate, but it looks to me a good way to see how he/she reasons about the various components of the system and asks for requirements.
Back in my day there were no interviews like these. You just dropped your CV, had a chat with the guy and if it clicked you got a contract with a month on (paid) trial. When I hear things like this I am thinking about all the people who are investing an (unpaid) day (sometimes even recurring interviews, what!) solving questionnaires with no real context during the job.
Is it really this bad?
But many people don't want to leave the security of their current job for a temporary contract, so you would need to spend more time vetting those candidates.
Personally... I don't like it. But I do understand the logic, and how it helps businesses from making expensive mistakes in regards to hiring.
I combine that problem with developng a simple end to end feature together and the combination of that with a high level design problem turns out to be a really good indicator of how someone actually performs on the job.
I stumbled across it doing a Google search for "interview question elevator design"...I haven't been asked in a technical interview about how I would implement the logic for an elevator controller, but I heard about the question second-hand at a college job fair...and since then, I still think about that question. In fact, I think I've thought about it almost every time I've ever waited for an elevator in New York (that, and the story of the poor woman who was crushed while stepping into an upward-shooting elevator in Midtown a few years ago)...so...4 times a day, times 6 years...and I still haven't come up with enough optimal algorithms to cover all the edge cases and scenarios that I come across :)
The Monopoly question that the OP mentions is good, but I think is a bit problematic since not everyone has played Monopoly. And not everyone is familiar even with the standard rules. But everyone's been in an elevator before, and virtually no one thinks about how complicated an elevator's operation might be: it's one of those incredible, life-changing inventions that you take for granted after the first time you've pressed a button. So I think it's great material for an open-ended technical question.
The single-elevator logic is convoluted enough: If the elevator is going from bottom floor to 10th floor, obviously it should stop if someone on the 5th floor signals their intention to go up. But what if, after the 1st floor door closes and the elevator starts moving, a 2nd-floor person wants to go up? What's the cutoff in last-second button pushes to prevent the elevator from too abruptly stopping?
And of course, the more complicated logic is if someone on the 5th floor press Down while the elevator is heading up. Then you have to implement a queue system. But simple FIFO may not be efficient...if the elevator is moving up, and someone on the 5th floor presses Down, and then, someone on the 7th floor also presses Down...it would seem that the elevator should stop at the 7th floor first, on its way down.
And then things get really complicated with even just one more elevator, particularly with race conditions. If both elevators are moving up, and a 5th floor signal for "Down" is followed by a 7th floor signal for "Down"...ideally, the first elevator to finish its Up job will take the 7th floor...Should it also take the 5th floor Down job? Or let the other elevator handle that? And should that decision change if the 7th floor person signals their intention to go to the 4th floor, whereas the 5th floor person most likely just wants to go to the bottom floor?
Beyond the interruption/queuing logic, there's a lot of interesting discussion about how the elevators should be positioned during idle time. Presumably, they should return to ground floor in the mornings, and stay near the top (or at least midway) during the beginning of lunch, and the end of the day.
What I like about the elevator question is that elevator algorithms are their own science. But a lot of the complexity can be realized just by taking the time to think about all the scenarios you've encountered as an elevator user. And so given the typical technical interview situation, it's a good test of the candidate's reflection, ability to ask questions of the requirements, the ability to apply heuristics for good non-optimal solutions, and the ability to not let edge-cases dominate the thinking.
I was never quite sure whether the 'waiting for input inside elevator' actually existed, because every once in a while the elevator would get summoned by someone else before I could give it a destination when inside.
I guess this is a bit off-topic, it's just to say that even very simple logic can solve the problem, in a possibly infuriating way. ;)
Are they? I think they have analogies in task schedulers. You have things like fairness guarantees (maximizing number of trips made or people moved is not ideal, as that could keep someone waiting for an elevator for weeks), livelock (elevators being stuck on short trips between floors 1 and 2 with people waiting forever on floor 55), elevator/CPU affinity (if we know it is highly likely that the person on floor 14 wants to go to floor 7, we might not let the elevator going past it to floor 3 stop at 14, but schedule the elevator now a few floors higher that we knew will stop at 7 stop there instead, but if we are fairly sure he wants to go to floor 6, where the elevator is going anyways, we might want the first elevator to stop there)
For the "ground floor in the morning" thing, task schedulers can switch behavior, too, for example by detecting the difference between batch-like and GUI-like programs. Extreme example (probably far-fetched): in embedded systems, the task scheduler could even know, say, when Wall Street opens, and make sure the cron task that should run then is paged in when it does.
This is also why many elevators perform very poorly under heavy load. You want to skip floors when the elevator is already full.
The idle time question is an interesting/important one, arguably the more important for elevators for relatively low utilisation, where the capacity is there, but you wait for a lift as they are all elsewhere.
In my building it seems that not only do the elevators not have sensible 'resting points' at peak times of the day, it seems that it also doesn't weight requests at all, so you get a massive number of people waiting at the ground floor, as only one lift is sent at a time.
And even if there are multiple elevators waiting at the ground floor, they will only open one at a time, once another lift has left.
http://en.wikipedia.org/wiki/Petri_net Simulation: https://www.youtube.com/watch?v=59jW9dngt_c
A good starting point might be to have a stopping distance table for whatver the definition of "graceful stopping" is. Then, the elevator stops iff at the time the button is pressed, the stopping distance is less than the distance from the top of the elevator to the top of the floor, i.e. it doesn't need to backtrack.
[1] http://www.worldcat.org/title/vertical-transportation-handbo...
http://bahmutov.calepin.co/functional-javascript-interview-q...
{:unowned-properties #{:baltic} :players [{:owns #{:atlantic}}] :property-states {:atlantic {:hotels 0 :houses 2}}}
Etc....data driven design...wins every time.