Just say No to brainteaser questions at interviews
wilmott.com
wilmott.com
However, i hate when i get questions like "how many golf balls can you get in a double decker bus" and "whats your most embarrassing moment", both of these were questions i was actually asked in an interview. My answer to the first was "I dont know and i'm not going to try and work it out because 1) i dont have enough information and 2) its not relevant to this position. I'd happily answer hypothetical questions about situations that could plausibly come up during this position, but working out number of golf balls in a bus doesnt show you how i would solve the problem of the database running slow or validation checks not working correctly." For the second one i simply answered "I have never been embarrassed in a professional setting as i mainly just do my job and i'm not embarrassed to say 'I dont know' or to ask for help. The only times i've been embarrassed were in my personal life and those are stories not appropriate for an interview."
Needless to say, these were questions asked by the CEO of a startup i thought was absolutely amazing, i had sat through a tech interview with the CTO before the CEO came in and felt i did well with the tech questions. Once the CEO came in and asked me these questions, i wondered how an awesome startup was achieving this success behind the leadership of total and utter moron. I decided not to work there as i lost all respect for the CEO.
I used to take this a step further. The split second one of the silly questions popped up, I'd stand up and very politely end the interview. Reactions were priceless. Interestingly, I'd often get invited back, or on a couple of occasion offered the job after a follow up phone call, where I give the sort of responses you have given.
The problem I have is that these types of question suggest to me that the company doesn't really know what it wants to hire, and there for is not one I would want to work for.
The problem most people have is that they really need the job and feel compelled to play along, which I can only assume these interviewers take as positive feed back.
I'd imagine it's a good way to spot well adjusted human beings.
This, of course, assumes (positive) things about the company and the interview process, which may not be true for 95% of the sample as far as I know.
Whilst I agree somewhat with the sentiment, I don't think that's at all an unreasonable question.
1. You do very clearly have enough information to make a decent informed guess. 2. It is a relevant question.
It's relevant because it assesses your general sense of numeracy. Whilst that won't show how quickly you'd be able to deal with a database issue, it does give an indication of how you might go about making broad judgement calls in the absence of any hard-and-fast numbers. For example: Is it worth looking into this database issue, or is it likely to be insignificant compared to the network overhead?
In my experience some people simply won't answer those questions - not because they can't but because they aren't happy working that way. Of course, in some situations you absolutely do need people who will always work things through from first principles and attempt to give as precise an estimate as they can, but sometimes you don't.
Of course one can trivialy estimate the number if one assumes that bus is a cuboid and that a golfball is an incompressble sphere - I would would probly have to look up the theorems on Google as its a long time since I used this sort of thing in anger - though bak in the day I did correct a bridge design for the 4th largest consulting engineers when the Engineer had use 2d rather than 3d :-)
It also drew out people who may not be a fit for our desk's culture of camaraderie where answers to banter along the lines of "that's personal" or "that's not relevant to my job description" would be awkward at best.
Additionally, I have heard from consulting buddies (the heathens!) that it's can good way to see how well a candidate can bullshit on little to no relevant data.
In any case, I don't think it's empirically valid to conclude someone a moron for asking brain teasers in interviews. I never asked them whereas many people much smarter than me did.
But for that, data is needed and/or your estimate has to be better than the BS estimate
If someone comes up with "The potential market for flamethrowers is $500Mi" you better have the ability to check this before someone gets burned
I salute you, sir.
I'd say the latter.
It's frustrating to work with people who pretend they know more than they do rather than asking for help.
And when a CEO does it, it can ruin companies.
If this were the only type of interview question used the points you brought up would be a legitimate concern. Given that it was, in my case, married in a much greater proportion with concrete questions I maintain that it generated unique, useful information.
Traders generally have, prior to entering a trade, the opportunity to think through everything thoroughly and ask for help on the bumps. There are also times when the cost of the time for analysis is so great that a gut, if rough, call is needed, e.g. holding hard drive manufacturer equity as news of Bangkok flooding breaks. Being able to ball-park figures, after issuing a disclaimer about the uncertainty of one's estimates, is also a generally useful analytic skill.
It should also be noted that part of recognising the priority of things to be analysed is the degree by which they deviate from expectations - generating many of those expectations is a form of intuitive estimation.
But, as I said, it's mostly to estimate whether this is a person we'd like to spend 10-16 hours a day near. Constant righteous indignation would probably get annoying as fast as serial bullshitting.
What about estimating how many I/O operations in a second the DB supports? You'd be surprised how many candidates out there lack basic arithmetic abilities.
"How many users do you think will ever concurrently use a certain subset of our website?"
Actually, the question was phrased as "how would you architect Feature X?" This naturally led to "do we need a distributed approach?" And of course, the answer to that depends on the answers to the Fermi question and "how many users can a single box handle".
The goal is not to get a precise answer, the goal is to see how you think.
Another reason I like Fermi problems, is that the kind of people who I want work with (read: curious-minded tend-to-be-geeky programmers) tend to love them, and solve them for fun. I can't count the number of times we sat over lunch, arguing weird questions like how many planes it would take to move a certain river, or how many people it would take to eat a rhinocerous.
If someone is put off by these kinds of questions, I would question whether they fit with the kind of culture I think a group of smart people have.
The only utility I can see in that question is to assume that it's a question you're expected to dodge, and that the deftness with which you deflect the crazy question is a measure of your interpersonal skills. To launch into a story about this one time at band camp is presumably the worst possible answer.
Still, I don't think it's good even for that question.
The genesis of these kind of questions is http://en.wikipedia.org/wiki/Fermi_problem (checkout the classic "how many piano tuners"). Fermi approximation's purpose it to estimate without concrete data. In the end, if the numbers aren't right, you just need to replace the numbers, provided your method was correct.
I am not contending its relevance(largely irrelevant). I am just pointing out the origin.
He then asked me this: "You and your family of 9 are on one side of the river, there's a flash flood coming. There is a boat but it only holds 3 people. What do you do?"
I said: "I get in the bow, row to the other side, and wave 'good luck' to my family. See, I don't get along with my family very well."
He got a little agitated, and told me that in this case I love my family and want to save them. He asked me to try again.
I said: "I flip the boat over to make it more buoyant, put the kids on top and the adults hang on the sides."
He became angry: "There are piranhas in the river so you can't tough the water. Try again."
I said: "I lay the oars of the boat so that more people can ride in the boat..." He interrupted me by saying: "Just answer the question."
I said: "I have given you 3 creative and interesting answers. Since you are NOT trained in psychology you don't have any way of understanding my answers you feel frustrated."
He said: "This interview is over"
I said: "OK, but you know nothing about me".
I have refused to answer these types of questions since then.
If a flash flood is coming, why would your family be any safer on one side of the river vs the other?
In hindsight, the office was shutdown and I would have been there only a few months. The scary thought is that he's out there somewhere continuing with this interview technique.
I was asked once - "how many gas pumps are in Raleigh?". I asked back "do you mean pumps for the general public, or do we need to include commercial/industrial pumps too?" and "do you actually mean Raleigh proper or the metro Raleigh area? if so, how do we define that? All of the county?"
Next question was "go to the whiteboard and design a house". I took out the marker, started to draw, then turned and asked "who will be living there? Is this a single family or duplex? mobile home? stick built or modular?" and a couple other questions.
I was told I was the first people to ask questions before drawing.
I take the point of the brainteaser-type questions to be "how do you react to issues that come up where you don't know a lot of stuff?". For some positions, this is less important, but if you're a customer-facing position - even in development, you may interact with customers or other business units - how do you react when you get odd-ball "left field"-type questions? Do you keep your cool? Do you just react and say "that's stupid!"? Do you walk out in protest? Do you probe for more detailed information before making a pronouncement?
More than ever, I think 'brainteaser' questions are far more about personality-judging than intellect or even raw "problem solving" ability.
I asked if it had the phone number, they said yes, and I said the numbers are the same as in English... They said "okay, what next?" ... I was stuck. Next? Well, I suppose I could take a photo of it and ask for someone to translate it online... Or find someone in real life to translate it for me...
After a couple of minutes, they were ready to move on, disappointed, when I said "Well, short of asking for help, or Googling the phone number, I have no idea off hand how to find it."
"Oh, well, searching the number, that's what we thought you were going for initially."
It was. I'd thought they meant "next" as in "other than that method". Having the number was so evidently "problem solved" for me, that I didn't explicitly say "search out online". ... I suddenly felt like I was in grade school again, and my teacher had failed me for not showing the work in math class. I had the right answer, but would never get credit. I went on to flop the next question or two, as I vaguely recall.
It boils down to me not being perfectly clear, I guess. No hard feelings or anything, of course. Like I said, I threw myself off my game anyway.
I understand fermi tests are intended to show broad problem solving abilities, but imho, these tests are for lazy people. Talk to a candidate for 20 minutes. If you can't tell if he/she is suited for the role, or at least for a follow up interview, one of you is not qualified.
That said, I don't think they're lazy questions necessarily. However, I'm a physicist, enjoy Fermi problems (actually compulsively do the calculations when I hear a new one - a million golf balls!), and do think that order of magnitude approximations and quick unit changes are important skills to have developed.
I do think it's very lazy when they come out of a book or are riddles, because there are so many short approximations we can do at any moment. How much will the electric bill be for this office when it's August? or I just made a pot of coffee and forgot to place it back on the burner - when will it be too cold to enjoy? No accurate answer is needed or cared about, but the ability to predict important factors and come up with a list of contributing components is important.
Besides, if you had said "then I search the number online" then it's plausible the interviewer would ask what would you do if you didn't find anything relevant right away. For instance, if you search for my cell phone number, you get 0 results connected with me.
The flip side is when the interviewer doesn't communicate well. Or asks a specific question, but means something slightly different.
I've dealt with that before. Assumed they meant what they said, when the reality was, they were being more precise than intended. My fault for assuming I guess. I now try to clarify the intent, and make clear the precision.
He's like 'notice the small bevel at the bottom of the center key". Apparently he didn't know what it was called and that it was named after the inventor. Well, technically it's not proper Braille, but i would have taken that as a correct answer.
I just wanted to highlight that even though the candidate/me was aware of the basic concept. The answer just wasn't the exact thing the interviewer was looking for, hence it was wrong according to him.
This is the best part really.. Instead of "just say no" remember it's a dialog. If the brainteaser is interesting or you can quickly respond to how you would approach solving it then don't be an jerk about it. And feel free to have a dialog ask something back and see how they approach problem solving etc. Make it fun and interesting for all involved. If you're going to work with these people you might as well start right now in the interview!
I never take an interview with a potential client or employer without having a very clear set of my own questions for them and getting those answered.
I often ask off the wall questions or talk about a deep problem we solved recently and ask for the candidate for ideas on how they would approach the problem. I always share the answer we came up with. The reason for the question is not a perfect answer but to see how a candidate approaches potentially solving the question.
Last four gigs I had involved nothing even remotely resembling an interview.
I wouldn't even hire my best friend without him interviewing with 4 people (individually) who would be working with or close to him. I also wouldn't take part in the interview myself, to ensure that the treatment was 100% fair and merit based.
A personal recommendation gets you in the door to the interview faster, but you still have to prove yourself, and if you're really that good you'd want us to prove ourselves as well before signing on.
This is not a replacement for evaluation of a candidate on their own merits, and having a good in-depth technical conversation. But I like having something to ground my evaluations in as well and make them even slightly comparable.
For reference, I do the same thing when I get interviewed; I have a few questions I ask any prospective company so I have some comparative way to evaluate them. Never bad to try to have some metrics in your process.
I am constantly surprised how many people come out of interviews without asking detailed questions about the company, job, culture and work atmosphere.
You have to be more sure that that they will be a good fit for you than you have to be for them. You will be one of many: they will be everything to you, probably for years.
Worthwhile candidates tend to have criteria and the self esteem required to ask questions right back at you.
These questions are about having a logical process, regardless of where it gets you, and spelling that process out in a sequential fashion. It's more important to communicate "I do X, and check for Y. Then I do Y, and check for Z" than it is to arrive at a reasonably accurate Z. The added component of pressure (i.e., you've only got a few minutes, in the uncomfortable setting of an interview, in which to answer the question) helps (or allegedly helps) detect your ability to remain calm and logical under tight deadlines.
Basically, the question is asking "Under pressure, and faced with a seemingly insurmountable problem, will this candidate give up, or will he try to address obstacles in a rational, collected, and systemized fashion?" They're essentially variants on the Kobayashi-Maru test of Star Trek fame.
Furthermore, I don't agree with the apparent consensus that oddball "puzzle" questions would be better replaced with more technically relevant brainteasers. That's not the point. You have technical questions to test your technical literacy. These questions are trying to test your character. (Brainteasers are often non-technical precisely because the interviewer doesn't want you to be able to fall back on existing knowledge as an escape hatch from the question). Some brainteaser/puzzle questions are much more inane than others, but the exercise itself isn't entirely worthless.
I have never asked a puzzle question in an interview. If I were compelled to do so, I would never hire someone purely on the basis of his or her performance on one. It's one of the least important variables in a hiring process. But I'd consider with some skepticism anyone who outright refuses to answer a brainteaser, or who gets totally flummoxed by one, or who gives up without at least attempting to work out an approach.
Ultimately, no single type of interview question is flawless. That's why you have a wide variety in your arsenal. You've got technical questions, case questions, puzzle questions, whiteboard questions, and even the oft-derided "Tell me about a time when..." questions. In isolation, none of these types is sufficient. In combination, they test different aspects of a candidate's thought process and preparation.
Sure, but then, almost all interview questions can be similarly gamed. If an interviewer asks me to "walk him through my resume," or to "Tell him about a time when I did X," does he really expect that I haven't thoroughly prepared and rehearsed those answers? And if I truly haven't, then that's saying quite a bit about how seriously I take the process.
I'm not suggesting that brainteasers are as valid as other question types. There are plenty of question types that better get to the heart of the candidate's aptitude, experience, skills and capabilities, domain knowledge, and so forth. But everything can be rehearsed for. I don't think that's a valid categorical critique of the puzzle type.
Better arguments against puzzle questions, IMO, are: 1) They might select for good bullshitters, as opposed to good thinkers; 2) The opportunity cost of spending time on a puzzle question is the time that could be spent on a more relevant question; 3) Some people just aren't good at (or interested in) puzzles, but that doesn't mean they're not awesome on the job -- ergo, passing on a candidate because he flunked a puzzle is much sillier than passing on a candidate because he flunked a technical exercise. (Conversely, loving a candidate because he kicked a puzzle's ass is even sillier).
I was quite dismayed when I called them out and they didn't come back with new questions or made up some on the spot. They simply decided to drop that part of the interview and move to the next section.
I've seen too many good people (friends and students of mine) miss out on jobs simply because they failed these type of problems.
With this I would absolutely concur. I have more than a few friends who are smart, and who are rock-solid performers, but who just don't have a knack for brainteasers. I would hope than an interviewer is able to suss this out in an interview, i.e., make the determination that a person is great in every aspect, would probably be a great fit, but just happens to suck at puzzle questions. Sadly, I'm sure it's more often the rule than the exception that an interviewer will reflexively pass on someone who flubs a puzzle.
And I agree with you that puzzle questions require flexibility not only on the part of the interviewee, but also on the part of the interviewer. If you're just tossing out questions based on a book of canned puzzles, don't be surprised when you get canned responses. The exercise becomes entirely meaningless when that happens, and there's really no use in proceeding with it if it does.
http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%...
sums up, current to 1998, much of the HUGE peer-reviewed professional literature on the industrial and organizational psychology devoted to business hiring practices. There are many kinds of hiring screens, such as resume reviews for job experience, telephone interviews, in-person interviews, checks for academic credentials, and so on. There is much published study research on how job applicants perform after they are hired in a wide variety of occupations.
The overall summary of the industrial psychology research in reliable secondary sources is that two kinds of job screening procedures work reasonably well (but only about at the 0.5 level, standing alone). One is a general cognitive ability test (an IQ-like test, such as the Wonderlic personnel screening test). Another is a work-sample test, where the applicant does an actual task or group of tasks like what the applicant will do on the job if hired. Each of these kinds of tests has about the same validity in screening applicants for jobs, with the general cognitive ability test better predicting success for applicants who will be trained into a new job. Neither is perfect (both operate at about 0.5 level in validation studies), but both are better than anything else that has been tested in rigorous research, across a wide variety of occupations. So if you are hiring for your company, it's a good idea to think about how to build a work-sample test into all of your hiring processes.
For legal reasons in the United States (the same consideration does not apply in other countries), it is difficult to give job applicants a straight-up IQ test (as was commonplace in my parents' generation) as a routine part of a hiring process. The Griggs v. Duke Power, 401 U.S. 424 (1971) case in the United States Supreme Court
http://scholar.google.com/scholar_case?case=8655598674229196...
held that cognitive ability tests used in hiring that could have a "disparate impact" on applicants of some protected classes must "bear a demonstrable relationship to successful performance of the jobs for which it was used." In other words, a company that wants to use a test like the Wonderlic, or like the SAT, or like the current WAIS or Stanford-Binet IQ tests, in a hiring process had best conduct a specific validation study of the test related to performance on the job in question. Some companies do the validation study, and use IQ-like tests in hiring. Other companies use IQ-like tests in hiring and hope that no one sues (which is not what I would advise any company). Note that a brain-teaser-type test used in a hiring process might be illegal if it can be shown to have disparate impact on some job applicants and is not supported by a validation study demonstrating that the test is related to successful performance on the job. Companies outside the United States are regulated by different laws.
Thanks, tokenadult. This is an interesting result, and may give pause to some of the readers here.
Still, don't give Al Sharpton any ideas, or brainteasers may be the new frontier for anti-discrimination lawsuits.
Perhaps the end-game is a system where job interviews are illegal; instead, you tell the Department of Labor your needs and the Department of Labor (in fourteen to eighteen weeks) sends you your new employee, chosen according to race, gender, sexual preference and [if at all possible] qualifications.
Anyone who's taken one of the commonly accepted IQ tests will recall that it's full of sequential pattern matching, word-unscrambling, and so forth. And I realize that most here have probably not taken the LSAT for law school, but that test is essentially a string of interview-style brainteasers. (The LSAT was initially derived from IQ tests, though it should be said that this topic is not without considerable historical controversy).
That said, I lived and worked for a few years in a country where IQ tests were the norm for hiring people. When hiring for my own team (C++ devs), I did not use them as I found they were a very poor filter with both high false positives and high false negatives in terms of indicating whether or not you should hire a particular person.
So not much better than a coin flip?
So no, a coin flip is far worse.
However, using a coin flip rather than a GMA score on a prospective employee would imply a score of either 0 (missed 'em all) or 100 (perfect), and thus would likely have 0 correlation with the actual employee performance, rather than .51.
So your conclusion is right even if your analysis is not.
The math puzzle had a definite correct answer that can be arrived at by anyone remotely qualified for the position, but which causes even statistics undergrads to scratch their heads for a minute. The real purpose of this is to see if someone is able, willing, and perhaps even eager to actually think.
The programming test does not have a single correct answer. As far as I know there's always a trade-off depending on the importance of competing priorities. I tried to lead them into discussing the design differences to handle the different scenarios. It's very interesting to observe the thought process. Some people think of one fairly workable design and then they are done. Period. Other people realize that a good design in one scenario is suboptimal in another and try a new design. Some go farther and look for an 80% solution to both.
You can call this screening to some degree. It's also about discovery, to be part of the larger picture of how this prospect would work with our team.
The expression I would want to maximize would be (r/(r+b))+((50-r)/((50-r)+(50-b))), where r and b are integers between 0 and 50 inclusive (I've forgotten the calculus required for this).
Personally, I'd make the candidate solve a decent intermediate level problem using a computer (and Google too). Give him/her the closest approximation of the typical job environment.
The first time I heard of it I was surprised about how counter-intuitive it seemed, but it didn't baffle me completely because I have a strong information theory background. From an information-theoretical point of view, it's not completely counter-intuitive that new information can change probabilities a posteriori, particularly when there is mutual interdependence.
I can only see you coming across as an unpleasant person if you just start asking the interviewer how many golf balls fit into a bus right after they ask you why man hole covers are that size. And then there are companies like google (and the many who blindly copy what they know about google's process) and the few available details about their interview process all point to the fact that these brain teasers are really important to them for various reasons, so that attitude or just not answering these questions at all or asking them in return will very likely rule you out there - which is ok, of course, if you do not really care much about working for them or not. But in most companies, especially those using these dreaded brain teasers, I doubt this will "fly" and in most startups they will just talk shop with you anyway.
And while these brain teasers might or might not tell them something about you, you asking the interviewer a brain teaser is most certainly not going to tell you anything about the company other than, well, they have a pretty default and quaint approach to interviews. So what remains is definitely a bad after-taste and if that is what you are going for then good for you. But don't forget, most HR drones are pitiable people who have to make impossible decisions about people they don't know based on all sorts of nonsense and lies and then they are responsible when the new hire turns out really bad... so, no wonder they rely on superstition and obscure magic to tell them the future they can't possibly know.
Most importantly: If you are really that good and they all want you, you shouldn't even have to deal with HR to begin with; you should be aiming for that. So the whole thing is a completely moot point and the OP comes across as quite arrogant.
Why is one any more unpleasant than the other? An interview is a two way street.
If the company values brainteaser solving enough that brainteasers are asked on interviews, why should someone considering working there just assume that everyone there is capable of doing them? If my job were really similar to solving brainteasers to the point where they are being asked on interviews to judge my employability, I don't want to work there if my potential coworkers suck at brainteasers, because then I'll just end up with an unfair workload picking up the slack for them.
And if the company doesn't really value brainteaser solving ability, having that pointed out to them by turning the tables is perhaps a very useful lesson for them whether they realize it or not at the time.
Anyway, surely the ideal thing to do would be to play along with the interview, answer the stupid question as best as you can and when the interviewer ends with "any questions for us?" you ask them why the felt it was necessary to test you on a general-purpose brainteaser with no relevance to the job when they could have asked something more relevant, like "x". This way you show that you roughly know what you're doing, you're aware of how the interview went and you've provided a question that will make people sit up and think.
"How many golf balls can you fit in a bus"
"How many piano tuners could be employed in Seattle?
These questions do have everything to do with a programming job, it tests your ability to take an abstract problem and break it down logically into reasonable parts. It's called a Fermi problem and if you are incapable of thinking through them, I don't want to work with you.
1. Don't know the volume of a golf ball
2. Don't know the volume of a bus
3. Don't know the amount of a bus's volume taken up by seats and other protrusions
I think that's a little silly. I strongly doubt a single one of those will make you a better programmer. Likewise, problem 1 is a physics problem. If you're applying for a job programming physics simulations or video game engines, it could be relevant, but otherwise I don't see how knowing the physics of water would be helpful in the day-to-day experience of programming.
I see how the underlying skills of solving the problems could be helpful, but in practice it seems like brainteasers tend to be more trivia than problem-solving.
You are expected to demonstrate you can roll the concept around in your mind and make progress toward answering the problem. When you do interviews you'll find people who literally can't think to save their own lives. Their minds have atrophied to the point that the only way they can solve a problem is to have their hands held by a competent person and told "now i need you to find the volume of a gold ball", at which point they would have to be instructed how to do even that.
There are some idiots out there who have brains that have gone into coasting mode. No new learning, no new problems, everything they learned up till this point is the only thing they are capable of. You want to weed these people out quick. This question does that wonderfully. The incorrect answer is "ummm. I donno, and i don't know how to start.... Hold me".
I don't think that's true. I love programming, and these questions make me yawn.