Hacker News Reader

> A programming puzzle is simply the intersection of a work-sample test and an IQ test.

No: a programming puzzle is the intersection of a work-sample test with one question from an IQ test. IQ tests get a good handle on your intelligence by testing you over a wide range of near-trivial questions, each known to similarly measure intelligence, and then adding the scores up. By giving you so many opportunities to succeed or fail, they don't fall victim to the "I just couldn't figure this one out" problem. Whereas programming puzzles do that all the time.

Now, if your interview consisted of 200 programming puzzles, each of which should take roughly 20 seconds to solve, then you could say it's getting a fair handle on someone's IQ as measured through the lens of programming. But throwing two 10-minute puzzles into the interview does nothing to help you measure anything. The "scores" you get are black-and-white threshold filters on a set of numbers (IQ) that are almost totally middle-grey.

> a programming puzzle is the intersection of a work-sample test with one question from an IQ test

In the worst case this is true. But ideally the question will have various components, and various levels of difficulty. I'm reminded of a blog posted here some time ago that had the candidate split a string based on a provided dictionary of words (or something to that effect). The best candidates produced maximally efficient code including memoization or dynamic programming. The worst failed to implement a basic "split on the space" functionality. Besides these two extremes there was a whole spectrum in between. The point is to not give questions that are Pass/Fail, but a question that has a spectrum of possible answers allowing one to find the limits of the candidates knowledge.

That still doesn't help the candidates who just don't see the trick.

What if you know memoization and dynamic programming, but it just doesn't occur to you that day, that to correctly split on spaces (not adding empty tokens at the beginning and end of the string, etc.) you need to use a state machine?

You could give the candidate that "step" of the solution and continue on to see if they know the rest, but the candidate will likely be feeling a loss of confidence. They'll think "hey, if I couldn't answer the 'easier' part, why should I be able to answer the 'hard' part?" and then their mind will lock up even harder when they search it for the next step.

This is why IQ tests are organized as a series of orthogonal questions. No question is dependent on any other question, so you can just move on quickly from a failure and prove your worth on another question.

In fact, on a usual IQ test, you're even given five or six "presentations" of each question, each slightly different--like math homework questions with different numbers inserted. It happens quite often that you'll get stuck on, say, recognizing how one specific pattern of shapes go together, but will be able to see a different pattern instantly.

Interviews do the opposite of this; they are presented as if intelligence is a linear quantity, and if you fail on any part of a puzzle, you couldn't possibly be able to solve a "harder" part. In other words, interviewers are using the "error" theory of intelligence, rather than the "bugs" theory[1].

[1] http://news.ycombinator.com/item?id=5233497

>That still doesn't help the candidates who just don't see the trick.

This may be true, but at the same time candidates who DO see the trick get positive points. Because a lot of these puzzle questions simply require you to think outside the box, if you're forgive the use of that phrase, and honestly if you have hard programming problems to solve that is a killer skill.

I have (once) had the problem where my mind locked up on a puzzle question in an interview. (Disclaimer: Usually I'm really good at them, but then I usually get most of the IQ questions right too...) But I got the job anyway.

The point is that HOW you use these questions is at least as important as the content of the questions. If you ARE willing to work with the candidate on finding the answer, and you make it clear that it's not a make-or-break question, then I think it CAN be (at least) a positive filter for awesomeness.

Yes, sometimes you could let someone fall through the cracks that way. Nothing about the interview process is perfect, though, and if what you really need is someone who is awesome, then letting someone get away who IS awesome is much better than serially hiring people who AREN'T awesome hoping that you find one eventually.

That said, and as others have pointed out in comments: 99% of the projects I hear about on HN would NOT require awesome. They just require competence. And solving IQ puzzles isn't how you detect competence.

If your point is that these sorts of tests don't have any statistical power, I would whole-heartedly agree. Certainly you have to be careful with the types of conclusions one draws from the results of such tests. But they are not useless. The probability that a candidate will work out after having solved the problem with the best and most efficient solution is certainly much higher than if she only gave a middle-of-the-road solution. They are useful as positive filters.

And just to be a unnecessarily technical: if you accept the premise that the combination of appropriate CS knowledge and intelligence increases the likelihood that a candidate will successfully answer the question, then the fact that they answered the question successfully increases proportionally the probability that the candidate has sufficient CS knowledge and IQ (good ol' Bayes' theorem). This then increases the probability that the candidate will be successful on the job (citation in ancestor comment). That was a long winded way of justifying these types of questions as a positive filter.

I didn't say they're not a positive filter; just that they're a black and white positive filter.

Usually, hiring isn't about "passing or failing" candidates; it's about ranking candidates relative to one-another, and hiring the "best" ones (under whatever utility-function you think will help your business succeed.)

And putting people into "really impressed me in solving a puzzle" and "did not really impress me in solving a puzzle" does not much help in ranking people. It just gives you two buckets, one full of people who were both intelligent and lucky enough to get that particular trick that day; and one full of people who might or might not be intelligent--more intelligent than the people in the other bucket even--but who didn't get the trick that day. Personally, I'd rather not allow "luck" into my hiring process if at all possible.[1]

And this isn't so hard! I don't see why there is such a strong digging-in against the notion of changing away from "one or two long puzzles, that each give a single bit of informational entropy about the candidate's capabilities" to "hundreds of trivial, short puzzles, that together give a strong, statistically-sound measure of the candidate's capabilities relative to the other candidates measured."

I imagine the answer is that "the long-form questions give me a chance to get to know the candidate better, and see their [cognitive, and social] approach to problem-solving." But the OP (this guy: http://news.ycombinator.com/item?id=5264735) never mentioned knowing someone's approach to problem-solving as a dependent variable for their workplace success. IQ tests--relatively ranking someone's general problem-solving ability--is a dependent variable for workplace success. Work samples--showing, technically, that you can actually do the job as asked--is a dependent variable for workplace success. That's it. Rely on those, not your gut.

---

But, if I may go on a bit of a rant here...

This implies that convincing your interviewer that you'll be a nice guy who stays on-task, doesn't complain about problems, works long hours because they're "dedicated to the company", will talk down their successes (that is, be "humble") and won't demand fair compensation for their abilities, and is otherwise the "Agreeable, Conscientious"[2] kind of person that schools are expected to produce[3] and HR departments want to consume... is not a dependent variable for workplace success. There's no correlation. People can be great workers and produce their best work with their boss hating them and their coworkers thinking they're an arrogant jerk. And people will not just put up with them--but even approve of their behavior--if they get the job done better than someone who isn't like this. (Anyone who's ever watched House will probably grasp the concept intuitively.)

In fact, in software development at least, I suspect that this "not submissive to workplace domination" attitude can even be positively correlated with success--it's what gives people a sense of ownership and responsibility over project components, makes them passionate about making things right instead of just working, getting coworkers to refactor their own APIs so it doesn't hurt to consume them, etc. And I think that at least some software shops know this--it's, as far as I can tell, the original meaning (before it got diluted to meaninglessness) of looking for a "rockstar" developer. A clearer term might be developer primadonna.

But these types of people--unless they're lying--interview very badly, since, implicitly, an interview is an hour-long test of meaningless workplace submission. This probably isn't a bad thing--if you have the type of organization heavily infected with stress addiction[4], then this type of person--who will avoid the gametalk[5] that powers reciprocal dopamine transactions--won't fit in well anyway.

But if you're looking to build an organization full of people who care more about winning[6] than about "being a company", my advice is: skip the interviews. Just hand them the IQ+work sample test, send them into a room, and get them to do it. Maybe we could even turn this process into some sort of widely-recognized certification [a developer's journeyman certificate, basically] so the interviewed would only have to put up with it once. That would reduce the hiring process, simply, to "let's go out for a pint." And that sounds, in a healthy industry, like how things should be.

---

[1] For the same reason, I don't want to know what school you went to; luck [of parentage] significantly determines what schools you'll naturally end up at, with or without trying, but when people know what school you went to, a peculiar kind of nepotism/cronyism appears, with Harvard interviewers hiring Harvard graduates even if the Ohio State University candidate is otherwise equivalent, etc.

[2] http://en.wikipedia.org/wiki/Big_Five_personality_traits

[3] http://www.overcomingbias.com/2010/06/school-attitudes.html

[4] http://the-programmers-stone.com/about/the-dreaded-jungian-b...

[5] http://www.ribbonfarm.com/2009/11/11/the-gervais-principle-i...

[6] http://lesswrong.com/lw/nb/something_to_protect/

I appreciate your comment. I didn't realize you were actually advocating giving a bunch of small programming questions in place of one or two involved programming questions. I totally agree that this is a far superior way to test programming ability. Actually developing such questions would be rather hard I think, it seems it would be very easy to degenerate to trivia. Perhaps a set of questions similiar to that Quixey one minute programming challenge that was posted here a while ago (http://challenge.quixey.com/)?

I actually have a very ambitious idea to completely disrupt tech hiring. You have echoed much of my rationale. I don't want to go into detail, but the premise is basically to remove ego from the hiring process and make it far more statistically sound. Of course, this is also the very reason would very likely fail. Everyone loves to employ their pet "interviewing hack" to find world-class developers (who happen to be a mirror image of themselves). Most people will not give that up lightly.

The "loss of confidence" part, I don't see why that shouldn't be taken into account. If a candidate can't recover from failing, they're not likely going to do the same when they fail in their job. It also may indicate that the candidate is not teachable if they can't understand or have a hard time with your hint. Hints are a good way to progress an interview.

It isn't that people are insecure about their programming skill and intelligence. It's that they are irritated by tests which don't measure what they're supposed to. If any insecurity is involved, it's about ability to ace unrepresentative tests ranging from trivial nonsense to outright hazing.

You advise readers to "just get better at [the tests]". That's practical. But it goes right back to the problem, that many of the tests mostly index whether you prepared for that test.

Everyone talks about how many applicants are terrible at programming, or how many employees are mediocre. But at least offline, we rarely mention the complementary problem, that many companies are terrible at interviewing and hiring.

We talk as if being on the other side of that desk, in a position of power, means you are automatically good at interviewing. It isn't so. When a company treats candidates abusively - or takes bad hires who are extra slick, cheap, submissive, or similar to themselves - that isn't because of bad applicants. It's a failure of that company.

We can recognize that a defensive and condescending attitude is toxic in a programmer. But the same attitude in an interviewer is completely accepted. They're on top, applicants are on bottom.

> It's that they are irritated by tests which don't measure what they're supposed to

I disagree that programming puzzles don't measure what they're supposed to. Programming puzzles are a microcosm of what programming entails day-to-day. They test the same exact mental faculties. Sure, no one needs to write strcmp using primitives anymore, but the task of doing so exercises the same mental processes that are involved in orchestrating the various components when handling a web request. Doing these type of "low level" tests then allow you to include components that do measure one's mental flexibility and creativity, i.e. intelligence.

Sure, we can give a candidate a test that exactly measures what they'll be doing on the job. But what if the job changes? What if we need them to use a language or a framework outside of their comfort zone? This is the reason these tests are not specific to particular tools. You want a programmer that has the intelligence and command of computer science concepts to quickly jump between various tools with little effort. Comparing strings and reversing linked lists do absolutely measure this capability. No, its not testing whether you memorized how to do these things, its testing whether you have the mental flexibility to figure it out on the spot.

Comparing strings and reversing linked lists do absolutely measure this capability.

This depends entirely on the quality if the test. I suspect for many of these ad-hoc tests thrown together for an interview, the quality will be low and the overlap with other tests high, meaning they can easily be passed by someone who googled 'top 100 programming interview questions' and read through them.

Comparing strings or reversing linked lists are not actually things that most programmers would/should ever do themselves or even nowadays bother to learn or remember outside of a CS course, because the code has already been written in every standard library, but they are things that programmers might still be required to learn in order to pass interview tests, and you would thus be testing at many your candidates on whether they had memorised a few of the problems likely to come up, not on if they could work them out on the spot.

I suspect for many of these ad-hoc tests thrown together for an interview

If you assume the process is going to be done in a stupid manner, it doesn't matter what process you use.

"Oh, I looked through your open source code on github, but there only 10% of it was Ruby. We are an all-Ruby shop, not a 10% Ruby shop."

Sure, no one needs to write strcmp using primitives anymore, but the task of doing so exercises the same mental processes that are involved in orchestrating the various components when handling a web request.

I don't know what strcmp is in the way that you're using it. Does that make me a bad web developer?

Something orthogonal to this came up recently on Reddit, and it was observed that in the real world (anecdotally), 95% of professional programmers don't know what the Lyskov Substitution Principle is.

I think that most interviewers will happily give you the definition of strcmp if you don't already know it. If you're not a bad web developer, you'll happily come up with some code on a whiteboard that meets the definition.

Anecdotally, perhaps, 95% of professional programmers don't know the definition of the words, "Lyskov Substitution Principle". That's not the same as not knowing what it is.

Two weeks ago I interviewed at a subsidiary of a major web property, and "[w]hat is the Lyskov Substitution Principle?" was on a written test they gave me.

I would have told them that I had no idea what that was, but then told them what the "Liskov Substitution Principle" (named after Barbara Liskov of MIT) was. I mean seriously, if a company can't spell, do you want to work there? :-)

I think this is basically true. But a point worth making is that a "puzzle" problem is a much heavier thing than lightweight tests like FizzBuzz. If you hand someone a puzzle you're pretty much guaranteed (absent the rare, truly gifted candidate) to watch them struggle at a whiteboard for half an hour. FizzBuzz (or "reverse words in a sentence", or whatever) will tell you the same thing in a minute or two.

Basically, the Venn diagram regions for "can code fluently" and "can solve puzzles" have a ton of overlap. So test the first one, not the second.

You make a good point. The question is how much of an overlap is there? Is there value in adding a component of raw intelligence to the question (say, applying a data structure in a unique and non-obvious way)? It is reasonable to expect that the act of programming fluently entails a sufficient level of IQ such that there is nothing more to be learned from attempting to craft a question that tests intelligence directly?

Two bits of information could potentially answer this: does intelligence as a predictor of future job success drop off past a certain level? Also, what does the distribution of IQ look like for those who are fluent coders?

most people (even around here) are insecure about their programming skill,

This has a lot of truth.

I think also most people say "the best interview technique is the one I do best at."

I don't think it's anything to with insecurity.

I would consider myself a very good programmer compared to the hundreds I've worked directly with - think good quality, clean, well tested code delivered on time that delivers outsized value in production.

I am however useless at these mental gymnastic and logic puzzle things.

Having a high degree of programming skill is weakly correlated with these kind of interview questions, if that.

> most people (even around here) are insecure about their programming skill, and more-so their intelligence.

> Instead of writing endless blog posts and internet comments about how useless these tests are ... just get better at them! You'll be better off for it.

This doesn't hang together. Will getting "better" at programming tests & logic puzzles make you a better programmer? Make you more intelligent?

If not, doesn't that illustrate precisely why people seem to have a problem with these tests? Namely, that they are primarily a measure of preparation & studying for the exam?

It depends on your strategy for "getting better at these types of problems". You can cram and memorize all the algorithms you can, and that may allow you to pass some filters that you otherwise would not have. This strategy has its uses: besides being offered more jobs, you now have knowledge of these algorithms in your head and thus more tools in your toolbox.

Ideally, the candidate would learn about these "exotic" data structures, understand their properties, learn the cases where they are useful, and most importantly develop mental heuristics to recognize patterns of problems and appropriate classes of solutions. This will also help you pass these filters, and in the process will absolutely make you a much better programmer.

Fair.

Speaking for myself, I have purposely avoided studying specifically for interviews. My reasoning was that by relying on my existing knowledge + logic, I would give my interviewers better insight into my thought process and approach to problem solving.

If the benefits of spending time with this material are as significant as you say, perhaps I will revise my strategy (or, lack thereof).