Three hundred programming interviews in thirty days

https://news.ycombinator.com/item?id=9766816

Which makes sense given that I tried out their online quiz this summer after reading this exact post.

EDIT: Here's a link to when it was posted to LinkedIn, June 23rd, 2015:

https://www.linkedin.com/pulse/three-hundred-programming-int...

No you haven't. You might have, but you'll have no way of knowing that until they've been on the job at least 3 months.

Personally, I think it's bullshit that I can show employers all the things I've built and they still expect me to write a reverse sort btree or whatever the fuck thing that has nothing to do with web development.

College grads are perfect at passing these tests and they have no fucking idea what they are doing, but will get more offers and better pay every time.

Experience counts and if you disregard it then I'm firmly in the "Fuck you, I don't want to work for you" camp.

Unfortunately, I've resigned myself to memorizing the typical interview questions or going into management. I'd love to stay a dev, but being ruthlessly quizzed about shit that you never use in real life makes changing jobs a pain in the ass.

There is a particular kind of dysfunctional candidate who once was a programmer, graduated to being a lead programmer, and then became an architect. As an architect they stop programming, and their programming becomes rusty. Once that happens, the feasibility of their proposed "architectures" becomes steadily less practical.

You can detect them by their first getting offended that they would be asked to program during an interview, and then failing to actually write working code.

By contrast people who really do program have no objection to being asked to program in an interview. To the contrary being asked to actually program is seen as a positive sign that they won't have co-workers to deal with who can't actually program. (This is an unfortunate experience that most programmers try not to repeat.)

To me, it seems like your team isn't that good at judging ability. To compensate, you rely on coding questions. This what the industry has been doing for years. What exactly is innovative here? I see the need for coding questions (and think that your method sounds pretty good) but too often culture fit & 'soft skills' are overlooked in favor of cramming as many technical questions as possible.

One of the best experiences I had on an interview involved no coding on-site at all. I completed a project for them, they liked it, I got an interview. 3 hours of culture/team/work philosophy. There were no surprises - if I wasn't "technical enough", I wouldn't have been let in the door. It took time up front, but at least the interview itself wasn't waste of time, PTO, and money.

The chart's y-axis is labeled "correlation coefficient" but the text blurb analyzing it describes the definition of R^2 instead (% of variation of Y explained by X). The clarification at the end does not disambiguate this. Also, how do you numerically quantify "interview performance"/Y?

Additionally 3 out of 6 of the variables have error bars crossing 0, which means that the correlation could be positive or negative but we don't know (it would handily fail a statistical test of significance at p = 0.05 significance). Why were they included at all?

Many startups (including statistical startups!) have been playing hard and lose with statistical analysis lately and it's getting annoying.

I have a big portfolio of technical questions, but I usually stick to a small set that I consider to be well calibrated by posing them to smart coworkers and seeing what they do with them.

I've found, counterintuitively, that the best indicators are the questions that deal with "programming in the small", such as coming up with a predicate expression to describe a test for a simple relationship between a few values, because either you can do it easily or you can't do it at all. And programming in the large depends on a lot of cases of programming in the small.

> 1. A fizzbuzz-like programming assignment. Applicants completed two simple problems. We tracked the time to complete each, and manually graded each on correctness and code quality.

Instead, I was presented with a single, rather convoluted, 1-hour long programming problem that was most definitely not anywhere close to the difficulty level of a FizzBuzz test (the solution they're looking for probably involved dynamic programming, which I honestly haven't had much practice with). This was before I even spoke to any humans on the phone screen or interview steps of the process.

I couldn't finish it in 1 hour (I kept going at it on my own time for another 2-3 hours before I finally arrived at a sub-optimal solution, at which point I had enough). I'm sure there are plenty of people who are much better at this than I am who can finish it in an hour. But I do want to question the wisdom of using something of this level of difficulty as a screening test.

Your clients seem to value the "Product Programmer" very highly. Using a question like that as a screener will weed out a significant portion of these "Product Programmers" before they ever get a chance to showcase their product skills. Most "Product Programmers" I know (myself included) don't do enough convoluted dynamic programming problems in their spare time to be able to solve a dynamic programming problem like that within an hour with any level of certainty (and yes, solving problems like that quickly involves either extraordinary talent or lots and lots of practice). Instead, in their spare time, these "Product Programmers" like to build things, and learn new technologies and techniques to help them build things in a simpler, more maintainable, and more productive way.

They'd always heard of everything, and they had a good idea of what everything was, even though it was quite far from the current task to hand. For instance, plenty of people have not had the opportunity to work with functional languages, yet still have a good grasp of why they're useful, what languages there are, who uses them, and so on. They tend to see coding as a vast field where you can make many valid decisions, each with their own pros and cons.

The kind of people I'd root out are the ones who merely see programming as a tool. Something along the lines of "I've got this data in a spreadsheet, I need to calculate this statistic, so here we go..." They almost always end up creating an unmaintainable mess, and they never mention any well known pattern. Quite hard to identify in an interview, though. Perhaps the lack of key words that suggest a deeper understanding?

(Hey, that's an interesting thought. You could machine learn what words good devs use vs less good ones.)

As for coding in the interview, I think this can only root out the absolute novices. It's quite hard to think of a short test that will demonstrate that someone can't write elegant code. Perhaps a take-home, open-ended test would work, but you're gonna lose people who don't want to spend a whole weekend doing it.

Today, you have to go through hours and hours of phone screens, technical interviews, team meet and greets, etc. You have to know how to implement a distributed KD-Tree with skip lists in brainfuck. And, then they nitpick the code on style. And, sure, you get paid 90-150k / year now. (Wow!) But, it's such a time consuming process. And, the smart (or motivated) people know how to game the interview anyway (there are lots of resources out there on how to beat the interview process). And, even when you ace the interview it's still often difficult to get a straight answer on whether your hired or not (strangely more so if you're over 40).

They say that software developers are in high demand... I say that people just hate the idea that software developers make a respectable wage.

But, the funny thing is, after working in this field for over 15 years, after working with 100s of people and seeing millions of lines of code, the truth is that the projects are still failing, and the code is just as bad as it ever was. And, projects still fail most often because of bad requirements, poor vision, and bad management. In fact, I drink the koolaid: I hire people who ace the interview, and you know what I've found? There's a 50/50 chance that they're just as good as the person who I hired who failed parts of the interview (sometimes I'm desperate). And, those people don't bring down the project - it's the guy in the corner office.

Luckily, I was smart enough to save a lot of money over the years. I'm just too old for this shit.

The claims are:

1. Performance on our online programming quiz is a strong predictor of programming interview success

2. Fizz buzz style coding problems are less predictive of ability to do well in a programming interview

3. Interviews where candidates talk about a past programing project are also not very predictive

Those claims may be true, but it misses a key point: no matter how well someone performs in an interview, it doesn't mean they'll perform well on the job. In particular, the post talks about hard skills (and does well there), but doesn't talk about soft skills.

It's absolutely true that someone can talk a big game about hard problems they faced that they solved, but failed to mention their use of Google or StackOverflow. So unique coding tests can definitely tell us whether or not a candidate can really code (though the tests need to be tailored to the sort of real work the candidate will face). However, programming is a hard skill (one that is easy to teach), and not a soft skill (one that you tend to learn over time or have a natural inclination for).

Case in point: early on we made the mistake of hiring a very talented programmer on the basis of his reputation and a brief interview. He was a disaster because he was completely unable to work without a very hard set of requirements. In the very rapidly changing, agile, environment we put him in, he flailed badly because he was good at implementing requirements, but he was very bad at projecting himself into the customer's shoes. Later, we put him on technically demanding tasks which required considerable talent, but had no ambiguity, and he was fine. In fact, he was able to implement tasks that would have been very difficult for other developers who were fine in ambiguous environments.

Soft skills are tricky and most employers have no idea how to interview for them. They require an understanding of the actual role someone will be hired for and the set of skills necessary for them to succeed. Are there tight deadlines? If so, can the developer prioritize tasks and make a case for delaying less critical portions beyond the deadline? Is it a rapidly evolving environment which makes it hard to nail down exact behavior? Are you on a team with strong personalities who often disagree and can the dev handle that? Is the developer required to report to non-technical people who have no understanding of technology and can't appreciate what refactoring is?

Or here's a fun one for you aspiring managers: we've had one project with a superstar developer who was so monumentally productive and so brilliant at his work that some very talented developers who worked with him nonetheless had negative productivity at times because they spent so much time playing "catch up" with the superstar and understanding how the project was evolving that they spent more time keeping abreast of the code than developing it. The code was "bus sensitive" because one brilliant developer was so competent. How do you manage that?

These and other soft skills are part of what you really need to evaluate because depending on your work environment, you'll have these and other soft skills as critical skills that a dev will need, but you almost NEVER see interviewers check for these because very few people are trained in interviews.

Pro tip: start reading up about structured interviews. They've been proven via multiple studies to have a strong correlation with employees being successful. Unfortunately, they're harder to conduct and require training to understand them. As a result, companies don't know about them or worse, ignore them, because it costs more money. But what costs you more money in the long run? Taking longer to interview or hiring the wrong person?