Hacker News Reader

Comment by sillysaurus3 | Hacker News Reader

While there are a variety of ways you can conduct a programming interview, the goals typically are the same:
Will the candidate be able to write working code if they join the team?
Can the candidate discuss code and problems with the people they'll be working with?
Can the candidate reason about arbitrary problems and constraints?

Is the candidate someone we would enjoy working with?

All of these can be solved directly by asking the candidate if they'd be comfortable doing an actual task on the actual codebase, having them sign an NDA, and then giving them a bugfix to perform. They're compensated for their time, of course.

I'm doing it right now as a candidate and it's by far the best interview experience I've had. The others were all artificial.

A lot of people have a problem with this model for one reason or another, but in terms of effectiveness it's probably at the top 1%.

View on HN

Almost. Instead of doing this, take an old set of bugfixes from the tree, 1-3 of them, give the same bugfixes to every candidate, and come up with a standard rubric for judging the fixes. (If it's an open source project, synthesize 1-3 bugs, using the history of bugs on the project as a guidepost).

This is a "work sample test", and it's the gold standard for judging technical qualification. Firms should rely on them to the exclusion of subjective interviews.

Within reason, to a first approximation, every candidate gets the same problems (you may iterate over time). You can judge candidates apples-apples.

You don't need to pay candidates for their time, because they're not doing work for you; you already fixed these bugs.

Yeah, I'm a big fan of your approach. Unfortunately it seems quite difficult to get anyone to do it. Most companies have interview pipelines that are well-established and immutable, so only new companies are inclined to try it. And even Triplebyte unfortunately fell back on a combination of abstract coding tests and amorphous "cultural fit" criteria.

It was a neat hack as a candidate to propose to the interviewing company that they do essentially the same thing as a work-sample test and hear them say yes. That only works if you propose they let you try your hand at doing their actual work. I wonder if it can be generalized?

I think asking candidates to do "real" work is bad for a bunch of reasons:

* It's intrinsically exploitative, even when compensated, because real work done under similar terms is paid far more highly than prorated salary (it also does weird things to salary negotiation, by extracting an up-front concession on rate).

* It requires different candidates to work on different problems, which means you're not getting a reliable record of how candidates do on the interview so you can tune it over time.

* Because the best candidates will generally refuse to work under these terms, most firms that do paid up-front work have a fast-path for elite candidates, which is an obvious process failure.

did you read the grandparent? you use the same bug by checking out a commit before the bug fix was applied.

Triplebyte's flaw is like that of many other take-home tests: they favor people who lied about how long it took.

When I did it, I had to implement a regex parser. I knew intuitively that the state machine approach was best, but if I read up on state machines and coding them, I would bust the 3 hour limit, so I did the best I could to deliver something working within the three hours.

Feedback: "Obviously the state machine approach is better, which you didn't use. You're not qualified."

Guess I should have just lied and gotten placed like the ones who did.

Time limits on these kinds of things don't make much sense to me.

The state machine approach can't handle perl-compatible backreferences (since true regular expressions can't use them), which apparently some people want. That's feedback you can defend yourself from to some degree.

When I implemented a regex parser for Triplebyte, the time limit was much longer than three hours, and I did use state machines. The feedback I got was "we thought you wrote a great, full-featured regex parser, but we need someone who comes off better in interviews".

You can already see that this is a broken technical qualification process. If you can "defend yourself" from a demerit on the challenge's rubric, you have a subjective process. The smooth talkers will glide past regardless of their abilities, and the gifted developers that don't have interview skills will get washed out by it at random.

If there's a point to outsourcing technical qualification, it's exactly to avoid this kind of problem!

> And even Triplebyte unfortunately fell back on a combination of abstract coding tests and amorphous "cultural fit" criteria.

I'm not sure when you went through Triplebyte's process, but I did recently and it definitely included a substantial practical programming section which involved writing an actual program and some debugging.

They do have the difficulty of not controlling the downstream on-site interview process, so there's a limit to how much they can innovate. Ultimately their goal is to find candidates who can pass company interviews, some of which are still heavy on algorithms.

I also agree. As someone who has recently co-founded a startup, this is something my co-founder and I have spent a lot of discussing and exploring.

My personal experience with this style of interview was that it really helped me to personally demonstrate my abilities. I was interviewing at a startup in San Francisco in 2012 who wanted me to do a quick version of "battleship" as part of their hiring exercise. I proposed, mid interview, "how about I help you solve a performance problem with your current system. What's your biggest issue right now?" My interviewer actually showed me, and we ended up solving the problem together and then deploying out the change.

The funny thing about that experience was that when we were done improving their platform, the VP of engineering was told what I did, and then had the interviewer go back and do the battleship exercise anyway. They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test". I did far more poorly with the battleship exercise than I did helping them on their platform.

Can't really blame him, it might seem kind of arrogant to refuse to do the task they give and propose one yourself, although it is pretty impressive that you could do something useful on a codebase you haven't seen during the code interview.

It's unfortunate, isn't it? We spend all this time constructing artificial tests when the real thing is all that matters.

> They needed to "benchmark" me against other candidates, and the only way to do that was with a consistent "test".

If your employer were ever sued by a candidate for discrimination and it came to light that you had received different treatment than other candidates, some very uncomfortable questions regarding the reason would be asked in the courtroom. An unfortunate bit of fallout from the litigious environment we live in today, but it is what it is.

Wouldn't it be (well, look) even worse if any number of the non-hires dud great on the battleship test?

It's also important to make sure people are well-rounded. If one can tweak an architecture to put a queue in the right spot, she/he may still be the person that cannot properly implement simple concepts like "separation of concerns" or forgets input validation or security checks.

At the end of the day, with start-ups I really value that mix of system and hands-on code. Without initial hires having both, it's harder to build a solid system while iterating so quickly. Amazing people skills are also key.

Over-leveling and bad hire situations are no good for anybody.

They're compensated for their time, of course.

Taking on paid work for another company, possibly a competitor, would put nearly anyone who did so in breach of their contract with their current employer, as well as complicating their tax situation. Not to mention the vacation time they would be burning on it. You will only be able to hire freelancers or people currently unemployed by this method. Do candidates set their rate, or is it just a token amount?

In my case, I offered to do it for free. (It wasn't their idea to give me a task on the actual codebase.) Almost nobody would be fine with that, but I want to work there. The other option was to hope that none of the other candidates seemed like a better choice. It's good to lock in an opportunity.

Perhaps it's unworkable in the general case. That would be unfortunate; it's the most effective type of interview.

I think it would be prudent for the interviewer to just say that "of course we will compensate for your time because this takes a couple of hours". If you happen to have a contract that forbids you from doing that, just say "Can't take any comp, I'm happy to do it for free". I don't think the key is compensation, but at least not assuming the subject will work for free. If you are the interviewer in that situation, give them the t-shirt and mug.

Unless your company was exceptionally unique in some way I'd probably just pass instead. It's awfully one-sided: you're just sending me some code and I'm supposed to invest hours on it. At least with an onsite interview our priorities are aligned: neither of us want to waste more time than we have to.

Yeah we all have different taste in interviews, I'd much rather spend 2h working for free than writing som tree traversal or fib function on a whiteboard for ten minutes. Interviewing is hard.

Is it? I feel like it's a skill you learn once when you're green and never have to think about again. Whenever I'm starting a new job hunt I just dust off my copy of CtCI, do a few practice problems on hacker rank, and I'm good to go. Yeah, it has almost nothing to do with my actual job but I've never felt it to be an exceptionally onerous process.

I used to think that, but honestly after recently going through a job search I think it's untenable for most scenarios. As a candidate, I want to be able to interview with multiple companies simultaneously—if I had to do a trial week with each company, I'd never be able to get multiple parallel offers.

There's also significant onboarding complexity which I hadn't factored in. One company had me do a trial period and it took me half a day to get productive—time which is a sunk cost for them (since they had to pay for it) and made me feel bad about my lack of productivity. In the same time period it took to onboard with their codebase/build tools, I could have easily completed a full interview loop with another company.

True enough, but it sounds like the above model isn't for you. It's for everyone who is talented but doesn't have the option to go interview with multiple companies simultaneously. Most devs are in that position.

I'd much rather commit to a single company that I want to work at and do this process to put me ahead of the other candidates. But that's not a decision that everyone feels the same about.

I get your sentiment. But this is quite problematic if you already have a job. It's not so easy to take a week or month off on short notice. What happens if you don't get your dream job? You can't just quit your current job.

pair programming should cut the startup time.

It would also substantially increase the cost of this interviewing method.

This is 100% the "best" way to interview/hire a person. A small company who screens someone for basic competence, then does this is great. It does has its drawbacks though

1. Requires a significant time investment on your part

2. Isn't the most efficient -- if you have someone work a week on a project, and decide they're not a good fit, you just lost a week.

3. You can't really have multiple candidates for the same position coding at the same time. They would feel like they would be competing (and really, they would be)

4. For the reasons above, this would never work for a bigger company.

By the time they are doing their week-long "trial", you should be pretty confident through screens anyway. Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site?

> Just ask yourself this -- have you ever said "no" to a person you've brought in to work on site?

I have. I've done plenty of trial weeks where the candidate didn't work out. They got paid for their time, but we didn't extend them further offers.

This is an interesting candidate-first approach, but I have some concerns:

This requires that candidates are experienced at the language or framework you use. Are you not prepared to take someone with different experience? What about graduates? What if their ability with one framework does not transfer to another that you end up assigning them to?

Not all bugfixes are equal. If you haven't solved it yourself before, how do you know how hard it will be? If you give candidates questions that vary in difficulty, how do you compensate for that?

Is the ability to dive into a codebase and fix (presumably) trivial bugs really what you should be optimizing for? Is it actually better than abstract algorithmic questions for determining if the candidate [can] reason about arbitrary problems and constraints?

How would you overcome your unconscious biases when answering the question Is the candidate someone we would enjoy working with?

After I proposed the idea, they said they liked it and were willing to give it a try. They offered 1.) slight changes to the UX, 2.) add a payments feature, or 3.) some back-end bugs that need to be resolved. They asked which of them sounds most appealing as a test.

This was perfect from my perspective, and after I signed the NDA I told them all of those sounded fine. I mentioned that fixing a bug is one of the best ways to get familiar with a new codebase, so I'd probably focus on those, but that I'd like to hear about the rest too. I can probably throw in some of the feature requests or UI tweaks.

That's a bit more candidate-driven than might be expected, but it's really wonderful so far. I feel like I have autonomy, while at the same time they'll get the data about whether I'm effective in the field.

The bias question is important, and I'm not quite sure the best way for us to collectively deal with it. All I know is that there seems to be far more bias in a traditional interview pipeline. One of my recent rejections came after being asked "A hammer and a nail cost $1.10, and the hammer costs a dollar more than the nail. How much does the nail cost?"

Obviously the point of that is to see how you think through a problem. But such problems are useless. They're abstracted from what we as engineers do in the field, and the candidate knows that their entire fate rests on how well they answer that question. Yet those are the kind of metrics real companies are currently using.

> One of my recent rejections came after being asked "A hammer and a nail cost $1.10, and the hammer costs a dollar more than the nail. How much does the nail cost?"

this just got me to see if i can still work through a basic high school algebra problem (yes, slowly, guess and check would've been quicker but less fun).

Algebra is how I solved it. :)

If the candidate isn't familiar with your particular language(s)/libraries/frameworks/etc, there may be a way where you can isolate a part of your process and provide a sample data set to the candidate and give them an operation to be performed on it.

> asking the candidate if they'd be comfortable doing an actual task on the actual codebase, having them sign an NDA, and then giving them a bugfix to perform. They're compensated for their time, of course.

At Google, we call these internships.

Well, more or less -- there are restrictions regarding enrollment in a degree program, the work is for a fixed duration rather than for a fixed scope, etc. But the goal from the company's perspective is largely the same.

I'd be surprised if internships at other companies in the sector don't serve the same purpose.

(obligatory disclaimer about how these are my opinion's, not the company's, etc etc)

doesn't work for those not still at university and there are strict rules in the USA about the role of an "internship" its meant to educate the intern not be a source of cheap labour

I never said anything about internships being a source of cheap labor. Please don't put words in my mouth.

While yes, an intern typically makes less than a full-time engineer, you have to realize that the intern reduces the productivity of the host engineer, who has to spend time training the intern, looking over the intern's work, and so forth.

The real value in the internship program is that it essentially creates a extended interview process after which we can evaluate the individual (given firsthand experience of how they operate in a work environment), and the individual can evaluate the company.

Rather than working on a real issue, I'd recommend a made-up task/feature, modeled after something they might actually do. Just put it in a repo and share to all candidates. It shouldn't take more than half a day's work. Then the next round is all about the candidates presenting their work and explaining their thought process.

It's simulated, but exemplary of real work. And easily repeatable. (Also no need for NDA, etc.) Just requires a thoughtful engineer to develop the task.

I wonder where companies or hiring managers can post their job openings that supports this hiring process sometimes called Trial Employment or Job Audition for Software Engineers. Looks like a great way to get the best hire and also prevent a bad hire, although it can only work for a very limited number of companies and candidates.

You can still simulate that within the more standard interview process. Pick a real bug or small task in your codebase and give that to every candidate. Obviously you'll want to complete the task for real in your codebase, so just keep an interview branch in your source control.

Reply on news.ycombinator.com