A Method I’ve Used to Eliminate Bad Tech Hires | Hacker News Reader

A Method I’ve Used to Eliminate Bad Tech Hires

medium.com

79 pointszerr93 comments

> I will purposely challenge their solution to see how they react.

> If they get defensive, it’s an immediate no-go. Remember, there’s a difference between defending which is good and being defensive which is bad. The difference is that the former is based on rational facts, the other is based on emotion.

Are you always able to challenge them based on "facts", or are you going to challenge them with your current world view? I can see someone getting defensive if your challenge is perceived as deliberately provoking.

If you're submitting something for review which is admittedly going to be used at least in part to decide if you are "good enough" to hire, it would be foolish not to expect to have to defend the choices you've made.

Using the author's definition, "being defensive" based on emotional responses is more a character trait than anything else, and a negative one at that.

I would much rather work with a slightly above average or average developer who is thoughtful and rational than a rock star who gets personally offended if you question their use of $DESIGN_PATTERN.

We recently started using the 'assignment' alternative to interviews and have had great success with it, although only a couple hires to go by so far. One issue I have with it though is that it asks a lot of candidates in terms of their time. Paying for that time seems like a great solution, as well as providing the other benefits you mentioned. Will definitely try this next time we're hiring.

I did a(n unpaid) coding assignment for my current job several years ago. Estimated time to complete was 1 hour, but I probably spent half my weekend on it because I wanted to make a good impression.

Once I was in a position to offer advice and assist in the interview process, getting rid of that altogether was the first thing I did. It took time, I got nothing out of it, and all it showed the company was that given 40-50x longer than necessary I could produce a working product.

Yes, this is another problem with it. However, from my (albeit limited) experience, it still beats classical interviews. Bringing people in and enforcing a time limit could help, but many people get very nervous in that type of 'exam' situation, so you lose a lot of the benefit (which is primarily seeing how they work in the real world, out of a pressure-filled interview situation).^

Personally, I prefer to make the assignments less open-ended than the ones in the OP. That way hopefully there's only so much time that could really be reasonably spent on them. It's still an issue, but (over-) paying for the "2 hours" helps too. The remaining downside is worth it in my opinion for the far better filtering it provides. (Also, classical interviews are no picnic for many people either. I know plenty of people who would rather spend all weekend at home working on something than be subjected to a regular interview.)

^Also we're all-remote, so in our particular case we really couldn't do that even if we wanted to.

Yes, always propose paid assignments, otherwise you'll miss many good candidates. Many of them will not explicitly refuse, but just move your company below in the priority queue... This happened to me - i.e. I did move the company back and eventually accepted another offer. If they had proposed me the paid project (instead of unpaid one), I'd immediately started working on it and continue the process...

Also, the OP makes a great point that the assignment should be a small, encapsulated, made-up thing. (IE, not actual work you need done.) It may seem like a waste to pay for such a thing, but you can just think of it as a (small) recruitment expense.

I would qualify this better. Most people take a day off for an interview. If you explained to the interviewee that they were to turn up at 9am, would be given a small project using X technologies, to be completed within the day to be ready for a code review at 3:30 and presented to the team at 4pm. Lunch would be provided and they would be paid for their time.

Another way to do this is to already have an existing mini project with a bunch of unit tests.

It should be functional but contains known bugs, of which, some are easy for junior developers, and some are much harder for senior devs. Whilst I don't like checking broken unit tests into source control, leaving a broken unit test in the source code is also another good test of the candidate, to see if they can fix the code to match the assertion.

It would also contain methods that a senior dev should be able to look at and then appreciate that there is a optimization that could be made to improve performance.

Everything is in a private Github repo that the candidate can be given access to and a week to play around with, giving people time (some people have families and busy lives) to complete.

They should issue a pull request when they are finished and their changes should be well documented.

Most companies have an existing product and developers are needed to support, fix and extend those existing products. It isn't often developers get greenfield development. They need to fix code written by other developers.

The really good developers:

- figure out how to clone the repo

- comment their changes well

- fix all the bugs

- figure out how to run the unit tests

- fix the unit tests that don't work

- optimize the poor performing methods

- manage to issue a pull request

If the developer is junior, then they might miss a few bugs and the potential optimizations, but if they can get to the point of issuing a pull request with some bug fixes and good documentation, then they have done pretty well.

Most importantly, check the developer's knowledge beforehand. For example, they might not have had Git experience before. If they can figure things out for themselves, then it shows they can solve things for themselves without someone else having to hover over them constantly.

Finally, if they ask questions to validate their assumptions, the developer isn't going to be one of those devs that goes off on left-field for 3 days, only to find out they got the wrong end of the stick. People that think they understand but don't, are dangerous and expensive to your organisation in the long run.

Atlassian does something pretty close. Once you've successfully passed the first step (automated online assignment), you receive a maven project by email 15 minutes before the interview and then use Skype screen sharing to show what you're doing. It starts with writing a unit test to show that some method is misbehaving, then fixing it, and it goes on.

I really enjoyed it even though I failed miserably, and I believe I would have done much better if it hadn't been done in one hour. The exercise was really novel to me after having interviewed with a dozen companies who think Google-style coding questions are a silver bullet.

Having a pet project for a team is a really important thing. You have some way to bond that is not so stressful, you have something to show for, some reason to spend friday nights and weekends with colleagues, can show of some of _your_ skills to potential hires, and a good chance for an interview task.

What company uses this methodology?

e2x did exactly this. We had a tiny web app with one controller, and the candidate had to implement some small feature. Looks easy. But some tests are failing, some are passing incorrectly, there's misnamed code, dead code, and various other traps. We did quite a bit of rescue work on projects screwed up by other consultancies, so it was even somewhat realistic.

At Pivotal Labs, we have an interview project where the code works and is tested, but is a horrifying big ball of mud. We ask the candidate to add a few features, but really, they need to refactor heavily first.

For someone who is a full-time employee with no freelance work on the side, this would add significant complexity to my income tax calculations compared to what I currently I have to do (which is literally nothing). I'm not sure i'd go through the trouble of that for a job interview, i'd probably actually be better off financially by doing it for free :)

That said, it is a great approach.

Maybe tell them to donate it to X for you. That way they don't get out of paying and someone in need profits from your work.

Yes, personally, I wouldn't do anything like this for less than a few thousand unless for some reason I found the assignment intellectually interesting.

Two hours of your time is almost certainly not worth "a few thousand" dollars.

Oh sure, but $200 is not worth the accounting / liability / legal hassle. Plus it depends on the problem. If it's a tricky problem that wants me to solve something that they're trying to solve (I've had that happen a number of different times), than I'm not just putting in 2 hours. I'm providing them with the wealth of my entire experience.

Solving someone's problem is worth...whatever its worth. Nothing to do with time spent. Consultants learn this quickly or go broke.

There ought to be a "donate the money to a charity of your choice" option for this eventuality.

Yeah, or there could be a tax-free allowance for such a thing.

By using this method, you exclude candidates that cannot (or are unwilling to) allocate the time necessary to do the assignment, plus the time needed to take care of tax-related issues just to be able to write you a receipt.

In other words, you are excluding people who have a job.

Also, I fail to grasp this idea that potential candidates must prove their worth through numerous hoops to compensate for the interviewers lack of skill in judging character and competence (i.e. seeing through the bullshit).

In any hiring process not only the interviewer is interested in avoiding a bad hire, but the candidate is also interested in avoiding a bad employer. Both stand to loose from a bad choice, but that's why even the most strict employment laws allow for an experimental period.

Certain hiring requirements tell a lot about the culture of the company. Avoid sending candidates the message that you have a culture of distrust. You may end up loosing the ones you wished to hire.

> Also, I fail to grasp this idea that potential candidates must prove their worth through numerous hoops to compensate for the interviewers lack of skill in judging character and competence

They say:

>> It’s very easy for someone to embellish (to put it generously) their resume, and coding trivia can be memorized

There are arguments about the efficacy of interviews, but judging technical knowledge is not one of them.

If I was hiring for a Java programming position, I might start out with an easier question, like what is the difference between an interface and an abstract class. Or ask what a class was, or an object, or a constructor or method. Or what a static class variable was. Plenty of people with resumes that look good will stumble through these questions, whereas others will give clear answers.

Perhaps you can't tell if someone knows all they need to know or can do a good job, but it is certainly possible to see who does not have the level of technical knowledge you want.

Personally I'd ask anyone who asked me to do this to also complete a project over the weekend. Just to see how committed they would be to his employees problems.

I'd ask them to do my laundry and my shopping and entertain my cats. Cuts both ways right?

> but it’s really hard for someone to fake actual skill

That's a funny way of putting it. If one can fake actual skill... well, hire them!

As a sales representative I suppose :).

Giving them the weekend / take home is a bad idea. You might get some enthusiastic fellow who spends 10 hours on something but claims to have done it in 2 hours. You really don't want that. You want someone who can do it in 2 hours and not lie about it, otherwise you end up with a workaholic who will burn out in short order trying to live up to unrealistic expectations.

Give them a cube and pay them to work in the office. Stop by to see how they're doing and interact with them.

I agree somewhat with what you're trying to do, but I really dislike the way you're doing it.

For example, I'm one of those developers that customise my development environment heavily and that have spent 99.99999% of my life using a regular keyboard (ie. not on a laptop). Now you want me to use a vanilla environment + laptop keyboard, which combined roughly means that you're going to divide my development speed somewhere by 4 to 10. Nevermind that the laptop you're going to give me will probably be a half-broken extra that's both underpowered and unmaintained...

Of course all interviews are a negotiation, so the onus would have been on me to inform you of these factors.

How much code do you type out daily that a change in keyboard reduces your productivity by half?

Unless you pay your programmers by the hour, how long they take to complete a project should not matter (within reason, i.e. a weekend). Judge the quality of their code, not their coding speed.

In fact, intense time pressure has a very adverse affect on cognitive tasks.

Well, except for the fact that taking 10 hours to perform a 2 hour task doesn't scale when I want them to do 40 hours of work a week. I really, really do want to judge people, in part, by their coding speed.

If two people say "yea I spent two hours on this" and one of them shows something that they actually spent 40 hours on, you're probably going to get the impression that they are the better developer because their code should be better. Is that a useful comparison to make between the two of them?

Well in that case you won't be hiring anyone with a job.

Lots of people take the day off to interview. Day long interviews are pretty standard these days.

"If they get defensive, it’s an immediate no-go. Remember, there’s a difference between defending which is good and being defensive which is bad. The difference is that the former is based on rational facts, the other is based on emotion."

And whose call is it to make that distinction? Presumably the author of the optimal-hiring-procedure #122327 that has been posted here.

A pretty good example of "being defensive" would probably be accusing the interviewer/author of not having sufficient knowledge/skill/ability to make that distinction.

Yeah, there's a very fine line between the two, and very little attention was given to that in the article. Is "being defensive" the new "bad culture fit"?

I like this a lot. Works well for both parties.

Pro tip for candidates: after you solve the problem, when you share it with the team this is a good time to show how friendly and open you are. "Gosh guys, I've never been much good at TDD, but I limited the code to less than 50 lines, so I thought for such a small project I could squeak by" or "What a fun project! Yeah, that screen design sucks. I suck at UI design. From a technical angle, though, it was fun and I enjoyed playing with the problem"

People want to know both how confident/competent you are and how you handle weakness. If you're an ace on the technology, can talk a good game, and enjoy having people help improve your work? You're a keeper.

I do not like the methodology. I was subjected to it 3 times in '09 during the recession when nobody was hiring. I went to the next level of interviews in all the cases -- but did not get those jobs at the time.

Hope will never need to waste my time on that again.

It does not test the speed and quality with which a candidate can absorb new methodologies/ways.

Therefore it is perhaps, reasonable if the job assignment to maintain an existing code base in XYZ tool chain, or participate in a larger team where the architecture/design approach is done by somebody else.

well one shop i interviewed with found it necessary to put me to the test this way (except them being cheap did not offer payment). I disliked the long process and the fact that they would make me do work for free. so when opportunity presented itself i signed with my current employer, who offered me a slightly worse contract but trusted me enough to sign immediately on the first interview. i think the method is only appropriate if you are looking for high level people and offer a corresponding salary.

In all fairness, the author explicitly stresses the importance of paying for the work.

I would probably make the work product open source (voluntarily), and maybe over a longer period than a weekend. Would still pay 50-150/hr for the desired number of hours.

Interestingly, this is exactly the same method I use to weed out bad employers.

Maybe when the market's a bit worse I'll be forced to jump through one of these hoops. But right now...

This is one of these articles that has changed my thoughts on recruitment and makes sense when hiring mid/senior talent. Not sure the approach works if you are recruiting junior/graduates but even then I see this as just a case of extending the development time at a lower rate with lower expectations (which hopefully will be proved wrong).

Front loading the cost of recruitment brings major cost savings in the long run.

This gem:

> I may say please use functional reactive principals in JS to solve this problem, but the decision of using Kefir, Bacon, or RX is left up to them.

I'm all for this technique (I might actually use it myself next quarter). But there will always be a few bad apples who would go home and get someone else's help (or worse yet, get someone else to do the whole thing).

They would probably have difficulties answering questions about why they did something in a particular way, though. The conversation you can have afterwards is probably the most interesting part, also for the good apples.

The interview mitigates the risk. An alternative is to let do the work on site during half a day or so. This removes also the risk of a candidate polishing his code all the weekend.

I wholeheartedly support the approach of "get them to build something".

It also works really well for sysadmins: give them a subtly-broken system, get them to fix it documenting their thinking along the way.

> did the person write a test for every line of code? No?

Does anyone expect this for 2 hour assignment? Seriously!?

Yes. I've twice been criticized on "short" interview coding assignments for not having _enough_ unit tests. I had unit tests, just not enough in their minds.

One position was CTO and the other was a Software Director position.

Tech interviews are getting crazy.

Why is someone applying for CTO performing coding challenges?

Because the company is not really hiring a CTO. What they are hiring is a senior developer who will put in a full week of work coding, and then will be asked to put in another 40 hours dealing with management tasks. The "CTO" title is there to try to make up for the fact that the position is woefully under-compensated.

Exactly. "Well, it will grow into a CTO role".

They also didn't have an answer to "How often does the board meet and how long is my part of the meeting, typically?"

Plenty of shops insist on strict test first development. If this is one of them, then why wouldn't they insist on it in their hiring process? As long as it's an explicit requirement of the assignment, what's the problem?

Because even using TDD there are some things that just don't warrant a test.

The author didn't really imply that it was important

> The solution addressed your problem using the technologies and techniques you proscribed

Prescribed...?

Yep, he meant "stipulated" I guess.

He certainly didn't mean "proscribed," which would mean that the candidate used prohibited technologies.

I am with the commenters who have a problem with this being a weekend project. How about assigning the problem on a Monday, and reviewing the results toward the end of the week?

If they are asking you to do your interview homework over the weekend, it seems likely they'll be "asking" you to work unpaid overtime, weekends, etc on a regular basis.

The only thing I dislike about this is the short timeframe of interviewing on Friday, and having to work that immediate weekend on the problem.

I'm a busy guy, I have kids, and I like spending time with them (on top of the 2 hours I try to get in every day). That removes the weekend daytime.

Weekend evenings - well, realistically once everyone's fed, bathed, and had a bedtime story, we're talking 7:30-8pm in my household - are then for eating, spending some time with my wife (shocking, I know!), taking care of household tasks, and then going to bed, ready for the inevitable 6am kiddie wake-up call.

Yup, weekends are by far the busiest time of the week, and that's even when everything goes right!

Give me a week, interview on Friday, deliver code/presentation by the following Friday, then I'm absolutely game for it, will get it done, and will get it done well.

That said: I guess this may be a deliberate filter against people like me. Always difficult to know.

The only thing I dislike about this is the short timeframe of interviewing on Friday, and having to work that immediate weekend on the problem.

I suspect that that might be a secret feature. If you cannot work that weekend due to other, more important plans, then that is a strong indicator that you might not be able to work other weekends in the future.

Then it should be clear in the recruitment process that working on weekends is non-negotiable so people who can't won't apply for the position.

That shouldn't be a secret feature, they are wasting their own time and resources by hiding it to applicants.

Oh I dunno, maybe if you ace the interview they may skip that. Certain senior engineers who are in huge demand don't need to do such things. They have folks lining up to hire them without asking them to do programming assignments. I doubt the filter is used 100% of the time.

Hidden ageism. Hire 22-year-olds with no responsibilities, and not enough experience to know when they're being exploited, is the scam here.

"I suspect that that might be a secret feature."

Yes, agreed. It may also test, if you are really serious with your application. If you are, there will be a way to free some time.

This goes both ways: companies blow time to interview candidates, who just want to see how far they get. Or who plan to negotiate better terms with their current employer (happened to us recently, we paid flight, hotel, lost 3 months and a whole application round).

Candidates blow their time by implementing stuff for companies, who just want cheap code or who want the best solution for a problem they cannot solve, or simply re-decide their plans after you invested your time for the coding.

Both sides should find out how serious the others are. Payment for code is a good step. But if you pay, you can also ask for something, right?

s/serious/desperate/

Is your "application round" limited to a single candidate?

No, we had a couple, but after the one we offered the job to finally let us know he wouldn't come, the whole shortlist had already found other opportunities.

Exactly, apparently everyone involved in the process is expected to work on weekends too:

> make sure to be available for their questions via email over the weekend

> A key aspect of this is that everyone in the group MUST come prepared, having looked through the solution.

However it's easily tweaked for employers who want to do some flavor of this but actually respect their employees:

> Have the candidate make assumptions and document their reasoning (risks invalidating the final product but that's a slim chance)

> Meet late Monday or later that week

One point I haven't seen mentioned here: If I'm currently employed and interviewing, I am not going to take two days of vacation just so that I can interview with you, work on something for you over my weekend, then interview with you again.

Well, context is everything. In principle, I do not want to be expected to work overtime, and it starts at the interview. Also, I am not against taking a day of vacation for interviews (but this is maybe cultural). But if the job is really interesting, I can adapt. And I appreciate the fact that if am asked to work during a week-end, the person interviewing me offers to be reachable too.

to be honest, an potential employer who expects this isn't one I would recommend to anyone. If they expect you to put them before family they are not worth working for. Seen it too many times, once you become a doormat you always will be one.

One benefit of making the timeframe short, is that the candidate isn't pressured into spending way too much time on it. You may say that they should only spend 2 hours, but in reality, someone who wants to make a good impression is going to spend far more.

That said, I agree that weekends might not be the best for all candidates. Maybe tailor it to the individual case (E.g. ask them beforehand if weekends work, or they'd rather do it during the week)

We usually have a take home quiz/test, it usually takes one hour or so to complete, and gives us a shot at seeing how one approaches certain problems. I once had a guy that built a whole web based suite that acted like a tour, took us through each problem and then ran the code and spit out the result and also showed the code running in the test in a syntax highlighted view. WAY over the top and took way longer then an hour, we hired him though :)

The only thing I dislike about this is the short timeframe of interviewing on Friday, and having to work that immediate weekend on the problem.

Agreed. Another issue with the timeframe described: When is the current team expected to have a chance to look at the candidate's solution prior to the presentation? Stretch the timeline out a bit and everything becomes a lot easier.

If we assume you actually want to work for this company (because culture, what they do etc.) then is it too much to risk one of your weekends? Do you think it's not worth the risk?

If it's not worth the risk do you think it's a good hire for that company? Because in that case you clearly don't care much about the company and will jump the ship when next slightly better opportunity arrives.

Possibly I'm assuming too much and don't mean this to be personal but I'm really curious, is this the case or am I missing something?

Personally I feel about $100-200 fee as well, if the candidate is looking for compensation to build something fictional in 2 hours, is that candidate really care about your company, and is it a good idea to hire people who joins to your company just because you happen to be paying decent salary and they don't have a better option at the moment?

> Personally I feel about $100-200 fee as well, if the candidate is looking for compensation to build something fictional in 2 hours, is that candidate really care about your company, and is it a good idea to hire people who joins to your company just because you happen to be paying decent salary and they don't have a better option at the moment?

The money is more of a token to show that the company values the candidate's time and is not having them complete this coding challenge lightly.

> The money is more of a token to show that the company values the candidate's time and is not having them complete this coding challenge lightly.

That's my point. If a candidate takes it lightly because you didn't pay some dollars, does she care enough? If not, is she still a good hire? Do you want to hire someone who doesn't care about the job or the company enough to even fail to provide full effort on their first assignment?

Why should an employee care a lot about a /potential/ job, especially if everyone they're talking to is doing the same thing?

> If it's not worth the risk do you think it's a good hire for that company? Because in that case you clearly don't care much about the company and will jump the ship when next slightly better opportunity arrives.

And what's wrong with that? You are basically asking for a loyalty from the employee that is almost never returned by the company.

It's not a matter of risk. It's a matter of whether a candidate is at all able to work on something like this over a weekend. A weekend assignment presumes a certain lifestyle and implicitly discriminates against candidates who don't have that life. Primarily this affects single parents and individuals who are caretakers, but I'm probably leaving out a host of other reasons.

I agree. Just had this come up with an interview I did yesterday. We agreed that a week for a ~6 hr assignment was reasonable. I explicitly noted that I have a family so they were aware I couldn't spend more than a couple of hours/night coding :)

I also completely agree; this article provides a really good idea to weeding out candidates but weekends are a no go (especially if they don't give you any type of warning ahead of time; at least if there is warning things can be moved around and scheduled). A week is a good amount of time. Since they have to articulate their work when they return it's not like they can cheat; they're going to have to understand it well enough to talk about it and discuss the pros and cons of their design.

In many countries you can't simply pay people unless they're incorporated and can send invoices. You need to hire them. Otherwise it's tax fraud.

This is ridiculous. Getting paid would make taxes complicated and would count as taking a second job, something forbidden by many employers. On top of two precious days vacation. This might work for student interns but no one already employed would bother jumping through these hoops.

Is it really that complicated to claim a couple hundred dollars in miscellaneous income on your taxes? (At least here in Canada it would be a non-issue.) I'm sure applicants are welcome to turn down the pay if they feel it will be an issue for whatever reason (including current employment contract), but I can't see how that part is a negative. Certainly calling it ridiculous doesn't seem productive. I do agree that the potential of blowing your weekend is an issue that should be addressed.

Yes it is, in countries where normal people do not have accountants.

No, it's not.

Line 21. Other Income. $200.00. Done.

Edit: Not saying I would do the project. But tax reasons are not why.

Maybe this is an American thing. Here in the UK it would mean going from "I don't have or need an accountant, I just do PAYE" to "err actually I do now". Which no-one is going to do for £100.

Reply on news.ycombinator.com