Hacker News Reader

Comment by ammon | Hacker News Reader

This is definitely true. The filters that companies and recruiters use mostly trend in a meaningful direction. There are a higher percentage of good programmers among MIT grads than SUNY Potsdam grads. But this fact does nothing to help the good programmers who look bad on paper who everyone is ignoring. This is precisely the problem that we're trying to fix with Triplebyte. We want to make actual technical evaluation (not credential evaluation) cheap and accurate, so we can just interview everyone.

View on HN

The filters that companies and recruiters use mostly trend in a meaningful direction. There are a higher percentage of good programmers among MIT grads than SUNY Potsdam grads.

Imagine how mistaken that would sound if you substitute "startup founders" for "programmers."

The reason you feel company filters are anything except random noise is personal bias. It's disheartening that your article notices that company preferences are essentially random noise, and then you don't really internalize that fact or take it to its logical conclusion:

Interviews where programmers do anything except work-hire tests tuned to the signal that the companies want are doomed to fail.

And it's almost impossible to do a work-hire test in an interview. A real work-hire test. The type of test where you actually do some work and then show that yours is effective.

We're so backwards that people probably read this and think I'm saying whiteboard work or something stupid like that. No. Real, actual work. The problem you're working on can be fake, but the problem needs to be literally the same type of thing that the company does on a day-to-day basis.

Want a 100% success rate? Do that, and then don't pay attention to anything but the results. Including whether the candidate has an MIT degree, or any degree at all. If you actually set up a work-hire test, a real one, and then stop asking about all the other stuff that doesn't matter, then you will get a 100% rate. Again, you have to tune the test to what the company is looking for, but that's not hard. "Here is an iOS app. Find and fix these bugs, then add a new feature."

How do you do a work-hire test for someone who already has a job, and is not allowed to work on another company's stuff?

> The problem you're working on can be fake

I'm assuming if it was a fake problem, it wouldn't be something the other company could actually use.

It's not about what they could use. It's about what they own and where it's going. If your employment agreement states that any code you write belongs to your employer and that you're not allowed to share company code with others, then a work-sample test essentially asks candidates to violate their current employment agreement.

Taking that interpretation would mean any interview where you write code would violate your employment agreement.

Employment agreements like that are certainly frustrating. I've had experience with a few.

One thing to realize is that your employer can't really retaliate. The hypothetical iOS app I mentioned is set up for one purpose: to let candidates show they can do work the company is looking for. Not only will the code be thrown away, but it wouldn't make any sense to use it.

I would say that if the new company is not in the same section of the industry, than enforcing that agreement is entirely unreasonable.

If all of an applicant's code is owned by their current employer would "whiteboard" style code also be prohibited in an interview? I guess I'm wondering where the line is drawn.

The cost of a real work test is much higher, both to the company, and to the applicant. Some companies take that approach (weebly for one), but it means that some of the best programmer will not apply, and it only makes worse the problem of having to aggressively pre-filter (based on credentials, or something else).

You are 100% correct that most research into interviewing has show it to be far less predictive than companies believe. However, I'll add that no one is doing the false negative studies that would really be required to show this. For example, Google has shown that among people who pass their interview GPA is not correlated with success. This does not at all mean that GPA is not correlated with programming ability. In fact, it almost certainly is. People with low GPA who pass google interviews have a lot of other things going for them. To really analyze this, you'd have to give a job to a random sample of applicants. No one has done this. My goal is for Triplebye to get large enough that I can.

> No one has done this.

Not for coding specifically, but there have been some studies correlating GPA with career success, and the correlation has been found to be very low. But the reason there are very few studies that look at this is that GPA A) isn't epistemologically meaningful B) isn't a measure in the mathematical sense.

"A grade can be regarded only as an inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite amount of material." -Paul Dressel

The best programmers do apply. Whatever you think a work-hire test is, that wasn't it. A work-hire test attracts the best programmers. It doesn't repel them. It's a chance to demonstrate their skill. It's also far better: anything is better than the fake contests they're currently forced to endure. Which would you prefer? Spend a couple hours remotely fixing some bugs and adding a feature on a fake iOS app, or spend all day playing "dodge the bias" for a hit rate of ~60% at the cost of a vacation day?

Why do you think the programmer from your article didn't get a job when you "sat back to watch him get a job"? You didn't tune your test to what the company was looking for.

Any YC founders who are reading this: Refuse to talk to Triplebyte until they set up a work hire test for your company. Force them to do it, and force them to work with you to tune the test to the work you need. You will get a 100% hit rate. And you'll notice something else: Your retention rate will go way, way up. Want to not deal with firing people? Get them to show you they can do the work. Nothing else matters.

The test needs to be set up so that the candidate can demonstrate they can do exactly the same things all other employees do on a day-to-day basis, or it's not a work-hire test.

EDIT: I thought that sillysaurus3 was talking about a work trial period (days or a week), but it seems they are talking about asking question during that interview that are similar to actual work. I retract me objection :)

If only it were that easy. I am not at all opposed to work-trial tests. They are almost certainly more accurate. But a high percentage of programmer will never apply to a company that requires one. Read any HN comment thread about the topic. It's very controversial. Lots of programmers are against it. Work-trials require MORE time upfront from the candidate, and of course still often lead to rejections (by design a hiring filter rejects most people). A failed work trial burns an entire week of vacation. An unemployed programmer (who is being paid the work-trial) may like it, but someone working another job may not.

If you're getting paid for a week of work, you can take a week of unpaid vacation from your day job.

No vacation days burned. No money lost.

Huh? You can't just take unpaid vacation because you feel like it.

Can't you? Your only obligation to the company is the time they pay you for. The time they don't pay you for is yours.

We're not longshoremen. Virtually nobody has a contract saying "just show up when and if you feel like it, and we'll pay you for that".

I don't think Swizec was implying that this time couldn't be coordinated in advance with your employer.

I think the point is that most employers would say "no."

I suspect that the flexible "unpaid vacation" scenario you're describing is much more rare than you think. If your team is pushing for a major release and you decide just not to show up, your days are numbered. If you spend the week working for another company, you are quite likely to be fired from the old one.

Ammon mentioned Weebly, above. They are one company that I've seen require an on-site week as a contractor. I know of a Weebly candidate who lost her current job because the employer considered the leave to be job abandonment. Thankfully, she got the job at Weebly. Perhaps Weebly's weeklong trial is effective for them at weeding out bad hires. But I suspect most good programmers would never consider giving up a week of their lives to an extended job interview.

Thing is, most companies will fire you as soon as they notice you're actively looking for other jobs.

And I wasn't suggesting job abandonment. Asking for a week of unpaid vacation is a different conversation than asking for a week of paid vacation.

> Thing is, most companies will fire you as soon as they notice you're actively looking for other jobs.

Only if they don't need you and think you are not being productive. So be needed and be productive.

You're assuming the rates are the same, and that your current company will allow you to take an unpaid week off.

Why don't we let the interviewee choose between a few interview methods? Or bypass it if they think their public code proves their ability?

I think this is the best solution. It does raise issues of how you keep a consistent bar between the various options, but I think it's better than the alternatives. Currently we let people do a project track where they do a project on their own time, or an interview track where they answer interview questions. We also give them a choice of interview questions in a number of areas, to try to increase the probability that we see a strength.

It seems like you're being forced into a false dichotomy of being either completely for work trials or completely against them, and that's unfair.

But if your biggest objection is that it will take additional time, that depends entirely on the nature of the trial. I was given a work trial at my first job and it took about 40 minutes. It didn't go as smoothly as I'd hoped, but I would much much rather have done that than spent 40 minutes answering brain teasers in front of a white board.

If I'm already taking a day off, I don't care if the interview lasts three hours or five. I'd rather work on a collaborative task with my potential teammates to see if it's going to be worth radically altering my life to take a new job. I understand that there are many programmers who are opposed to work trials, and they can say "I'm opposed to work trials and would rather demonstrate my abilities in another way."

It seems like your research so far has been very comprehensive, why dismiss an effective technique because some people are opposed to it?

It's really frustrating when people are clearly talking around each other, and either failing to notice or willfully continuing to do it.

sillysaurus3: "Spend a couple hours remotely fixing some bugs and adding a feature on a fake iOS app"

ammon: "A failed work trial burns an entire week of vacation."

You're not talking about the same thing!

sillysaurus3 is talking about doing a small amount of work that is similar to what the company does, under conditions similar to those you would have as an employee. You seem to be talking about short trial period contract-to-hire. Yes, the thing you're talking about sucks and is a huge turnoff to candidates, but (mostly speaking for myself here) what sillysaurus3 is suggesting is massively appealing.

> A work-hire test attracts the best programmers. It doesn't repel them.

I'm not the best programmer, but I'm comfortable I'd slot in the top quintile, probably top ten percent, of people that walk through a startup's door. And I'd never even return your call. No work-sample test I've ever seen would take less than four hours. That's a $550 opportunity cost for me at my standard rates. What the hell makes you think you deserve that for free? You aren't Google and you aren't Facebook. You need me more than I need you. And almost everybody else who's like you--and given the way you are acting, probably you too--will forget about me or blow me off at some point in the process, wasting my time even further.

On the other hand, an in-person, discussion-based interview isn't work and isn't priced as work; not only does it tells me about the company and whether or not I actually want to work there, but I enjoy meeting new people in a professional setting (there are multiple companies where I've turned them down, but have become friends with people I've met through the process!).

Look at my Github and decide if I can hack if you want, that's why it's there, but I don't work on effing spec.

Our culture is rejecting the one effective test we have.

Do we care about equality, or not? I tried not mentionining this aspect, hoping people would realize on their own. But a remote work hire test is also mostly anonymous. It doesn't matter whether you're black, white, male, or female. All that matters is whether you can do the work.

On the flipside, what you're saying is that you genuinely want to spend a vacation day meeting a new company instead of with your family or working on your own projects.

And it's like, if you think you're a good dev, why wouldn't you leap at the opportunity to show it off? I get that it's a little annoying to spend a few hours on it, but the standard interview is literally random noise. Why subject your future to a random process?

I don't know. I respect your view. I'm going to bow out now. Have a good week.

I get what you are saying. And equality is tremendously, spectacularly important; I have more than once been That Guy at companies, making management uncomfortable when asking why the whole place is White Dude Central. But I very strongly feel that your value prop isn't an effective one from the perspective of the employee. You are not telling me why I should give a damn about your work-hire test, as somebody you need to hire to make your company work. What are you offering, aside from Yet Another Startup with Yet Another Startup Problems and Yet Another Startup Under-Market Salary? Why should I be your monkey?

And, FWIW, when salaried, I not once have taken a vacation day to interview. I've said "hey, I'll be in late," and because this industry is so deranged as to think that 50+-hour weeks are normal, no manager has had the temerity to get mad at me for taking a morning off because invariably it will be cashed in when I have to pull a sixteen-hour day to deal with a problem. (The days around Heartbleed earned me some goodwill at that gig, if you follow!)

It also doesn't matter if you are actually the one taking the test.

If you're a top ten percent programmer who charges $137/hr then this whole discussion probably isn't oriented at you. By definition, most programmers are not in the top ten percent. They make money by sitting down and writing code, not by projecting confidence and machismo. So why shouldn't they prove their abilities and evaluate the company and it's general environment by sitting down and actually coding?

I'm very old and out of the loop, I have no idea what modern work-sample tests look like. I remember when I did one many years ago it was to the effect of "Write a script that checks to see if a URL returns a valid response code, and trigger an event if it does not". My prospective employer liked my coding style and I got hired. What was so wrong about that?

As I recall, they intended to modify the script and use it in production whether they hired me or not. I also don't care about that. It took less than an hour, and it seemed like a valid form of evaluation. Maybe it's a pride thing, but I had no problem with it and I think a lot of other people would have been ok with that practice as well.

Because you shouldn't work for free. That holds true for writers, that holds true for artists, that holds true for web designers, and that holds true for programmers. And unlike those other professions, the creative has the power in this relationship. What these companies hate admitting but all know is true is that even a mediocre programmer, a sit-down-and-write-code programmer, needs these companies more than the programmer needs them.

If a company wanted to pay you for that work-sample test, that's different. But they don't, of course, because that costs them money instead of costing you. There's no need to let profit-seeking entities use you for nothing. You just don't need to be their monkey.

I don't get why you perceive the realistic-work-sample test as "working for free", but the unrealistic-whiteboard test as "isn't work." They are both taking a similar amount of time and effort, serving the same purpose, and benefiting the same people.* The only difference is that one does a better job.

*The primary beneficiary being you, if you're a top-10% programmer: It gives you a better chance to accurately show off your ability, which is probably worth thousands of dollars in salary.

Work-sample tests, in my experience, are universally "do this crap and then maybe we'll talk to you." It's nothing you'll get serious feedback on, it's nothing you can open-source, it's garbage code and wasted time that benefits the company unilaterally.

But the alternative--it's not the whiteboard, it's the meeting people. Right now, I consult, but I do take interviews for "permanent positions". But we all know they aren't actually permanent positions. The last thirty years of corporatism have made this loud and clear: we are all expendable tools. and I go into them with that in mind: a W-2 doesn't mean I'm anything other than working for a client under a tax-advantageous status. Interviews are business development; I am effectively offering my exclusive services on retainer to a single company. So meeting people to discuss the gig (whether or not the whiteboard comes into play, the only interview I've been on in the last two years where I was both interested in the company and wrote code on the board was Google) benefits me, too: maybe it's not a fit, but maybe I meet somebody I'll remember as "hey, I'd like to work with them again." Or "hey, I don't want to work here, but I know somebody who will, so I can do them a solid and they can help me in the future."

So, yeah, it is work. It's just business development, which is a different, and valuable, kind of work. And it applies to everybody, not just consultants; it's the game we're all playing even if one doesn't realize they're playing it.

OK, I agree it's also really important to spend a few hours meeting people, and I would never accept a job where I couldn't do that first. But I am personally thrilled to be able to spend two hours demonstrating that I am a really good programmer in all of the ways that a different kind of test doesn't pick up. That seems to benefit me a lot, and it also speaks well of the future coworkers I will have if I work at the company in question. Google's process (for instance) makes me nervous about this.

(Disclaimer: I strongly prefer to quit my current job before looking for new jobs. If that weren't the case, I'm sure I would be more sensitive to anything that takes time, but I think I would still feel this way.)

You're OK with spending two hours doing something before you get anything out of it? 'Cause I'll readily admit that there are probably companies out there that do such a test afterwards, but I've never seen one personally. Instead it's the gate to protect their precious engineers' time. Yours, though? Yours doesn't matter, and that rustles my jimmies real fierce.

Look at the opportunity costs: what if you could spend the time they want you to give them, for free, building something not only personally creditable but maybe even generally useful on Github? If you're good enough to show off, as you suggest, then your time is valuable enough that it should be respected. (A work-sample test that's a useful, valuable problem and can be open-sourced? I'd be down for that. But that would be haaaaard.)

I just spent the past month looking around for interesting new work after quitting my last position. Two of the ten or so smaller companies I've interviewed with so far had work-sample-ish tests as part of the hiring process: Keybase and AltspaceVR. Both of them had already made my short list of potentially ideal companies to work at, and I had ample time to ask them questions and chat before doing the work sample project. (At Altspace I went and had lunch with the co-founder and the director of engineering, and also tried out their VR software.) I felt no compunction at all spending 2-3 hours doing a interview project after that. Afterward, it was clear that both companies read my work and judged me based on it.

If companies were cold emailing me on LinkedIn asking me to do their two hour project, I would not do it, but neither would I bother going to interview with them. Given that I'm already cherry-picking which companies I care about, I don't mind investing some time to make them care about me.

EDIT: I agree that it would be nicer to spend time doing something really useful. I hope that if I handed them a FOSS thingy that was representative of my ability, that these same companies would respect that in lieu of their work sample -- although standardization is valuable for evaluation purposes, so I can understand if they wouldn't. (I just did the projects they suggested because I thought they sounded fun.)

Yeah, I've never had that experience! I'm sure I could be flexible about that--after I've been sold on the gig in the first place. But I feel like that's maybe an inversion of what it's usually used for: to weed out the shitty applicants. Looking at my Github and my blog (both linked on any documentation a client or prospective employer would see) should be enough for that to not waste either of our time, but at many places The Process Shall Never Be Modified.

(Right now, I'm helping out a few days a week over at a fairly large Boston startup, and while they have a general work-sample test that they roll out, I never took it. I probably wouldn't have followed up past the initial phone call if a monkey dance was necessary to get in the door.)

Most professionals do work for free, in small amounts relative to the amount of money at stake. If you clearly only need 2 hours of lawyer time, you pay for both of those hours. But if you're talking about maybe hiring a lawyer to do something complicated for you, the lawyer will give you an hour or two of time for free, in which they will use their professional expertise (i.e. do work) to assess whether or not your case will benefit from their services. But maybe you think law is a situation where the client has the power in the relationship? No worries, accountants do that too, as do architects and building contractors, as do SaaS companies, as do.... basically, as do members of every industry where the customer isn't obviously a penny pincher (i.e. retail sales does not do this).

You made this exact case yourself in a related comment, in which you called meeting people business development. I agree there's a non-zero distinction here between work-hire-test and consulting-about-consulting, but claiming that the issue is that "you shouldn't work for free" is misleading.

Also, you suggest that if they pay, that costs them instead of costing you. It already costs them more to interview than it costs you. But sure, if making sure they don't get any benefit out of their relationship with you unless you get benefit too, tell them you'll only do a work-hire if they make a donation to the EFF in the value of your hourly rate.

> But if you're talking about maybe hiring a lawyer to do something complicated for you, the lawyer will give you an hour or two of time for free, in which they will use their professional expertise (i.e. do work) to assess whether or not your case will benefit from their services.

Of course. That's the sit-down-and-chat "interview". But lawyers don't draft up a contract for you to demonstrate that they can write up a contract to your satisfaction. If you want a lawyer's expertise applied concretely to something of your direction, you pay. And while lawyers do have bar exams, they don't have a very detailed demonstration of their work up on Lawhub for your perusal!

> I agree there's a non-zero distinction here between work-hire-test and consulting-about-consulting, but claiming that the issue is that "you shouldn't work for free" is misleading.

I think it's misleading only if you consider the beneficiary of that work to be the same in both cases. I don't. The only beneficiary of every work-sample test I've ever been given was the company--it goes into some black hole and whether it was even good or not, to say nothing of any actionable feedback, has literally-literally never been forthcoming. On the other hand, I have very rarely not benefited in some way from sitting down and chatting with a technical leader at a company, whether it was directly about their problems (or their solutions!) or about tech in general. Both in a consulting-about-consulting capacity and an interview one.

> Also, you suggest that if they pay, that costs them instead of costing you. It already costs them more to interview than it costs you.

In an absolute sense, it definitely costs them more. In a relative one, it emphatically does not: they've got plenty of bodies that can parallelize. I can be interviewing with them, or I can be working, or I can be mopping my bathroom floor. I can't be doing all of those things at once.

Who said anything about working for free? The only work trial I've ever done was paid in full at the same rate as I would be making if I had already been hired.

It was fun, and more profitable than a free boring discussion-based interview.

eropple's entire spiel was in response to "A work-hire test...doesn't repel [the best programmers]", so saying that this discussion is irrelevant to a top 10% programmer doesn't really follow.

Actually, $137/hr is pretty cheap..

Yup. It keeps me busy, but it's not anything extreme.

>That's a $550 opportunity cost

The typical cost of hiring a programer is way higher than that, more like $30k I'd guess so they could always try paying you $550 to do something semi real? I'm not experienced but I've got a friend who hires remote workers and he says the only way to tell if they are good is to actually hire them to do a small job.

Triplebyte does notionally allow you to apply (to Triplebyte) via a trial project. But it's in addition to the interview(s), and the feeling they get from you during the interview trumps the project. They gave me the following feedback:

> This was a tough decision and one that we were on the fence about. We really appreciate you taking the time to work on the take home project. We're aware this requires a substantial time commitment and we are really grateful that you invested the time in completing it. We thought you wrote a great, very full featured [trial project]. It was especially impressive how much you dug into the academics behind [the project].

> However we made the decision because we felt that while going through the project together during the interview, we didn't see the fluency of programming when adding to it that we had hoped for. While we specifically designed the take home project track to help overcome the difficulties of coding under time pressure with someone watching, we do still need to see a certain level of programming during the interview. This didn't seem to be the case here, where making changes to the project seemed to be slower and more difficult than we'd have liked.

tptacek had some advice for hiring-through-a-work-test that I (from my armchair) agree with, which is that it's important to give everyone the same work test so your applicants are comparable. There's a little bit of tension (I don't think it's unresolvable) between that and "The test needs to be set up so that the candidate can demonstrate they can do exactly the same things all other employees do on a day-to-day basis, or it's not a work-hire test".

It always surprises to me to no end when I see people get any real feedback from the interview. Do these employers not care about getting sued, like everyone els.

That feedback may not be hopelessly vague, but it's also totally unactionable. They wouldn't let me reinterview, and interviewing was the only problem they identified.

This is absolutely true. Doing work-hire tells me if it's work I'm interested in doing as well as showing that I can do the work. Interviews are a crap shoot of bias avoidance as sillysauras3 says. Please actually work on fixing it.

"The best programmers" aren't all fixing bugs in iOS apps. The blog post perfectly outlined people who are motivated by other things. I don't want to work at a company where "realistic" work is me working alone on doing assigned tasks.

I identify with the "product engineer" concept they outlined, I want to talk UX and user testing in an interview not prove I can understand someone else's code base. Honestly many companies will get far more value of a programmer who has user sympathy and writes user-friendly applications than one who writes a fully tested, bug free and technically sexy app which users don't want to use.

"A work-hire test attracts the best programmers. It doesn't repel them."

Not really. The best programmers have other stuff to do besides your make-work silliness.

"Get them to show you they can do the work. Nothing else matters."

This is even more silliness. Suppose you have a diverse team, and the a candidate comes up, and passes the test, but is Donald Trump. Are you then going to say that only the work matters?

I don't know if I'm a "best programmer", but I definitely enjoy "add a little feature to our fake app" tests.

A friend of mine's company usually sends out a little canvas-based browser paint app with a bunch of features, and asks them to implement a few things, like flood fill. It's really enjoyable.

Favorite interview I've done was something similar to this; they gave me a small, pre-made project with tools they knew I was familiar with, asked me to implement some specific functionality, and then told me to "have fun with it for a few hours". So I did, and the project blew them away.

Unfortunately, they then flew me out to their office, whiteboard hazed me, and sent me home. Called me a few weeks later to tell me I didn't get the job, but that they sent me a hoodie in the mail (felt like a consolation prize).

It's a pretty comfy hoodie. They spent that VC money well.

> Imagine how mistaken that would sound if you substitute "startup founders" for "programmers."

Isn't that pretty common among incubators and VCs? It's not the only factor, but for first-time founders, being from Stanford is a large plus when it comes to getting in the door. If you already have a track record then of course they just go by that. But when it's your first company, I don't think the "startup ecosystem" is any less reliant on these kinds of signalling credentials to filter the large volume of people who want funding.

And the vast, vast majority of startups fail.

So.

For purposes of discussion, let's take it as a given that a work-hire tests are the best way to screen candidates.

The effort required to administer an on-site work-hire test is non-zero, therefore I cannot administer such a test to every applicant.

I therefore need a way to determine who to bring on-site, in order to administer such a test. I cannot phone screen every single applicant due to the cost involved.

That process could also be a work-hire test of some sort (e.g. a remote coding project), but regardless of what has been said on this thread, many good applicants will drop off at this step. I know this empirically, because I've experienced it, many, many times.

Additionally, the people who tend to drop off because of this extra effort tend to be senior applicants, who often have multiple offers from multiple companies due to the competiveness of the current hiring market. It also disproportionately drives away passive candidates, who are often the best candidates, because the best candidates are often not looking for work (since they are good, people who have worked with them previously want to work with them again, and they get poached).

So I need some method of sourcing and filtering candidates down that is non-intrusive to both our development team AND the applicant. This is the reality. This simply has to happen in a startup's hiring pipeline.

Currently, most companies do this by looking at resumes. That is obviously sub-optimal.

Any suggestions for alternatives?

EDIT: spelling.

I do. I was going to bail from this thread, but it sounds like you see the great potential that this idea can have, if it can work. The truth is that it works. Okay then, one last try:

Set up a system that can spin up a droplet for a remote candidate. Any time a candidate expresses interest in your company, spin up an instance and email them a link to it.

What does the link do? That depends on your company. Are you making an iOS app? Then the link takes them to where they can download source code for a fake, hypothetical iOS app. It says "X, Y, and Z bugs exist. Find them and fix them. Then add a feature: here is a clear description of what to add."

When the candidate is done doing this, they zip up their code and send it back to you.

If it sounds way more effective to look at that than to look at resumes, it is. If it sounds like it will repel candidates, well... Two things. First, if you're chasing a specific developer, then that isn't really the normal hiring process. You want them already. This pipeline is for everyone else. It makes no sense to subject them to a work hire test when you're actively seeking them out.

Here's the other point. The type of candidates you will find with this method will shock you. They will be so skilled that it won't matter whether they're called a senior or fresh out of college. You'll know immediately that you want them.

Everything I've described up to this point is a remote process. There is no on-site work hire test. By the time they come on site, you're mainly checking they can show up, and telling them about your company. You're no longer trying to filter them based on ability; they already demonstrated it.

Let's say your company's website is the primary focus, not an iOS app. Ok. The link will take the candidate to a hypothetical, fake website built with a similar framework. Again, it will have multiple bugs and a missing feature. Tell them what the bugs are, and tell them what the feature needs to do. Then have them send you their code when they're done.

I feel like at this point no one will even try to do this. You can think of so many reasons not to try: it takes too much work, it will scare too many people off, it will... Etc.

These reasons turn out to be largely fake or mistaken. Try it. Invest the resources to build this pipeline, tell HN when it's ready, and you win.

If this sounds prohibitive or unlikely, remember how counter-intuitive the most effective techniques in life are. Penicillin was discovered by accident. It sounds pretty unlikely that it would work. Same deal here.

I've explained this as clearly as I can. It's up to everyone else to either try it or to watch others win after they try it. Because the filter I've explained is the only way to let talent find you.

The type of people you'll discover will range from passive people who found the process amusing, to well-off senior developers who are demonstrating why you should pay them X equity or Y salary, to high school dropouts who turn out to be one of the most valuable people that join your team.

I'm not even going to touch the topic of what tech companies currently do. It doesn't matter. I've described what works, and if whoever reads this suppresses their instincts and builds this, they will discover it's practically the key to winning.

I think what you're filtering for here is 'initiative'.

I don't hire programmers, but have been a hiring manager, running fairly technical business teams now for 10+ years, and have found that initiative is often the deciding factor in any given employee's success.

There are other ways to select for it, and clues you start to pick up on over the years, but once you've figured out how to gauge relative levels of initiative across different people, I've found it makes hiring/resourcing questions quite a bit easier.

I really like what you've been advocating in this discussion.

We've evolved the same process in my own startup - in the last 2 months we've hired 5 remote programmers in Vietnam and India by giving them 2 rounds of remote work-sample tasks (a simple web app), then ending off with a text or voice interview. We ended up retaining 2 out of 5 of them (both Vietnamese, incidentally) who demonstrated a higher level of ability after they started.

The retention rate is low, so we've been tightening up the process. What we've found is that we probably needed to raise the bar a little by asking candidates to design and architect a feature (DB-backend-frontend) during the chat interview. That would have given us a better insight into their code structuring and teamwork/communication skills.

We've just brought on a 3rd hire (Vietnamese again) who passed our improved interview, and I have a good feeling that the process works very well to weed out candidates who don't have the needed level of ability and scrupulousness.

The best of our hires didn't attend college or have much relevant experience, but was clearly very capable and possessed great initiative. We're really glad to have managed to hire him. Another guy is finishing his 3rd year of college.

In summary, I think remote work-sample tests are fantastic, and I hope you're right that they're the key to winning :)

http://www.stackoverflowcareers.com/blog/2014/05/20/getting-...

Try segmenting potential hires into different categories and using different strategies for each group. You should also clearly communicate the process, so they don't get the wrong idea.

>Additionally, the people who tend to drop off because of this extra effort tend to be senior applicants

Is this before or after the phone screen? If they think the work-hire test is going to be in addition to a phone screen and an on-site whiteboard session, they'll drop. If they're senior devs, you should be able to tell from their resumes. Do a phone screen, and if they pass, explain that they can either do a work-hire test or a whiteboarding session. Explain that the work-hire test should be doable in 4 hours, so they don't envision a week long project. Also explain that it's a substitute for the whiteboard portion of the on-site. People who tend to hate whiteboard coding will choose the work-hire test and vice versa.

If you're trying to find diamonds in the rough among junior talent, sending out a first-pass work-hire test might work. You'd still get better results if you explained that it was short and a substitute for the onsite whiteboard session.

This also depends on how clear the company is about the hiring process. A lot of companies have started using the coding projects before the resume screen, and then also throwing in an all-day whiteboard session. So when someone's been burned before, they're going to do a 180 when a company replies, "Do this project."

Yes to this 100x. I actually did this last year with outstanding results. We were looking to bring on paid interns. We narrowed all the apps down to 5 and paid each $50 to complete real stuff that we use in our environment. 2 out of the 5 completed the work within an hour, 1 took 3 hours and the other 2 returned uncompleted work. We hired the 3 that completed the work and the one that took 3 hours ended up being the best out of them. This person also had very little prior experience but what I liked was that she stuck with the task and not only found the bug but improved some other code within the app.

All 3 ended up being very valuable and 2 of them stayed on well beyond their internship.

Oh, and these projects were done from home before we ever brought them in for an interview and we hired them before ever meeting them.

Work-hire test works for positions that requires failry narrow skill sets. You need iOS 2D game developer or nodejs to MongoDB plumber or single page JS app developer? Sure it would work beautifully. However most companies are not looking for such very narrow skill sets. Companies look for people who are generalists at heart and can be specialized in given area with little ramp up on demand . One week you might write build script and other week debugging Rails and yet another week fixing javascript in UX. When doing these tasks, it's not expected that you already know all these tools and platforms. You will have some ramp up time of 1 day to a week. But the key is that you should be able to move around stack, across projects and get shit done given reasonable ramp up times.

Also a lot of experienced programmers are generalists as well. You might have been C++ system guy in your past life, worked on JavaScript couple of years back and now working on Hadoop backend. You may not remember all the nitty gritty details of past platforms and tools but you can ramp up on demand.

So this 2 hour work-hire test that narrowly focuses on specific tools and platform won't work for vast majority of programmers who haven't branded themselves in to one narrow area. Also remember that a lot of programmers at places like Google, FB and Microsoft work in highly proprietory environment with their own internal build system, platforms and tools. They are likely not familiar with everything that's latest and in fashion. However they are very smart people, can change gear easily and, most importantly, ramp up on demand very quickly.

Yet another big issue is that work-hire test usually fail to measure hardcore computer science problem solving abilities. Sure, you are good at fixing some iOS bugs but can you figure out how to do driving directions on maps or may be scale email system to 100 million users? Now I do agree that lot of jobs don't require hard core CS but many do. And for a startup, it's usually impossible to tell if your future 6 month down the line will hing on having someone with hard core CS skills.

If you want generalists, you won't really be able to test them properly by asking questions at random: All you are hoping for is that they share the same background as you do. The moment it's you asking the question, and they can't really check sources, you have already lost.

As far as hardcore computer science abilities, a test still won't help, because what you are really testing is how far people are from college: Generalist cs is touched in college, but there are entire sections of computer science that someone with 10 years of experience might not have had to touch. I've had to do pretty specific CS stuff for work, but to get any of that done, I started with weeks of research, and then add my own spin to our specific situation top. You can't test the ability to do that by just checking if someone has memorized Raft, or black red trees, or any other random problem. I don't have a lot of CS algorithms memorized: that's why there's the internet, and why I have Knuth in my shelf.

You want to check for hard cs, ask the candidate to choose his favorite CS topic, and ask for am impromptu presentation on that. anything else is just going to fail at finding candidates that have different backgrounds than you, and that's PRECISELY the people you want to hire.

>The filters that companies and recruiters use mostly trend in a meaningful direction. There are a higher percentage of good programmers among MIT grads than SUNY Potsdam grads.

Uh are you sure about that? Google's experience seems to be that academic achievement is not a good predictor. And they know a thing or two about hiring.

http://qz.com/180247/why-google-doesnt-care-about-hiring-top...

Reply on news.ycombinator.com