Hacker News Reader

This is my approach for evaluating new candidates as a first step. For front-end, for instance, I start with a codepen that has the basic skeleton that includes the libraries we use internally, has a few html elements styled and placed, and the required javascript boilerplate.

Then I provide 10 tasks that are written much in the way the company would put tasks in asana or git or jira. Each task builds on the previous and gets more complex. In the boilerplate they start with, there is one obvious error and one subtle one. Each step, I encourage them to write notes on how they solved it, and any questions or concerns they might have. I make myself available via chat or email if they have any clarifying questions.

They then can do it in their own time, at home. They can google, hit up documentation, stackoverflow, etc., because that's exactly how developers work. Nobody's over their shoulder, there's no arbitrary limitations, just build us some good UI code.

It's true that I've been amazed at how many candidates were stopped in their tracks at the simplest things that are well within the domain of their resume. But when I get a candidate that gets through them, hits the requirements and has clean code as a result, I've hardly ever seen them fail at a later step where we might evaluate more complex problems like working around GC slowdowns, obscure browser compatibility, first meaningful paint performance, etc.

And most of all, I feel confident that I haven't tried to "trick" anyone by having them do long-solved algorithms offline on a white board.

A huge advantage of this approach is how much more confident you can be that you've qualified candidates technically. You can --- and should --- exploit that confidence commercially. Change the posture of your outreach to reassure people that you're interested in applicants regardless of their background or resume. Change where you run job ads and how you word them. If you have a strong qualification process, your job ads no longer need to do any heavy lifting to get candidates to screen themselves; after all: your screening process is going to reliably distill your candidate stream down to a technically qualified subset.

At Matasano, we went from hiring almost exclusively people with a background in our field to virtually never hiring people with a background in our field in a matter of about a year --- and the quality of our new hires went way up.

This is some pretty outstanding insight. I work with middle school kids to get them excited about computers and programming and it's amazing how many of the ones that "get it" are not the ones you would pick. Frankly, I think they are surprised at how much they like it and you can almost see the light bulb go on. If we could reach those type of people and get them to interview it would almost certainly change the industry for the better.

> Rather than trying to read subjective, noisy signals from in-person interviews, give candidates a small battery of work-sample tests. These can be bug-fixes, small features, diagnostic challenges, or even specification writing assignments.

I agree that work sample interviews are a huge step in the right direction.

Recently though, I've been thinking about taking things even further: instead of giving sample projects, why not give candidates actual work that you need done, and pay them for it, like you would a contractor? Sample projects are inevitably going to be somewhat contrived, and they'll never be evaluated in quite the same way actual code submitted in a PR will be.

This approach lets the candidate and the company "date" for a while, letting each see how the other one actually works, before they decide to tie the knot of full-time employment. I think it's the most natural way for professionals to get to know each other, build trust, and develop well-informed opinions.

Of course there are plenty of practical logistical challenges, but it does seem like a freelance-to-full-time process could lead to better outcomes all around.

There are all sorts of problems with this approach.

Contract software development --- relatively short projects with no up-front commitment to long-term work --- is, when sold and delivered competently, much more expensive than salaried software development. That's why people hire full-time.

The very best software developers know this. They're not as a rule especially good negotiators, but they're familiar with the market. For every firm offering them temp-to-perm work, there are 4 others competing over who can offer the most attractive immediate full-time offer.

That makes the "standard" recruiting pipeline for temp-to-perm companies an adverse selection problem. And, I believe, it's also often an exploitative one. People taking these temp-to-perm contracts aren't getting paid 2-3x their salaried rate to work on temp terms. But that's the lower side of what contractors demand to accept the kind of risk we're talking about with these projects.

Really, companies that do temp-to-perm are telling the market, "we're so bad at qualifying candidates that we're going to fob the entire problem off on our candidates, and hope there are enough competent people with such poor negotiating skill in the labor market that we can get away with that indefinitely". For now, they're probably right --- although taking this pathway is, I think, doing damage to the long-term integrity of their team and processes. But I wouldn't bet on it.

You make a lot of good points. Temp-to-perm could easily turn into an adverse selection situation, and there is potential for it to get exploitive.

That said, I do think there is appeal for candidates in contracting for companies as a way to get to know the teams a bit better before deciding where to take a full-time offer. At least it's something I've been glad I've done at times, and wished I could have done at other times.

Plus, most developers are not the very best developers, and they won't have four competing offers after only a couple of weeks of job searching. For an awful lot of developers they'll need to spend hours prepping for interviews, working on take-home projects, going to onsites, and talking on the phone with hiring managers and recruiters. They spend a lot of time and effort for which they're not getting paid anything, and which is largely orthogonal to their actual work. When they do get a few offers they have to make a decision with limited knowledge of what working at each company will actually be like.

I dunno. I still think it would be awesome if job-seekers and hiring teams could get to know each other in some more natural and meaningful way than the feats of strength performed in today's interviews. Maybe temp-to-perm is too problematic though.

For my first job-search out of school, I actually started off by accepting a temporary, fully-remote contract position for one of the first startups I was interviewing with at the time.

After a short 1 hour pairing session with their front-end lead, I was offered a choice between going through the full 1 day interview gauntlet onsite at SF to arrive at a full-time offer on the spot, or a 2 month remote contract to give us both some more time to decide if we want to proceed with working together full-time (the actual length of the contract was negotiable, I just went with 2 months because I felt that's about how long it'd take me to finish interviewing with all the companies I was interested in). They made it clear that I could continue to interview at other companies during that time frame, as long as it wasn't so disruptive that it impacted my quality of work, and that I should feel free to turn down the full-time offer at the end of the contract if I find a better offer elsewhere. I felt the latter option suited me better at the time, because it was still fairly early in my interview process, so I didn't have any offers coming up on the immediate horizon, and I was still definitely interested in exploring my options some more before deciding on my first full-time job. So I accepted it.

I had a blast working there for those 2 months, learned a lot, and among other things, completed a decidedly non-trivial addition to their app. At the end of the contract, they offered me a very generous full-time position, but I ended up taking a competing offer at my current employer instead. There was no hard feelings, and we parted on good terms.

I guess what I'm trying to say is, yes, temp-to-perm definitely isn't right for every candidate (or most candidates, for that matter), but it will give both you and the candidate much more confidence in the decision on whether or not to go full time than any interview process ever could, and there's nothing wrong with offering it as an alternative to the standard interview gauntlet and letting the candidate decide what's best for them.

You definitely have to come to terms with the fact that it's a 2-way street though, meaning you have to accept the risk that the candidate might find at the end of the contract period that it's no longer in their best interest to proceed with a full-time position at your company, and that they are free to act on that realization. Otherwise, if you try to place limits on that aspect of the arrangement, the only candidates who would likely accept it would be the truly desperate ones whom you were probably never interested in hiring in the first place. I for one certainly wouldn't have accepted the arrangement if I wasn't free to continue interviewing during the contract and free to turn down the full time offer at the end of it.

>>Rather than trying to read subjective, noisy signals from in-person interviews, give candidates a small battery of work-sample tests.

I tell people the same thing, but the typical response is "that just discriminates against people who have families and other obligations and can't set aside 10-20 hours on evenings and weekends".

A home test must be doable in 1-2 hours. Not 2 days.

Mostly right. The better way to think of it is that you have a bucket of hours from which to draw phone screens, in-person interviews, and challenges. If you do no challenges today, you shouldn't expand the bucket of hours to accommodate them, but rather take offsets from other activities.

Phone screens are particularly useless and a good place to scavenge hours from.

Other than the pain of being rejected (or rather the lack of validation), where does this reliability problem manifest?

Elite tech companies have no problem with the current implementation of technical interviews.

Engineers whos skills range from Great to Mediocre don't have a problem finding a job.

Software quality has generally improved or stayed the same.

This is demonstrably false. Lots of great engineers are terrible at interviewing. If you ask around, you can get even some very high-profile devs to tell you stories about why they're not working at Google or Facebook or Amazon because of how badly their interviews went. Equally importantly: lots of terrible engineers are great at interviewing, and most people in the industry have had the experience of working alongside someone who is failing up, 2-year-stints at a time.

What's true is that Google and Facebook have such enormous budgets and such huge positive profiles in the talent market that they don't need to work hard to put together functioning teams. They're paying more than they need to and having to deal with more bad hires than they should, but it's for them a marginal problem.

> lots of terrible engineers are great at interviewing and most people in the industry have had the experience of working alongside someone who is failing up, 2-year-stints at a time.

You mean lots of companies are terrible at firing terrible engineers.

The argument is basically the current implementation leads to false negatives. What impacts does that have?

Presumably those very high-profile devs aren't lacking employment opportunities, even if it isn't at the elite tech companies.

Companies, from Elites down to pleb-tier, while perhaps pressed for resources, are still able to execute.

So why is the problem of false negatives a problem we should focus on?

Almost every company is terrible at firing ineffective engineers. Even some of the companies that are famous for doing so ("'meets expectations' earns a generous severance package!") don't necessarily manage to accomplish that in less than a year. This stuff sounds very simple on message boards, but is much harder to do in real life.

The first N months of a developer's tenure at a job is difficult to measure because of ramp-up and calibration (even good developers might hop to different teams after getting hired based on work style fit issues). Whole projects can flounder and need rescuing so that attention to ineffectiveness gets deflected to group leads. Most good development shops will try to make things work for a new hire that isn't performing at the 6 month mark, and that process takes a few months.

Once the decision is made to fire, most big companies will PIP rather than fire. Sure, once you've hit the 7-8 month mark of poor performance, your severance from the firm may be a fait accompli. But the ultimate process to make that happen can easily be made to take more than 12 months.

Once you have a year-long stint at a company on your resume, you can bank that "experience" on your resume.

And that assumes a relatively well-run companies. Most companies aren't well run in this regard, and you can be a terrible developer but last 2+ years in most jobs. If you can reliably put more 2+ year jobs on your resume than 1-year jobs, you will end up with what will appear to most companies to be an extremely attractive resume.

Remember, many management-track developers only write code for ~8-10 years. Once you fail up to Dir/Eng, you're basically tenured.

Since almost nobody qualifies candidates effectively, failing up is a super powerful strategy for this field.

Why are the terrible engineers terrible, leading to them being fired (after 1-2 years?)

I've not hired anyone who turned out to be bad at coding, but I have hired people who had other performance related issues, which were difficult to suss out during an interview.

From my own personal performance, my approach to interviewing has led to far more true positives, than false positives, so I am biased towards continuing.

> other performance related issues

What are some examples of these?

Typically it's patterns of repeated performance issues, such as providing estimates then routinely not completing work on time.

Or, failing to complete tasks they've taken on, requiring other teammembers to step up in order to meet deadlines.

There are more extreme cases, like insubordination, theft etc, but they are pretty rare.

> Engineers whos skills range from Great to Mediocre don't have a problem finding a job.

How are you defining "skills" in this case? "Skills" as in the skills needed to perform well in the traditional technical interview gauntlet?

If that's the case then you have a tautology on your hands.

And if not, then maybe it'd serve us better as an industry to try to find a better proxy than the traditional technical interview for whatever definition of "skills" we really want to optimize for.

Skills: writes code that's clean, efficient, simple, etc

My question is this: Why does the industry need to change the interview process?

If companies are able to maintain a quality bar of it's employees, and engineers are able to find meaningful employment, doesn't that mean the system, even if it has a flaw (false negatives) isn't really a problem?

Perhaps the problem is that many of us who write code, consider ourselves good at it, and getting rejected is a hit to our ego. We want to change how interviews are done solely for the benefit of our egos, when instead, we should understand that if you interview, you might get rejected, and that is okay.

Nothing is fundamentally wrong with the process, nothing is fundamentally wrong with you, and in 6-12 months, you can try again.

> Skills: writes code that's clean, efficient, simple, etc

> My question is this: Why does the industry need to change the interview process?

Because at the end of the day the process makes very little effort to optimize for any of those metrics of code quality you just laid out, and purposefully offers candidates programming problems and working conditions that deviate significantly from those they're likely to encounter on the job for no apparent reason, thereby hampering its reliability as a useful proxy for developer skill even further.

> If companies are able to maintain a quality bar of it's employees, and engineers are able to find meaningful employment, doesn't that mean the system, even if it has a flaw (false negatives) isn't really a problem?

No, it really doesn't. Just because a majority of software engineers manage to stay employed, doesn't mean we should ignore flaws in the system and insist that it's not a problem.

And if you would, take a moment to consider widening the definition of "companies" you have in your mind when you made that assertion. Yes, the process works well enough if you're Google or Facebook and have a constant stream of candidates and can accept just about any arbitrary false negative rate because you have the law of large numbers on your side when it comes to hiring. But startups and many non-tech companies large or small (most of whom don't have a lot of developer mindshare) often won't have that same luxury, because every month wasted in the hiring process to a high false-negative rate will take you that much closer to the end of your runway if you're a startup, and to the eventuality of being out-competed by a more efficiently run, technologically innovative competitor if you're just another technologically stagnating company. Countless numbers of these startups and technologically stagnating companies die off every year, and likely in no small part due to their inability to attract, recognize, and hire skilled developers in a timely fashion.

> Perhaps the problem is that many of us who write code, consider ourselves good at it, and getting rejected is a hit to our ego. We want to change how interviews are done _solely_ for the benefit of our egos, when instead, we should understand that if you interview, you might get rejected, and that is okay.

I don't doubt this is _one_ aspect of why some people want to change the interview process, but I emphasized the _solely_ part because I take issue with that part of your assertion. Getting rejected hurts, sure, and that can certainly bias one's judgement towards wanting change for their own benefit. But that's certainly not the _only_ driver for why people want to change the traditional interview process. You can see countless well-reasoned, rational cases, that make no appeals to emotion, for the many issues with the traditional interview process, both in this thread and elsewhere, made repeatedly by people from all walks of the industry from both sides of the interview process. Some people definitely do want to change the interview process for purely selfish reasons, but that shouldn't invalidate any of the countless well-reasoned cases constantly being made against the traditional interview process.

> Nothing is fundamentally wrong with the process, nothing is fundamentally wrong with you, and in 6-12 months, you can try again.

I would offer the counterpoint that, if nothing is fundamentally wrong with you, and you can get accepted again simply by applying again in 6-12 months, without having made any significant improvements to your skill level, then that is a clear indication there's something fundamentally wrong with the process itself. It all comes down to the fact that the ideal technical interview process should optimize for vetting for developer skill, and do so in a reliable fashion, minimizing both false negatives and false positives.

The traditional interview process has been well known to to exhibit a high rate of false negatives. If your company is deep-pocketed enough and attractive enough to candidates that you can afford the false positive rate by virtue of your naturally high rate of hiring, then by all means, keep using the traditional interview process. This is a process that has been proven to work well enough for companies hiring under those circumstances, and it's likely that for you, the risks posed by the high rate of false-negatives won't outweigh the risks resulting from adopting one of the less well-adopted and well-studied interview processes.

For everyone else, however, I'd strongly advise very carefully weighing how much risk you're really taking on by accepting the high rate of false-negatives that come with the traditional interview process, and consider one of the many possible alternatives to that approach before jumping blindly into that calgo-cult.

Anecdote, but a telling one: in my previous round of job-hunting, I had a success rate of exactly 0% with whiteboard coding interviews and a success rate of 100% with other interview methods. That's a pretty stark correlation.

Is there a case study somewhere for such an interview? Like a description or a video or such?

Also how do you deal with the standardized interview being published? As in 'company X' uses a live coding exercise and they ask you to do X in a succession of changes. It would quickly come out for most larger companies.

You're smart people. I'm serious. For any of these types of concerns, you will come up with reasonable answers. My only advice is: don't overthink it. Solve problems when you know you're going to need to solve them, but err initially on the side of making things less onerous for candidates.