Hacker News Reader

Comment by planetjones | Hacker News Reader

I don’t like the timeboxing aspect. It’s putting pressure on the candidate. I don’t know why the website says there is less pressure when it’s time limited. Do you have a study or other evidence to support this claim?

For code exercises I set when recruiting I send a pdf out and then expect a zip file or link to GitHub or bitbucket with the solution. I might give an advisory along the lines of this shouldn’t take more than half a day. But if the candidate screws up and starts again that’s ok. Mistakes and wrong turns are taken all the time in the real world. It works very well. Simple. No complications. And free.

View on HN

Hey, thanks for the feedback.

The idea is that the candidate knows they can do the best they can within the time limit, and then it's done. No agonizing over whether it's good enough or they should spend another few hours. No imagining how many hours and days other candidates for the same role might be putting in, wondering if they're going to stack up.

I've used this approach with around 100 candidates hiring for various roles in my own business and for clients. The response I've gotten from candidates has been overwhelmingly positive.

The problem I have with the traditional "don't spend more than about X time" approach is that it's not a level playing field. The responses to a challenge exist in relative terms, not absolute. It doesn't matter whether your response was great or not. It matters whether your response was in the top 5 of all the other candidates responding. It follows that you probably need to spend at least as much time, if not more, than all the other applicants for the role. How much time the other candidates _might_ be spending is basically unbounded, leading candidates to spend vastly too much time working on a challenge and then lying about it.

If the candidate screws up a timed challenge that's also totally okay. It'll be a great conversation to have in the follow-up interview. "I started down this road, but if I was doing it again I'd do X, Y and Z because of A, B and C."

Incidentally, I'd suggest that half a day is way too long. Especially if that half a day isn't actually enforced. When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max. I'm not looking for a product that's ready to ship to production. I'm after a sense of whether the candidate can actually interpret a problem then string some code together, and something meaningful we can discuss in the next interview.

>Incidentally, I'd suggest that half a day is way too long. Especially if that half a day isn't actually enforced. When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max. I'm not looking for a product that's ready to ship to production. I'm after a sense of whether the candidate can actually interpret a problem then string some code together, and something meaningful we can discuss in the next interview.

While I agree with what you said about leveling the playing field, I find 40 minutes sound awfully short. In university I noticed I was doing much better on 3h exams than on 1.5 hour exams, often only taking 1.5h for the 3h exam, even though it had more material. I couldn't relax enough in the short exam to do my best, even though the time was long enough for a twice as long exam.

> In university I noticed I was doing much better on 3h exams than on 1.5 hour exams

Same here. I was the person who would do poorly on the 15 min physics quizzes we had in college just to get the highest grade in the class on the final, only because I had more time (half the answers on my quizzes would be blank just because I didn’t get to them). In high school, I would barely squeak by on the multiple choice AMC to then do fantastic on the 3 hour AIME.

But I clearly knew the material, and I knew it very well, so are you going to penalize me for being slightly slower on a test? As an employee, I complete my real world tasks just as fast as everyone else, so why should a time constraint on a high pressure exam relate to the real world productivity and profitability of an employee? I would argue that it doesn’t—at all.

But then again, a candidate that puts in way more effort is also a strong signal for future performance.

I suppose it depends on the employer, does he prefer candidates that put in more effort but might be less talented, or very talented but might not have much resolve (unproven at least).

If I'm given a time limit on an assignment, I'm going to take the time allotted and it will reflect in my (admittedly easily faked) git history.

If the employer is being honest, then we're all good. But if they're going to go with the candidate that puts in 5x the time allotted, then I clearly don't want to work for them either.

It's an obvious red flag IMO. If they ask for n hours in the interview but expect more than that, then they're likely to do the same thing when you're on the job. Don't be surprised if a few months down the line you're working 12 hour days 6 days a week or you're an underperformer and the first to get cut.

How often in their day job will someone be under artificial time pressure to solve a task and get it to a point where it's "good enough"?

You're actively removing the possibility of finding out who would think "meh thats good enough I guess" is the effort needed for production code and people who would go the extra step of spending time analysing and fully solving the problem.

2 people with equally average solutions...would one have done more given the chance? Both? Neither? How can we find out? You've added uncertainty to the process rather than solved any problem.

Often enough someone will be under real time pressure to solve a task, right?

That's more indicative of poor management and planning on the company's side than reflecting on the performance of the candidate if the time pressure is a regular occurrence.

Few people perform well under pressure. Do you want to test what is their mediocre performance under time limits like on a school test or what they are really capable of?

With this approach you will hire people able to churn out crappy but "good enough" code in the limited time. Not people who actually can do quality work.

No real company has perfect management and a lot of them have poor planning.

> Few people perform well under pressure.

Yes, but majority of jobs have some level of time pressure. That being said, there is such a thing as reasonable deadline.

> With this approach you will hire people able to churn out crappy but "good enough" code in the limited time. Not people who actually can do quality work.

Inability to perform under pressure does not imply high skill without pressure.

> Inability to perform under pressure does not imply high skill without pressure.

Ability to churn out crap in a time limit doesn't imply competence neither.

Well, it implies competence at churning out crap within a time limit, which for a lot of managers out there is precisely what they're looking for.

Well isn't that the point of these: to see that they don't churn out crap, but do a decent job.

There's never enough time or resources to do everything you want to do. Believe me; I've gone from one extreme to the other in company and team size. For any but the most mechanical of development tasks (e.g. CRUD apps), it's very common for a developer to make an estimate, realize that it was too optimistic, and then have to reduce the scope. So it's valuable for a developer to spend some time up front thinking about the design, break it down into tasks, prioritize them, and then implement the highest-priority part first. Then, even if they have to cut tasks, they can deliver something of value without having to do shoddy work.

I do think an interviewer should be completely up-front about what they want out of a candidate if they give a coding challenge that's too ambitious to implement perfectly within the time limit.

Many times, markets and business dictate time pressure. So yes, this is very real in startups at least. Like for instance, an inevitable market development might want you to ship code by the next week, however improbable or impossible. I know I'll get a lot of flak for saying this, but that's ok. I've experienced it first hand. In an early stage startup, when faced with the choice of losing revenue/delivering on time, you'll choose the latter. That means your shipping slot is fixed and so are your features. You'll just have to slog it out or work real smart. Even if you cut features, you'll still realize there's a lot to be accomplished in a short span of time. This is mostly a one off thing. If such events repeat endlessly, that is classic bad management. So you'll need to deliver on time, sometimes :).

So, I guess that companies which administer such time based tests foresee at least one such situation for that role. I'm just speculating.

> How often in their day job will someone be under artificial time pressure to solve a task and get it to a point where it's "good enough"?

There are quite a lot of software jobs like this. Maybe the time pressure is more 2 weeks instead of 4 hours, but deadlines are a real thing.

Two weeks is decidedly not 4 hours, no matter how you try to relate them. I could write the Falcon Heavy takeoff sequence in two weeks. (joking, of course, but the compression factor of 4 hours over two weeks is overwhelmingly and obviously higher)

> How often in their day job will someone be under artificial time pressure to solve a task and get it to a point where it's "good enough"?

In my job, pretty much every week. Many of my tasks are "implement feature X to a passable amount in 2 days", and then it goes to someone to use and get feedback on immediately. My team are mostly the same.

This sounds to me like the nature of every job. Secondly, the interview process has never, to my knowledge, reflected on the job performance, and I'd suggest it shouldn't.

> When I'm hiring I usually set challenges of about 40 minutes, an hour and a half max.

No. Forty minutes is far too little time. Many people take 10-15 minutes to even get settled into coding. I can't imagine a meaningful, applied, realistic exercise that would take 40 minutes, unless you're wanting basic data structures like a linked list, or a basic sorting algorithm, or something.

You're missing a lot of good people that way, but I guess that's just collateral damage in the quest for the perfect hire, right?

Except your path is the one that has lots of long, burdensome take home tests. Some people have time commitments and/or other jobs to apply to but I guess those people are just collateral damage in your quest for the perfect hire, right?

Is it possible that there could be a happy medium?

An 1-1.5 hour test is a happy medium.

I would agree that 1.5 hour test is a happy medium. I'd prefer 2 hours.

I am guessing fizzbuzz or something similar. I was applied for a job and got asked to do an n queens programming task. It was for a commodity trading firm. I told them no thanks. When I set code exercises I craft them myself and make them appropriate to the job on offer.

Robust answer and well reasoned argument. I agree, and would go further to say that less pressure and more satisfaction has been had with time restricted homework in my experience as an interviewee.

You have your reasons, but I'll walk from a time limited take-home, just as I'll walk from a screen captured one.

Both methods are ways to tilt the control and convenience back in the favour of the company - defeating the whole purpose of doing the work at home.

With all due respect, but attempts to optimize the interview process invariably end up doing this, due to the priorities of the paying parties (ie - the companies) - and always at the cost of the interviewers. No thanks.

I worry you might be projecting your experience onto the experiences of others. It sounds like you’ve been stressed out about having an unlimited time slot in the past.

But lots of people (myself included) get debilitatingly stressed out when an arbitrary clock is put in front of your face. Honestly, my recommendation to any company that gives timed challenges is to... not.

What if you gave the option to not time challenges? And maybe didn’t make it seem like having untimed challenges are inherently more stressful all the time.

I mean, it's strictly better than the typical code interview, which is timed too. And the reality today is that the majority of people have to be able to deal with a ~45min code interview to find employment in software.

I recently interviewed for my next role. Not a single company was doing the in-person timed code interviews. Where there were code tests, they were all take home and had no time limit beyond "when do you think you can get this back to us by?". If you need a time limit, ask the candidate for their estimate. It has the added benefit of telling you whether the candidate can assess their own schedule in relation to your code test, while not screwing the candidate over.

Hi David,

I like the idea a lot, but I think it is up to the team and hiring manager and not the tool the decide whether to use a time box or not. I haven't tried the tool yet so I'm only speculating here, but as a potential user from the team / hiring manager side I would expect I could disable the time boxing completely if I wanted.

Cheers!

If you’re setting deadlines of forty minutes then I suggest you’re losing good candidates, who like to assess a problem and think through a solution first. It’s just creating pressure. Unless your code exercises are takes on these fizzbuzz exercises? I.e. trivial. From my experience (and I have a lot) you can tell a good developer by their deliverable. Even better if you see the code developing through multiple commits. Some developers might take a long time, but if it’s bad it’s just bad. I would worry that if you only set 40 - 90 minutes exercises you aren’t giving enough time for the candidate to differentiate themselves at all.

All this does is compound the stress. Instead of "agonizing" over whether it's good enough, now a candidate gets to agonize over 1) is it good enough, 2) why didn't I just push the updated code 1 minute earlier before the cut off time! and 3) sh^t I should have put more comments in to explain where I was going with that!

>The problem I have with the traditional "don't spend more than about X time" approach is that it's not a level playing field.

No, it's not. And that's actually fine. I don't care if a candidate took five minutes, or five hours, or five days! In fact, the response time can tell you something about a candidate. Don't take that away from me as a hiring manager. I've actually never asked or been asked how much time I spent on a challenge. It's not an issue to solve!

>If the candidate screws up a timed challenge that's also totally okay.

What follow up interview? They fu^ked up the challenge. You really think a hiring drone is going to proceed that candidate when they have 10 others who did ok? That goes completely against what this sort of canned test is all about; getting past a hurdle. If hiring managers were going to interview all candidates who took the test and have a nice chat about their code, the test is completely redundant. They'll just ask for a code sample to talk about. Saves everyone the time and the money!

All in all, 40 minutes is way too short and artificial cut offs are a nasty way to try and "level the playing field". In fact, you're not leveling it at all! Just making sure candidates who may be taking a bit longer to get into the swing of things that day are discounted. Heck, if the candidate has simple network issues they're screwed!

I think your approach is reasonable. But I'm curious, why provide the challenge in PDF rather than HTML? Unless your PDF generator can create a tagged PDF, that's going to be less accessible for blind people to blind people than HTML, .docx or .odt, or even plain text. I don't mean to guilt-trip here; I figure you weren't aware of that problem. Is there a reason why PDF is better? For example, do your challenges tend to include graphics or complex math notation?

In the past when doing timeboxed, we gave the candidate a weekend. The task was too much work for a weekend, but that's part of the job. Choosing to prioritize and what to prioritize is also an important skill. In our case: if you were going for the perfect code abstraction but didn't deliver any business value in the form of a working product, that says something about the candidate and their performance in a team.

I hope you communicated that clearly beforehand?

This! I've done more than one code challenge in my time that was secretly testing for how far you get on a task that is underbudgeted on time. Your best applicants are going to spend too much of their own (unpaid) time finishing it and be annoyed at you before you tell them how clever you are for testing their ability to prioritize...or they're just going to not bother.

I would assume this is how the company will treat me after being hire.

Far too many companies forget that they are also being tested and interviewed. The sort of stupid "let's secretly test whether the candidate can figure out we're screwing with them" kind of thing lets good candidates know one thing: the company is toxic and won't treat them like a human. This is the exact sort of behavior to run away from.

Yeah because most peoples biggest priority going into in a technical interview process is focusing on good code. If they want a candidate to "MVP" their coding challenge for business value reasons, they should make that very clear. It's not like it takes MORE skill to write hacky code.

Definitely

Do you tend to give a two day deadline in the workplace and then expect the deadline to not be met?

We were open and honest about it being too much. The pattern that emerged is that it allowed us to see the candidates' strengths. Some didn't tackle the backend part: that was not their forte (and we hired them and sent them off on a course). Also, in any IT job I've worked there's generally more business requirements/wishes/etc than can be finished. That's a fact of life. It helps to have team players that have are independent and give some push back where needed, as well as realistic deadlines. There were some that handled the deadline badly, that didn't help their case. (but apparently, many believe giving someone a task to do is not done)

I guess what I'm saying is the expected reaction to this in the work place would be to trim the requirements and set expectations before you ever start work. If that's part of the test, cool, but it seems like that wouldn't be something a candidate would assume heh.

Do you want crappy code that get's harder and harder to extend and maintain? Because that's how you get that.

Correct, but time to market has it's value too. Also: mobile dev so Apple forces a rewrite every couple years anyway (on both design and tech fronts)

Reply on news.ycombinator.com