Hacker News Reader

Comment by mbil | Hacker News Reader

This is a great list. Thanks for sharing it.

I actually had the pleasure of interviewing at one of these companies before. They had a take-home project, which was to choose and implement and couple enhancements to a toy app. In the subsequent conversations, we discussed how I approached the problem, details of my design, technical tradeoffs, etc -- all the sorts of things you would expect a professional sw engineer to be able to do on the job. It was challenging but fair, and I was struck by how reasonable the process was. It left a wonderful impression on me.

Take-home projects are not a perfect solution to the interview problem. A major issue is that they require candidates to have free time outside of work to complete them. Personally, though, I'd much rather spend a few hours per application working on projects than countless hours prepping for stressful whiteboarding puzzles. As a developer who prefers to digest problems slowly, I look forward to the demise of on-the-spot whiteboarding interviews.

If you work at a company that subjects candidates to such high-stress algorithmic riddles, and you have the ability to make your process more sane, please consider following the lead of cos on this list.

View on HN

Take-home tests are fine in isolation, but if every company does that you'd pretty much be spending all your time doing take-home projects and nothing else.

Also many companies will insist on a take home project in addition to everything else, including a whiteboard interview. So you can pour time and effort into this project only to be eliminated further down the line (or just ghosted).

It's like they assume applicants are applying to one job at a time and have no other work or life commitments.

I dunno, it feels like the tech industry has forgotten how to interview candidates, as in "just chat with someone like a normal human being to see if they are a good fit for what you need", and like many other aspects of the profession have fallen for dumb processes, fads, cargo-culting, outsourcing to third parties, and dubious technical solutions in lieu of experience and social interaction. I learned this early in my career, and found I can pretty much size up someone's technical chops - assuming they're in my wheelhouse - and get a good idea of their professionalism - with a short, half hour phone/Zoom interview. Yes, I can see how this might lend itself to bias and prejudice, but it seems the solution there is to train interviewers to overcome biases and have more diversity (in all senses of the word) in your interviewing team.

Just look at their GitHub and if it looks good, dive in at the interview to make sure they actually wrote the code. Isolated take-home tests seem ridiculous to me if the candidate has public code available. You're filtering out anyone who likes to avoid duplicated effort.

I don't think you should require GitHub, but if a candidate came to me with projects on GitHub I'd skip all technical aspects of the interview, since they have already demonstrated their technical ability.

Requiring github as a CV is worse from the POV of not forcing people to spend a lot of time outside work programming.

Not everyone has the inclination or time to code outside of work. Many of these people are still very competent developers.

If a person has some interesting Github activity (that they can back up) I would certainly discuss it with them. If I'm hiring for a Django developer and the candidate has contributed to the Django core or a related library, that would be a strong signal - but it should not be a requirement. A strong candidate is typically a mixture of good work history, side projects, hobbies and interests, personality and other things that indicate they would be a good fit for your team. How much and in what way depends on the job and the person. Maybe the startup company needs a person who does lots of side projects, builds stuff on their own etc because you need self-starters that don't require lots of hand holding and are ready to take the initiative to learn. On the other hand your 1000-strong engineering subdivision at big corp needs more heads-down 9-5ers who just produce reliable output. Maybe the things you ask a junior dev out of college are different to what you would ask a mid-career developer. There's no one size fits all approach that people seem to be looking for.

Sure, that's why I said "if" - if they don't have a public presence then give them a test.

But frankly I don't really buy that excuse. Those people have time for multiple coding tests but they don't have time for a (small) public project? Ehhh. Not convinced. If you can get away with no portfolio to begin with then you're not going to be inclined to accept coding assignments.

I hate take-home projects more than I hate whiteboard interviews.

I think the best approach would be giving the candidate a broken app and have they debug it live or in a take-home debug-and-fix or debug-and-improve exercise. Could also have some refactor assignment.

Has any company tried something like this before?

I don't understand how take home interviews are being held up as a better solution. They have two major flaws:

They lower the cost for the company to zero. So they can widen their pool, and take a chance on more candidates, with no extra cost incurred for them. If you take this to the extreme – every company asking every candidate to complete a take home – then the result is far worse for candidates.

There is no feedback mechanism for candidates during the process. I can say "you shouldn't spend more than two hours on this" until I'm blue in the face, but candidates think they need to polish far beyond what is reasonable. On occasions where we had a system for capturing progress, I found that most candidates far exceeded the time recommended to them. Some were spending up to six hours. The frustrating thing was the extra time never made a difference to the outcome. I could tell from their initial hour whether they were going to pass.

As a hiring manager I've used them, but work hard to be considerate to the candidate. I'd prefer to design a better in person interview than take the easy route that is disrespectful of candidates' time.

My company does take-home interviews (or rather, we give candidates the option to do them; nearly all candidates choose take-home instead of a live coding session) and I don't think I agree with the "lowers the cost to zero" point. First of all, they are never the starting point; we only give a take-home problem to a candidate after we've had an initial discussion with them.

When we get a take-home submission, we essentially run it through our code review process, and as one of the engineers who does reviews, it usually takes me nearly as long to thoroughly examine a homework solution and write up my feedback for internal discussion as it would to do an in-person interview.

Being able to review a candidate's code on my own schedule rather than being yanked from the middle of some knotty problem to go do an interview at a fixed time is a big quality-of-life win for me. But the rate limit isn't much different than it would be with regular interviews.

All that said, it's totally possible our process is super atypical.

At Caltech, most exams were take home with a time limit of 2 hours. Some were "infinite time". The students hated the latter, because it means there was no end to the time you needed to spend on the exam, meanwhile there were other exams, etc., to take.

With a 2 hour exam, there's an end to it, and you know how much time it should take to do the problems. Much better.

It's even worse if you're being graded on a curve, as you would be in a job interview. You're competing against others who will spend infinite time on it.

Like you said, "there were other exams, etc., to take.". Presumably other competitors have other exams as well? You'll thus be competing against people with different time management skills.

> I can say "you shouldn't spend more than two hours on this" until I'm blue in the face, but candidates think they need to polish far beyond what is reasonable

> I'd prefer to design a better in person interview than take the easy route that is disrespectful of candidates' time.

Why don't you just let candidates decide for themselves how much time they want to spend on the task? You can set the expectations, saying that you expect that the task of a given complexity will to take between 1 and 2 hours; but candidates should be free to spend as little or as much time on a task as they like. Some candidates will value their time and spend no more than is advised; others will be sufficiently obsessed by the task to spend more time. Why should the latter group feel guilty about exceeding the recommended time limit?

You're trying to evaluate a candidate's ability to accomplish something with finite resources. If you allow someone to put additional time into it, you're biasing the results towards the more desperate and less in-demand individual.

- If the more desperate and less in-demand individual writes bad code, or makes poor judgements, you will be able to see it in their submission, won't you?

- Whereas if such an individual, because of their desperation or lack of demand, has enough time and motivation to elegantly solve the problem, then what's not to like?

I think a greater concern would be that with take-home assignments it's impossible to be sure you are actually evaluating the right individual. How can you be sure they didn't receive help from someone else?

The programming test is just one aspect of a candidate, in my estimate about a third of the total "score". Take-home exams bias towards evaluating programming ability but bias against other desirable candidate qualities such as experience.

Their desperation to get the job doesn't necessarily carry over to their day-to-day programming. If they take a month to do a week-long task, I would want to know.

In terms of cheating, that can be easily resolved by asking detailed questions about the candidate's submission. It is also helpful to show that you took the time to review it. (There's nothing as frustrating as taking a day for a take home test and getting no response).

> but bias against other desirable candidate qualities such as experience

Won't you see, in the code submission, at least some evidence of experience? The way the candidate organises their code; the overall style; the third-party libraries they choose to use (or not to use); the tests they write (or don't write) — doesn't it all, combined, speak of experience or of lack thereof?

There are still other stages of the interview, where you can assess candidate's experience more directly, by asking about what they worked on.

I agree that it's useful to know when a task that should have taken x long takes 5x or 10x; but it may be because the candidate set a higher bar for themselves.

Totally agree. I've almost never completed a take home project (even though I liked the concept) because there were always other companies I liked that didn't demand as much of my time whose interview process moved faster and who gave me offers before I finished the take home interview process.

> I can say "you shouldn't spend more than two hours on this" until I'm blue in the face, but candidates think they need to polish far beyond what is reasonable

I'm going totally nuts about this particular problem. I have written in as clear terms as possible that I really don't want the candidate to spend more than 2 hours, but I don't feel I am going to convince anybody - especially as this is a (mini)game programming test, and with games you can always add one or ten more things.

We do this at String and Key (and I adopted the process from my prior teams at Digitally Imported. I've posted jobs from both here on HN in the past).

Its a one-hour hands-on programing lab via screenshare - a pretty straightforward task of consuming some data from an API, processing it a little, and displaying it. We use Ruby for our work, and ask the same of our candidates in the lab. No handwritten syntax, exotic/tricky algorithms, or gotchas. Just a task to see how a candidate approaches a problem, deals with requirements, and implements them in our language of choice.

Happy to go into more detail if anyone is trying to implement something similar in their own hiring processes.

I'm interested in hearing more. I'm currently in the process of improving my interview approaches — currently a live coding session with a real-world problem involving an internal dataset.

It sounds like you're on the right track. I think the important parts are:

1) It should be a live exercise (at least mostly live). I'm less interested in the end result than how someone got there, and most people chose to narrate their thought process as they work through the exercise.

2) The candidate should be able to work in their own environment. Online assessment tools can be different enough to introduce friction in a timed interview scenario. I'm interested in seeing how fluid a candidate is with the tools they'll use every day.

3) It should have the same expectations as your day-to-day work. Did the editor have a linter configured, and did they use it? Were tests written for the parts of the lab that introduced nontrivial logic? Is there a README? Is it all one git commit, or did they habitually make commits as they worked?

4) The design of the lab should have a decision point - I'm currently using a lab which could be implemented successfully with either a relational database or some kind of cache store. The strongest candidates recognize there are multiple paths and can justify why they chose the one they did.

This works best for reasonably experienced devs who can actually be expected to create something from scratch from a set of requirements on a short time frame. For more junior people, I'd typically give a version of this lab that I completed myself and introduced a couple of bugs into. The task is to fix the bugs.

Apple did this in one of the sessions last time I interviewed there. They handed me a project in Xcode that had a number of bugs with varying degrees of difficulty — some caused crashes, others were UI glitches, etc. The assignment was to identify the problems and fix them. If you have more than a couple years’ experience the issues were pretty straightforward.

I really enjoyed this session, but I can see how it could be stressful to others.

We used to do this at an old job. We were a consultancy which specialised in a particular e-commerce framework, and were hiring people with experience in it.

So we set up a small project using this framework, and mildly wrecked it. Made some really basic mistakes, made some more subtle mistakes, added a bunch of dead code, wrote good tests for some of the broken stuff, wrote broken tests for some of the good stuff, etc.

Candidates came in, and we asked them to add some simple feature to the app, which we warned them may be broken. We had someone sort-of-pair with them as they work, to watch what they do and keep them on the rails to an extent. We then evaluated whether what the candidate did was sensible: did they run the tests, did they pay attention to the failures, did they find and fix the mistakes, did they get distracted by the dead code, etc. Candidates didn't have to do everything perfectly, but there is plenty of scope to go really wrong. I remember one case where the very first thing the candidate did was delete all the tests. Thanks for coming in, we'll let you know!

The company did quite a lot of rescue missions, where we would pick up a project that another consultancy had started and failed to deliver, and get it live. This interview wasn't just a trick, it was testing everyday skills!

This sounds really cool.

> I think the best approach would be giving the candidate a broken app and have they debug it live

I had a company do this and it went really poorly for me. It was a python app and at the time I wasn't very expert in python. They told me to use my laptop and all I had for python was a vim environment. I knew enough python to do white-boarding problems. But debugging a complex app exercises a whole different set of skills I just hadn't developed in the language. I was also learning vim on the side. I looked like an incompetent amateur.

So there's a language and tooling issue. The bigger problem is that debugging an unfamiliar code base under time pressure is almost as niche as doing algorithm problems under time pressure. I've been on enough company wide on calls that I actually think I'm pretty good at debugging unfamiliar code under time pressure. But if I have someone who understands the code looking over my shoulder, they're usually at least trying to help.

It's still time pressure. It's still a somewhat contrived problem and code base. The interviewer is still comfortably omniscient. And it introduces some more complexity due to unfamiliarity of the language, tooling, code base or problem space.

Ultimately, I'm not sure it's any better than white-boarding.

A lot of these problems are solved with a take home question but then all the problems with take homes rise in their place.

Take-home projects are frustrating when they basically just say "Build an app that does $xyz", and we'll be in touch if we like it.

Obviously no one is going to submit an app that doesn't accomplish the primary task. I feel like you get judged more-so on how you decided to get the initial app up and running, which likely isn't something you'll need to do working there.

> Obviously no one is going to submit an app that doesn't accomplish the primary task.

I would have guessed the same thing before my company started doing homework interviews. But in fact, the ones that don't get to the finish line are sometimes the most interesting ones to review, and they aren't an automatic no-hire. (And we say so explicitly, if a bit more diplomatically, in our instructions.) If someone hands us clean, well-thought-out code and it's clear that they honored our time limit and just didn't quite get there before the clock ran out, we'll still talk to them.

We are a remote company. I give take home projects too. Mostly I give them couple of days but mention they shouldn't work more then 4 hours on it.

Our test is actually quite simple and everyone that did any kind of development should have done these basic things.

But it tells me a log of the current state of the candidate. Most code samples on internet are not production ready code. Security and error handling are mostly not included. So this is where a candidate can shine... but I'm happy if you can string it together working even if it is by googling.

> they shouldn't work more then 4 hours on it

So you're saying spending 3 and a half hours on the assignment is fine. It's not.

Take-home assignments have little to no value for the candidate, they're only feasible if you are jobless or in college. The last thing I want to do in my spare time is to stay focusing on a bogus problem that may end up discarded.

What would you like to do instead?

Talking on-site or on the phone? Is that better spent time

Yes. I worked at Quora and we had a "practical interview" that was exactly this (it wasn't take-home though).

https://www.quora.com/What-is-it-like-having-a-practical-int...

Paid (premium rate) take home tasks is the way to go, after the initial screening.

Take homes are too time consuming.

I would rather spend 3-4 hours working on a take home test rather than writing on whiteboard.

Last year, I did take home test for 5 different companies and all 5 of them invited me to the next round. I yet to have a successful whiteboarding interview.

But did you get any of those jobs? I wouldn't call being invited to the next round successful if you don't get the job. In fact it's a loss, more lost time for the candidate.

Yeah seriously - "rounds" don't mean anything unless they publish numbers, and even then it's sketchy. You need to be able to verify that they're actually there to shrink the pool, not just provide the impression of it. Take-homes are free for the company to issue.

So you're saying you spent 15-20 hours just in order to get interviews. Many, many folks would not have the time for that.

> A major issue is that they require candidates to have free time outside of work to complete them.

Just like onsite interviews. Any interview process will require candidates to spend time outside of work. The only way to fix that would be a true, universally recognized certification/diploma that would prove that you can write software, but I don't see that happening any time soon.

Just like onsite interviews

And unlike an onsite interview, I don't have to take time off during working hours to do them.

My mental model is that I'm willing to spend N hours in total for the job application process. I'm happy to spend some of those hours working on a take home problem as long as that is instead of and not in addition to spending those hours doing multiple rounds of interviews.

I've had companies straight up pay me for doing a take home project as part of an interview. I think that's the ideal way as it gets rid of the potential for exploitation/excessive burden that can come from take home projects.

This can add a lot of overhead, and some candidates may have jobs that forbid them from accepting outside work.

A good middle ground would be to have the company make a donation to the candidate's choice of nonprofit.

>This can add a lot of overhead, and some candidates may have jobs that forbid them from accepting outside work

How is this enforceable? Even in the case that someone finds out about it, what’s the company going to do? Sue the interviewing company for the rights to a few hours of work?

Well, if the employer finds out you violated your employment contract in this manner they could sue you, or perhaps even the other party that paid you. But IANAL, I'm just saying that contracts forbidding side jobs are not unheard of.

>or perhaps even the other party that paid you

What are the damages?

As for suing you? Again what are the damages?

It’s almost certain there are no damages assuming you are just working on a take home project not something that they are going to actually use to compete with your employer.

Just because something is in the terms of your contract doesn’t actually mean you really need to worry about a lawsuit.

A company might fire you if they find out, but the chance that they fire you is to small to worry about.

And even if you don’t do take home work for pay, you don’t want your company finding out you are interviewing. Taking a token payment doesn’t change anything practically.

IANAL. I am just saying that some employment contracts forbid side work. I imagine many companies would prefer not to get in the middle of that, and so might be reluctant to pay people for work done in interviews.

Some people have contracts that forbid divulging any information or techniques they learned during their employment. This doesn't stop companies from asking questions about previous work projects, or asking them to solve problems on white boards despite that fact that some candidates are certainly violating the letter of their employment contracts to do so.

People have contracts with all kinds of stupid things in them. If they aren't practically enforceable, they can safely be ignored. Any company saying they don't pay for take home projects because of employment contracts is just using that as an excuse not to pay.

There are people out their with contracts that say they can't even interview at rival companies, how many companies out there refuse to conduct interviews because of this?

Or just offer both options. I certainly wouldn't eliminate the opportunity for unemployed people to get a little cash, just because of contract issues that gainfully employed people might have.

There are tax implications, for both employer and employee, to paying the candidate, at least in the UK. It would be quite a lot of headache for not a lot of money.

Donating to charity has tax implications for the company too, but they're much simpler.

I'm not a corporate accountant or anything, but I've been on both sides of this and it's very simple to pay people for outside work in the US.

And for that matter, giving real-but-unpaid work to do as part of an interview may be illegal. So paying for your time may be something companies are doing to protect themselves taxwise, not just for good will.

Yes, but...

Now one worries, is this itself a test? Are they judging my desire to donate.

Well there has to be some benefit to passing on the money.

It's kind of like at salary negotiation. You can impress them with how you really are there for your passion not materialism. Or you can get money.

Well, honestly - if I wanted to donate, I could do so myself after they pay me.

Then I'd be worried... are they watching to see whether I engage in hollow virtue-signalling?

No, they're watching to see if you're worried about whether they're watching to see if you're engaged in hollow virtue-signalling. :P

> choose and implement and couple enhancements to a toy app

I think this is a key piece why this kind of project is useful, because that's exactly what they'd do when they start in their coding role: jump into an unfamiliar code base and fix a bug, implement something tiny, make things better along the way.

When hiring for a React role, I went to find an open-source React app and simply asked the candidate to mod it a bit. We then discussed the why and how, but also how they liked the app and why.

We only gave this task to the final candidates, and the ones that had React experience were done in very little time, because typing wasn't the issue here, but reading unfamiliar code and understanding the mechanics of the app.

Writing a hello world app from scratch or coding fizz buzz doesn't tell me enough.

Was this a timed take home project or something with a flexible timeline? That is, was it "You have to respond within 3 hours of receiving this email with your solution" or something more flexible?

I mostly understand why some companies do the x hours routine, and when I've done interviews this way I've historically performed well, but it just felt like it added unnecessary stress. For instance, anyone on the job doesn't have to worry about issues like, "What if I couldn't get the app up and running because I had the wrong version of X installed?" longer than maybe their first day on the job. And if you plan on hiring an engineer for a few years, one day is irrelevant. At best this felt like it favored contractors who were more experienced at jumping into new projects frequently.

In this case it was a flexible timeline. It took a few hours to properly complete, and I was given a number of days to return it.

I'll play devil's advocate and defend whiteboard interviews as compared to take home or online assessments.

Take homes are not very respectful of the candidate's time and, especially if they are not time bound, often leads to a candidate wasting a significant amount of time on these assignments. Plus, there's always the possibility that desperate candidates would cheat and have the assignment done by someone else.

I would only use take homes and HackerRank style quiz for applicants that are from institutions I don't fully trust with quality or applications that are very spammy in nature, i.e from a college/bootcamp that has an abysmal application-to-hire ratio [0].

For candidates from serious institutions I favor a two-step approach: a screening interview with a light coding question and a full whiteboard interview. The screening's purpose is really only to answer questions about the hiring process and check that the candidate can write very simple code by himself when given a trivial problem [1].

Now, a lot of folks do whiteboard interviews wrong. They often expect to get the exact implementation of an algorithm they found in a textbook and for code on the board to compile. This isn't the point of whiteboarding. Doing this only promotes rote memorization. A good whiteboard interview should be a toy problem that can be solved in several different ways by using different strategies or data-structures. The idea is to see how the candidate will break down the problem. Is the candidate able to formulate test cases, write a simple implementation, verify his code and correct the implementation should it fail a test? On the more meta side of things is the candidate able to take feedback and explain why a certain strategy was chosen? Of course it's not representative of real world engineering but it's a good way to peek at someone's ability to debug and reason about programs; these abilities translate well into debugging and design. Especially at the college level, I really can’t make any assumptions on what the candidates know. I’m not judging their knowledge of the standard library of X programming language or the framework-du-jour but their ability to learn it fast.

[0] https://economictimes.indiatimes.com/tech/ites/95-engineers-...

[1] https://blog.codinghorror.com/why-cant-programmers-program/

Reply on news.ycombinator.com