Don’t do the long take home assignments like coinbase
reddit.com
reddit.com
It’s amazing what a short assignment can tell you about a person, and it makes the follow up on site productive because there are so many ways to ask about their work.
Personally for Apple I asked specifically to be given one of these instead of the whiteboard, as I get overly stressed in a “pop quiz” type setup. They still ended up pop quizzing me twice, and one of them I didn’t do well on. It was my take home that pushed me over the edge and got me hired.
If you want me to spend more than 30 minutes programming for you, that's called work, and I would like to get paid for that.
I think it all comes down to personal pref, but the data seems to show that whiteboard interviews are good to filter middle of the road talent, but top/bottom can look the same.
I never take the "a couple hours long assignments" because the scope is never "a couple hours", once company wanted me to build a chat service that would take at least 20 hours of work to get properly done with testing, etc.
I'm talking one that I recall that asked for parsing Apache logs from a stream, displaying moving averages for most popular URLs, aggregates of visitor counts, having high water mark "alerts" for ingress traffic and monitoring rolling averages to "de-alert" when traffic dropped below that, including of course unit tests and documentation...
One, that's more than a couple of hours. And two, that sounds awfully like something you're planning to use, whether I'm hired or not...
This is probably the single best interview session I've ever had. Give-and-take is always good for easing the process, but in one case I had a programmer actually sit down with me at a laptop and pair program our way through an hour task. Still time pressure, I suppose, but it was vastly more natural and informative than any other interview I've done.
I pressed him and he eventually said "I ran this by 3 Senior Python experts I know. All of them said the code wasn't good." No other details.
Before he even gave me the assignment he made his decision. He lied about asking 3 other people.
Coding assignments are always one sided. Unless someone from the company is going to sit with me and invest equal hours into the project, I will never do another one.
They are used by companies to filter out desperate employees (young engineers who don't have reputation to fall back on, or nothing to lose by doing it) and it signals a lower value to the company.
I disagree with the notion of desperate employee — I’m anything but that, but I prefer a take home to show my engineering worth rather than a whiteboard if those are the two choices. Of course if you’re super famous then you probably won’t get either, depending on the company and it’s processes. That said I’ve been down the M&A path and still got (many) whiteboard interviews.
I would be shocked if any company in the world gives you this choice.
I've been at other interviews in which the on-site interview is a walkthrough of the code I'd written, asking questions about why I made various choices. I think this is really a much better way to do programming interviews. It still weeds out people parroting code without understanding it, but it's a better example of what a candidate's actual code looks like -- and for the vast majority of programming jobs, that's probably more important than whether they've memorized algorithms that rarely come up outside other whiteboard tests.
I get a brain freeze when the clock is ticking. I've been practicing with ticking clock but its still hard.
None of the companies showed even 1% flexibility with their process. They say its a standard and they can't compare candidates if they use different methods for everyone.
Companies will differ depending on the role, team, and who’s interviewing you.
Does this happen with experienced positions open in other departments?
Are experienced CPAs interviewing for the new position in Accounting required to balance an example set of books?
Are experienced Sales interviewees expected to go out and sell a sample product before being hired?
A lawyer for Legal expected to write a sample brief or appear before the court in a sample trial?
HR expected to do sample HR things?
Think about it.
CPAs presumably have been externally certified, so perhaps you only need to check "team fit" things.
Sales interviews: if you can't sell me on yourself, why would I hire you? The interview is enough.
Lawyers I would assume have some number of public items for me to look at.
Software engineers: none of that applies. If they've contributed to open source, maybe you can look at that, but many of the great engineers I've worked with have not.
I've now worked at places that did CTCI interviews and "traditional" interviews, and I've noticed no difference in overall quality amongst the employees. I've known plenty of idiots who work at big, famous tech companies, and plenty of amazing people who never ran the whiteboard gauntlet at GooAmaFaceSoft.
My opinion has evolved: tech interviews are the result of generations of cargo-culting amongst a group of people who copied Microsoft, and never really questioned their assumptions. They're just as random and noisy as any other kind of interview, but far more arrogant. Spolsky was right that you should do a FizzBuzz test, but that's it. That's all you need. Everything else should be about communication, personality and the other intangibles that matter far more for every job that involves working with other people (which is all of them).
FizzBuzz + short work sample + strong communication skills is probably not that far from optimal if you want to hire lots of people who are competent and work on problems that they have solved before. Note that I said "optimal" and not "good"; this is still a noisy process.
But if you're doing something where you need to people to excel beyond what they've done before, and perhaps beyond what your company has ever done before, then I think it's naive to think that additional testing for things like on-the-spot thinking, creativity, and diligence under pressure convey no useful signal. This, in my opinion, is an extraordinary claim and requires strong evidence before anyone should take it seriously.
[0] https://www.joelonsoftware.com/2005/07/25/hitting-the-high-n...
The extraordinary claim is that whiteboard testing (or take-home projects, or...well, anything in the current tech interview) does any of those things.
You can't whiteboard-test for excellence. Excellence is both contextual (i.e. it depends a lot on your company, team, culture, etc.), and based mostly on squishy, intangible factors that go beyond "code": picture the brilliant coder who dons his headphones, falls down a hole, and produces a pile of undocumented, complex code of zero business value. It's a cliche, but do we interview for it? No. We ask people to do a graph search on a matrix.
My contention is that we'd do far better with some simple, basic screens for technical competency, and then spending most of our time on communication skills, personality, clarity, organization, planning, business sense and team fit.
...but of course, these are questions with no single correct answer, so engineers are afraid of them.
This is just a naked assertion. This kind of testing happens in many places and industries, and many people seem to believe it's useful. Maybe they're all deluded, but that's precisely why I'm calling it an extraordinary claim.
Do you really think it is obvious, prima facie, that asking people to demonstrate some skills on their feet, or to do a short sample of work for you, tells you nothing about their ability to do good work? If so then I suppose we'll have to agree to disagree.
Edit: I'll add that I personally "interviewed for excellence" as a professor for many years. There was, as far as I could tell, no doubt among professors that interviewing was a non-trivially useful part of evaluating a candidate for graduate school. Again, maybe we were all deluded. But that's a claim that demands some proof.
Provide evidence that coding interviews does what you want it to do. Saying that "other people do it, and therefore it must work" is cargo-cult analysis.
I'm trying not to be a jerk here, but I already know the answer: there's no evidence. People do this stuff for exactly the same reason you're biased toward doing it -- because someone else with a big name did it, and nobody goes wrong by doing what Google does!
"Do you really think it is obvious, prima facie, that asking people to demonstrate some skills on their feet, or to do a short sample of work for you, tells you nothing about their ability to do good work?"
I think it tells you something about that person's ability to do the skill you've tested. Sort of. Under extreme pressure.
Does asking people to code on a whiteboard tell you how they're going to work with their peers, communicate clearly and efficiently, document their code, focus on business goals, and generally not be an asshole (all of which are far more important skills for success in a group)? No.
Even as far as coding ability goes, I've many, many "brilliant programmers" who eat leetcode problems for breakfast but can't be trusted to write clean code on their own. It's a borderline useless signal.
I'm saying "lots of people do it, I can give theoretical arguments for why it might be useful, and it seems common across industries, and this is the only industry I've seen that has a meme about it being totally useless ... so I'd like to see some evidence that it's useless". You're saying "prove that it's useful". We disagree on which direction bears the burden of proof.
Edit: to provide something a bit more explicit, the kind of thing that I would consider persuasive here (again just my opinion) would be several companies that have succeeded like Google, Amazon, Facebook, Microsoft but have employed hiring practices based just on FizzBuzz and communication skill evaluation.
Can I just point out how absurd this argument is? Even putting aside the issue of Salieri’s talent (which he had in abundance), the fact is that most of the Requiem wasn’t even written by Mozart, except for scattered bits here and there. And as far as the movement that everyone knows is concerned — “Lacrimosa” — Mozart only penned the first eight bars!
So yes, even people with a fraction of Mozart’s talent have and will continue to create things that are just as good as his own work. (Heck, maybe even better!) The rest is inspiration and marketing.
Having said that, I appreciate you pointing out these facts about Salieri and Mozart. Thank you for educating me a bit today!
Since a long time I try to educate companies to at least pay a symbolic sum to the engineer, like 10-20 USD/hour.
The reason companies don't do this is because they think that if the programmer fails, paying 100 USD for "nothing" is a waste (x). Most don't value the positive reputation and word of mouth this will generate. They just assume that if you're rejected they will never meet you again and after all you "agreed" to do a free homework task, so you are not supposed to be pissed.
(x) Paying recruitment agencies five figures however feels okay because it is somewhat linked to the success of contributing bringing a new person in.
This angers me, as it suggests the programmer wasting time for nothing is fine..
Also many companies live in the illusion people are supposed to be excited to work for them or even worse they think they're Google although most lack the brandname. They look at how Google hires and think "Google is successful", so we just copy them, which will make us successfull, too. Well, this is a guaranteed way to fail because Google can allow themselves to do this due to Sergey Brin & Larry Page who turned tech into value, which turned into prestige/brand. This reminds me of PG, who wrote in one of his essays: "Prestige is just fossilized inspiration. If you do anything well enough, you'll make it prestigious"
People place a lot of value on human to human interaction.
Or less...
Slightly off topic but a tip to companies that give these tests who might be reading.
If you must give a technical assessment I'm a fan of giving the candidate unit tests and asking them to write code that passes.
Then you have a slightly different unit test on your end. Same algorithm but different input values so they can't just cheat and write code that just echo's the expected results.
If the unit tests pass the candidate passes. If the unit test partially fails on a small number of the unknown values it's not an immediate fail for the candidate. I'll usually look at the code for 10 minutes and see why it failed.
The only time I did one of these and actually got accepted, they used it as a justification for negotiating down my salary. "Your code was about 5 out of 10 on subjective code quality cool points, so we're cutting your offer by $10,000." Then they cut me loose after 6 months because the project was "over." This was for a full-time position.
Now, if I get asked to do a take-home assignment, I ask instead if I can pair program with an engineer for an hour. If not, then I probably won't even continue because they clearly don't respect their engineers' time.
I was told many times that they don't have time to objectively evaluate this so they don't even look at these. Even the companies that explicitly ask for this never look at those.
With code submissions they said they can objectively compare cadidates since they were measured by the same yardstick.
On being measured up with other candidates, I'd really rather they assess me based on my skills and experience, not on a time-unlimited mini-project. The major factor in those projects is free time. I am always going to lose to a person with more free time than me, even though I may outmatch them in skills and experience. And if all they're looking for is someone with a lot of free time, that's not the sort of job I'm looking for anyway.
I like this statement from the replies there: "People here really think a $90k job is owed to them for getting a piece of paper and meeting a few times and answering a few questions right."
I don't know how people are getting hung up on 4-6 hours for a task. I won't even rate any of my software tasks less than 2-4 hours in scrum, because I know that with planning and/or testing no non-trivial task takes just an hour.
Maybe because once you have finished there's a fair chance they won't even tell you what they think?
This thing is often deeply asymmetrical and smart people realize this and try to avoid it.
In other words: if it is well known that doing the assignment will lead to a real on site interview, meeting the team etc you'll probably see more experienced people take the test.
Hey, even just being known for giving real feedback would put you in the upper tiers.
(Is it this bad? Yes! I've crossed the mountains to be told they couldn't interview me that day. I've been held waiting for months, been told under no unclear terms that my results where almost perfect, - then radio silence.)
When you already have a 60+ hour role and may also have travel+++ ?
When other firms are moving faster to the real meat of the process?
If you assume a freshly graduated engineer with the summer off who is only applying to your firm, then a 10 hour assignment is reasonable. Heck an unpaid internship for the summer is fine for both sides.
If you're in a competitive market hiring for reasonably normal positions, you'll have a bad time.
Firms that are offering unique, incredibly highly compensated, strategic positions can require substantial effort and investment by candidates. Or firms where there is so much competition for a few roles that they can be exceptionally demanding despite low pay and poor conditions.
You need to make an honest assessment of where you really fit to see if your demands are reasonable.
Exactly! When time commitment is symmetrical, the companies would only move forward with candidates they are seriously considering to hire. Because for a 4 hour interview session with a candidate, they have to invest at least 4 hours of their engineers' time. Probably more, as there is typically more than one interviewer present at a time.
The company may spend only 10 minutes on average to review an assignment. Most of the submissions are dismissed in the first 5 minutes for something trivial or subjective. The more interesting ones get more attention, maybe 30 minutes of the company's time. Suddenly, "interviewing candidates" get cheaper. And as is the case with supply and demand, decreased price increases consumption.
Before, the company would have invited only 10 candidates for an in-depth interview for each position. Taking 4 hours of the time of each candidate, 40 hours in total, and spending 40 hours themselves. Now, they can give assignments to 240 candidates. That would waste about 1000 hours of candidates' time, while taking the same 40 hours from company time.
If it catches on, for every job, instead of sending 100 resumes, having 20 phone interviews and 5 in-person interviews (arbitrary numbers), you would have to send the same 100 resumes, get 20 phone interviews and 20 assignments, plus a couple of in-person interviews. Suddenly the cost of getting jobs increases multiple times for the candidate, while staying the same for the company.
This is not good.
At the same time, these companies are complaining about there being a "developer shortage".
There wouldn't be an interview.
I really appreciated it, and I’ve decided to adopt that as a policy in my future hiring.
Overall, a 4-6 hour task, to be completed on your own over a 1-week period, seems reasonable to me.
Those who complain that it takes too long to do homework assignments are not being honest about the actual time it takes to prepare for the traditional trivia-based interviews where it takes weeks and sometimes months to prepare. Somehow it's hard to find 2 - 3 hours for a homework assignment, but 6 month to prepare for a quiz interview is totally ok. Exaggerating a bit to make a point ;-)
Technical interviewing shouldn't be a separate skill engineers have to master because nobody wins here. The companies get people who are good at interviewing instead of doing the actual work the companies care about. Of course, the communication part of the interview is important because the engineers will need to work in a team environment.
That's one way to put it!
It's obviously none of my business, but I can't help but be curious whether you are currently responsible for hiring decisions, and if so whether or not you use take home assignments, and finally if so whether or not it is paid for. Or perhaps as an employee the process worked out for you. In either case, more details documenting how take home assignments work best in practice would always be appreciated!
A take home assignment beats many alternatives, but places the entire burden on the potential employee (employers love it!).
Yes, it's not perfect because you can end up dealing with somebody who doesn't like how you comment your code or they don't like your tests... It's actually a good thing in most cases because it also filters out the companies where you wouldn't want to work anyways.
It's an opportunity to have a meaningful and relevant conversation.
This sounds like the key and something many who complete take home assignments don't get.
[edit] Thanks for taking the time to add a bit more context.
When I hire people I craft the assignments for each candidate depending on the role and the person on the other side. It's also important to communicate the expectations and the desired outcomes. And it's also important to give an opportunity to the candidate to have a follow up requirements discovery step where they can ask any questions about the assignment.
There are a lot of details and finer points that need to be considered, of course. The quality of the assignments and how the process is executed are big factors here. It's also important for the assignments to have a well defined scope. Having an opportunity to ask follow up questions is also important (also mirrors the actual software development process where the requirements might be a bit fuzzy).
Homework assignments do have an effect on the job searching strategies. It's not ideal to do a homework assignment for 10 companies at the same time, but usually you can serialize the process where you can work on the assignments sequentially.
If the caliber of people you work with is a major factor in how much you enjoy working somewhere, you should be applying to high-bar companies even though it's more work.
Caliber of co-workers matters more or less with different kinds of work. In a tight software development team, a small number of doofuses or toxic people can make working there awful. In a typical outside sales team, co-worker competence might not matter as much because you're each closing deals independently. So there's no single answer.
The exception was a take-home that took 16 hours due to its massive scope, which in retrospect I should have rejected.
Job I had in 2002 I did whiteboard and a take home assignment. I was hired but once I fixed the speed problem in their code running slow I was fired because they didn't need me because they got my take home assignment and white board info to give to the next programmer they want to screw over.
I'd rather pop popcorn or bake pizzas than work with a company like that again.
The last take home code challenge I ignored didn't stop them from bringing me in for an in person. I gladly coded there for them on a whiteboard since BOTH of our time would be at stake. Apparently my refusal to do the challenge wasn't a big deal since they still extended an offer. Companies will adjust if the majority of candidates refuse to participate in this kind of thing.
I interviewed with Algolia as a solutions engineer. Initial phone screening, followed by a pair programming session, followed by a take home assignment. I spent a good 4 hours on the take home assignment and as those hours passed I began to get more frustrated in the amount of time I was investing in an opportunity I still wasn’t sure about. They were still mostly faceless people and the team still a mystery. They poked holes over Skype in some of the ways I implemented their requirements. Never got an offer and I’ll never forget that time I wasted. Looks like their questions are still up...
First this: https://github.com/algolia/solutions-hiring-assignment/blob/...
Then this: https://github.com/algolia/solutions-hiring-assignment/blob/...
Part of the time sink was learning their product which I had no experience with. The second was deciding which of their js libraries to use (I picked the wrong one first so that wasted a good hour). One hour learning/setting up Algolia, loading data, etc. One hour writing meticulous html/css. 30 min writing js for the wrong Algolia library. 30 min writing js for the correct library. One hour answering the final written questions.
I interviewed with another company (Clara Lending) that actually had a very similar process but we reviewed the take home assignment during the onsite interview. I think I respect that company a lot (even though I didn’t get an offer from them either) because they gave me the reciprocal time in person.
I base this on having done a few and been given absolute deadlines. One time I was moving that weekend, but the company wouldn't budge on their deadline.
Another company-- I do two phone screens, and the guy who would be my manager asks if I would be willing to do a take-home test. I ask him, "About how long should it take me?' "I don't know." "How do you grade the tests?" "I don't know." "Whats the subject of the test?" "I don't know." "When do you need it?" "Two days from now." So I say, "So the only thing you know about this test is you need it NOW?"
It didn't go well. Now I just quote them my consulting rate.
Don't code for free.
I've interviewed at Big Tech and it's always been a time consuming nightmare...HR phone call, 1+ hr. online coding session full of esoteric BS, entire day spent onsite to ultimately get rejected for reasons unknown. This is with me having 8+ yrs experience. I never would've imagined getting recruiters from Google, Faceboook, MS, etc. reaching out and me being annoyed rather than excited, but fuck the CTCI, time-consuming interview gauntlet.
I've resigned myself to working on shitty enterprise CRUD apps rather than somewhere interesting.
This definitely ups the pressure, and things like hardware/internet failures would muddy the waters.
It sounds like even a commitment guaranteeing opportunity for discussion & feedback would be enough to satisfy several commenters here, but that is rare for liability reasons when employees are turned down.
http://www.smashcompany.com/technology/embarrassing-code-i-w...
The tech sector is a small part of the entire economy from a jobs perspective. They can be picky if they want, but the rest of the economy will absorb talent they won't. Always be able to negotiate from a position of power (ie emergency fund, not needing to take a job you don't want or that is a poor offer). Employers are still adapting to the new norm (having to pay market rate and above, not being able to dictate terms, etc).
Personal opinion: If you're in the position to, decline take home work unless you're being paid. Would you work an unpaid internship?
For all of the talk about getting people into coding, especially non-traditional students, stuff like this really hurts the rhetoric. If you're already working 2 jobs, or have kids, etc, asking someone to do 6+ hours of unpaid work is frankly offensive.