A Method I’ve Used to Eliminate Bad Tech Hires
mattermark.com
mattermark.com
As many others are saying, I find that this is the most comfortable way for me as an interviewee to demonstrate that I do in fact know what I'm talking about. Put me in front of a room full of people and a whiteboard and throw something at me, and I'll freeze. Freezing in that situation and being a bad engineer are wholly separate things.
Anecdotally, when I get one of these problems, I spent 20 minutes reading the prompt, looking at resources and making an outline of my code. Then I step away for the afternoon and don't think about it. Let the subconscious internalize the problem for a while and when I come back, I've got a much better mindset to approach it. Solving new problems on the spot is just a skill I don't have. I prefer to think that this is what makes you valuable as an employee and engineer, solving a problem by the end of the day. Not while under scrutiny. Personally, my resume and experience don't get me far interviewing by the big companies, but on the occasions that I get take home code assignments, I tend to get pleasantly surprised and pleased reviewers and it takes me to the final rounds (where I face the dreaded white board and actually-there-is-a-correct-answer situationals).
My two cents, this just really resonated with me. Does anybody else happen to solve problems this way?
As such, if you hire any of those, you're hiring somebody with a proven willingness to ignore their contract, which is a strong anti-pattern.
"Work for contract for other employers" could literally mean I can't help my neighbor mow his lawn in exchange for a beer. In this case what it really means is a non-trivial amount of work done for meaningful profit.
If I fix a friends computer in exchange for a nice meal, I am using my relevant skills to perform work, but I'm not actually violating a contract forbidding outside work in a way that could ever possibly result in damages or firing for cause.
For a potential employee to be so rule-stricken as to concern themselves with performing a coding interview, that would be the potential red flag for me. So this could actually be another positive outcome of this approach.
You can call that a red flag, but it is the reality I live in. Maybe most of your candidates don't have that problem, but I would definitely be eliminated.
But since almost everything we do is in some way technically illegal, it is important for everyone to be able to function under that premise.
I mean, just to cut the check technically the employer would need a W-9, a work for hire contract, and who knows what else. Let's call the lawyer and spend $5k deciding whether we can do this, and then decide 6 months later it's too risky.... Or, we do it, get better hires, execute on our plans better, meet revenue targets, and succeed in the market, by not following all the rules, just the rules that matter.
Knowing the rules to follow and the rules to forget is the hardest part, but a crucial skill in life and business. The more you push the envelope, the faster you can run, and the more likely you are to combust. See Zenefits...
Learning of your little deal to pay them for software development work will be the stick they use to hit you with that interference suit.
The thing is, there are always reasons why this method isn't viable. It's not till you try it that you notice huge gains.
I avoid these problems like the plague and very much prefer to whiteboard. If you pay me consultant rates, I can more easily justify the effort.
Edit: I most likely would have completed it if I were offered some type of compensation, even below market. They claimed it was an exercise and they wouldn't use it, but this was a startup with about 10 employees, and the deliverable was definitely something they could have used. That alone rubbed me the wrong way.
Who would have thought? Maybe - just maybe - not everyone has the same strengths and weaknesses. The real answer here is obvious: present the interviewee with options. Forcing all potential hires to whiteboard is a terrible idea. Forcing all potential hires to do a take-home project is a terrible idea. It's extremely short-sighted and a little pompous to assume that any single interview format one chooses is going to magically sort everyone into neat little buckets of "good" and "bad".
If you force the whiteboarding approach, you are only going to wind up hiring social butterflies who have absolutely no nerves standing up in front of complete strangers and having all the answers on the spot. If you force the take-home project approach, you're turning off a lot of people who can nail a first impression presenting themselves and their skillset in person.
People are different. Applying the same interview type to everyone is going to target a specific set of strengths, and a specific set of weaknesses. And frankly, no business should be composed entirely of one type of person. Some diversity does wonders when assessing the overall strength of a team.
Whiteboarding is a poor reflection of what people do day-to-day, and can be misleading if a candidate practices interview technique over coding experience. In other words, just because someone can regurgitate an algorithm on command, doesn't mean they'll be a good developer.
If all you want to optimise for is finding people who make a good first impression, by all means carry on with whiteboarding. However, if you want to find good developers then there is a one-size-fits-all solution, and that's to see how they develop code in the real world, which is exactly what the 'weekend project + appraisal' approach is a great fit for.
If you're a company that does any kind of business with the federal government, that's simply not an option. You have to be super-consistent in terms of your interviewing and recruiting process across all candidates for a job req and can't do things for some candidates without doing it for all of them.
I think that would be misusing this approach.
We build recruitment software.
The recruitment process is a funnel. At each stage, the employer tries to weed out the bad candidates. Simple maths says they try and use cheaper tests early on in the process when there are more candidates.
For example, first stage is often eyeballing the resume and a quick search on GitHub/SO. 5 minutes, costs say $10, weeds out say 65% of candidates.
At the next stage, we use a more expensive test, since we have fewer candidates to filter.
This particular test sounds like it would sit after an initial phone screen and before a team interview in terms of cost.
Personally I would only use it after a first interview, or tell the candidate that you will decide immediately after completion of the first interview whether it was successful, and if so send them away to do this project afterwards,i.e. reduce double handling
That means you don't care enough about changing jobs. There are people who will happily do that "interview" for free even when being employed full-time. The only relevant question for the company is whether there are enough qualified candidates willing to do the test for free.
...as someone that's fully employed I'm probably only looking at jobs I'm really interested in. I don't mind doing some free work for stuff I'm interested in. I also enjoy doing small unpaid side projects or programming puzzles in my free time...basically the same concept.
That being said, my guess is (ignoring legal implications) paying 200$ or whatever for it is actually more beneficial to the company than expecting you to do it for free so it seems like a nobrainer (once again assuming you can handle legal). Reciprocity and all.
I do something that I enjoy and throw it away afterwards or whatever I tend to do with little programming puzzles and project Euler stuff or I do the same thing and a company profits off it. The profiting doesn't really hurt me so I don't mind it. I'd also argue that if the intention of a company is to profit off my interview code I'm hopefully filtering them out (as they tend to be pretty boring)...having people write code for you by pretending to offer a job is also all kinds of horrible strategically so I don't see the potential for a mass exploit.
I can understand your point of view but I hear the dreaded "but my time is too valuable" more often than "it's unfair that they profit of it". My argument was mostly that...if you want the job, your time probably isn't too valuable to try to get it :)
In California, it's illegal to prevent employees from moonlighting: https://californiaemploymentlaw.foxrothschild.com/2015/03/ar...
I would be much more willing to do the project if I knew the employer would lose $200. Among other things, this shows that I'm valued.
Moreover, if you have a current employer that is sufficiently disrespectful of their employees that they'd even consider enforcing an anti-moonlighting clause in this circumstance, with the preposterous assertion that it qualifies as employment - run away now.
The law is not a programming language.
Well that was very, very dumb of them. Maybe we engineers ain't so smart after all.
And I don't have an employee handbook. I own my own business.
I'd much rather travel and meet you in person and do stuff there, but if you don't have the budget for it, we could figure other stuff out (shared docs, skype, whatever, I've done a few of these on both ends with some luck).
"Since the candidate is getting paid" seems like far weaker motivation than "since the candidate wants the job you're offering," which is what's going to determine my level of effort.
The "not a real problem" thing is key, though. I've tried testing out a few different "this is one of our real problem" things but there's almost always more hidden business rules or assumptions in there than you think.
There's a concept around a lot of preliminary engagements between two entities called "skin in the game." The idea is that when something is completely free the other party will take advantage of it without any real serious intent to follow through on anything.
Mostly this seems a reaction to the homework-type assignments where candidates are expected to spend a lot of time on some interview assignment with very little real cost to the company--which raises the possibility that it's effectively a cattle call.
I could run 50 candidates through a $200 problem for 10K. That's still a pretty large mismatch between "amount of work done by the candidate" and "amount of work done by the interviewer," and is cost-of-doing-business money for recruiting for a lot of companies currently.
Compare that to the cost of me flying people out (which is still done in this approach), or even the cost of spending an hour of mine or someone good on my team's time on the phone with them.
I guess it's just down to the difference between trying to hire fresh-out-of-college (or still in) free-time-to-spare junior devs and experienced people. That's actually a topic I should write a blog about somewhere myself, one day - I pushed pretty hard at my current company for moving towards a different process for industry candidates, and are extremely pleased with some of the people it's helped us hire.
I do agree that if people are flying around, that represents a fairly significant investment in any case. But in Silicon Valley, that often isn't true.
It's hopefully a case of coming up with processes that don't encourage people to waste others' time. But that also assumes some reasonable balance of power in the process.
Even after that, the "experience candidate phone call to see if they're worth flying in" part is definitely the one where I have the hardest time, being not in the Bay Area. I was lucky to find some great local people, where the cost was lower, because that's an expensive one to have to get good at fast. You lose a lot of nuance over the phone, though some other people in the company have been having very good luck with homework in this case - but as a candidate-choice option, where they could still go the traditional route if they wanted.
It's easier than it seems.
Just put a HackerRank test with a single question. "Print the numbers from 1 to 10 [inclusive, one number per line]".
20-30% of candidates won't attempt the test. 20-30% will be unable to give a decent solution.
If you're running 50 people through this, you're doing it wrong (unless you have 20+ positions).
I don't understand why you think improving your employment situation wouldn't be hard work.
The other problem is that when you pay them you generate all sorts of nightmaris legal liabilities.
To save dragging people into an office for that time period, I set up a custom git server that marks the timestamps of their pulls and pushes. The instructions for the task are contained in the README of the repo, so as soon as they pull it they've started.
https://github.com/Prismatik/codescreen
It's critical to me that the limit is adhered to so that over time I can compare candidate responses on a level playing field.
I prefer to see what people write in their normal way of working when they're not under extreme time pressure.
This goes against the core message of the article. The candidate is supposed to think about a problem on their own, and propose a well-reflected solution on Monday. That's pretty much the opposite of "sit down in this noisy corner in an unknown environment and give us something in two hours".
> The other problem is that when you pay them you generate all sorts of nightmaris legal liabilities.
If hiring a freelancer to do a little thing tends to generate all sorts of nightmarish legal liabilities, your company is either a government contractor or it has even bigger structural problems than hiring.
It might not be an issue for your company, but your candidate might have a non-compete (or other clause) preventing them from accepting compensation or even performing work for another company at all.
Things get trickier as soon as money is involved.
This payment looks like easy money for cheaters who would copy ready made solutions or ask friends for help. I'm not convinced, but this is indeed how things should be done in an ideal world.
I guess we could also ask candidates to pay. That is what is done in engineer schools in France. Not sure this is a good filtering method because of the Donner Kruger effect.
Well, it's a good thing we have (now domestic only) banking secrecy in Switzerland, for exactly these reasons. You only get into trouble when your livelihood doesn't make sense from what you earn and own - as long as you pay what's generally considered a 'fair' amount of tax, no-one will blink an eye and come after your documentation. And even if for some reason the government finds out, all that will happen is a penalty tax ranging from 20 to 300% of the missed revenue, usually 100%. There is an explicit difference between tax fraud (document forgery, which could mean jail) and tax evasion (when you 'forgot' to declare something, which leads to above penalty.
Overall, the system is set up so that the government can't abuse its asymmetric power over individuals, instead keeping it as close as possible to an 'employee' of the people.
But you are the one that placed yourself in a bad situation. They have a sane procedure that fully respects you, but isn't fine-tuned to your current problems. You can try negotiating their procedure too, but in their place (what I'm not, I'm in the "can't bill you" place right now), I'd refuse to.
Did you read the article?
That's what the article author is saying not to do, they prefer to see a well worked solution after you define a particular technology, showing they are adaptable and have thought it through properly.
Limiting them to your office, timeboxed, advantages people who spit out lines of code quickly, which is definitely not the right fit for a huge amount of coding positions, especially when you are emphasising logical choices and design decisions.
> Please don't insinuate that someone hasn't read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."
But probably only 10-20% of people will have set that up, and for people who don't have an ABN, you have to basically fill out all the forms you would to make them an employee (you need their tax file number, have to submit a form to the tax office, have to get their superannuation details so you can pay them the mandatory 9% additional retirement savings on top of what you pay them, etc.)...
I was wondering about the tax and legal implications around paying someone for an interview challenge as well.
The overall idea seems fantastic, but paying or receiving money for an interview coding challenge is a little weird.
A company recently asked me to do a task they expected me to take ten hours, unpaid of course. I told them I thought it was kind of ridiculous.
"Well do what you can in two or three hours and if you really impress us with what you have done..."
The company really didn't impress me enough to bother.
As I understand it, the more common form of pressure would come from having to deliver things in a tight schedule, and perhaps in certain kinds of meetings and presentations. Not from being forced to work in a foreign environment with foreign computers & tools and unknown people hanging around, watching and JUDGING you. If a company puts an engineer in the latter position very often, it might be doing something rather wrong. Whereas in the former situation, employers can find relief in at least having colleagues they are familiar with and an environment they're used to & comfortable with.
Yes. And to make sure they don't cheat on it, maybe don't let them use a computer or the internet. A whiteboard ought to be enough. And just to be really sure, make them explain what they're doing in real time as they do it.
That will do quite well, and offer a wonderful alternative to the standard, awful, whiteboard coding interview.
Don't let me connect to internet. I'll also leave my phone with you just to make sure I don't have access to the internet. I don't mind taking the time away from current employment to go through the process.
BUT PLEASE LEAVE ME ALONE THOUGH WHILE I AM AT IT. FOR GOD'S SAKE DON"T SIT NEXT TO ME AND TALK TO ME OR EXPECT ME TO TALK TO YOU WHILE I AM SOLVING THE PROBLEM !!!
Let's discuss the solution or the non-solution AFTER I have spent some alone time with the problem. I don't mind getting rejected AFTER I have made a serious attempt at it.
It is simply beyond me why people think it's all right for a complete stranger to sit next to a programmer while he is trying to solve a problem under the significant cognitive load of a new environment, new people possibly a laptop/keyboard he has never used.
This did sound like a rant :-/ but what I wanted to say was that the process in the article seems fair enough. Interviewees should be given the alone time to solve the problem.
Most people are decent and sociable enough that once they are one board and are comfortable and familiar with the rest of the team they will be able to function and be reasonable in pair programming envoirnment or other technical discussions as well. BUT not when they are in an interview. No matter how it is sugar coated, interview is where a programmer is getting JUDGED and that by default leads to pressure.
But cutting off internet access means no StackOverflow, no online documentation. The test should be very similar to normal working conditions.
I am fine with white board type situations, don't worry about writing code without IDEs etc (in my very first job vi editor was the only thing we had to write code so I still have instinctive memory to be resourceful without an IDE but I love IntelliJ for most things). Happy to discuss/draw out ideas/solutions on whiteboards.
But find it very seriously annoying talking or being expected to talk while I am writing code !
I think the internet-access requirement depends on the problem. If I am expected to solve a problem by first-principles on a core CS topic then I guess internet is of no use ? Online Java Docs for e.g. may not serve a purpose here.
If the problem requires using many different tools/APIs for the solution then I guess you are right, internet access adds value.
Here, I'm like you. But it carries over to normal job - I work best when left alone. Especially when I have a tough problem to solve, just having people sitting close to me makes me frustrated, and if they're talking, then I won't be able to concentrate at all. It takes a lot of energy for me to be able to work in the presence of other people.
I think somehow the world was agreeable enough to drink the Pair-Programming KoolAid. The consulting companies or body shops were happy since they could bill 2x the number of heads for the same amount of time. The "buy-side" of Pair Programming were happy because the KoolAid evidently worked.
Don't get me wrong, I am all in favor of getting my code eye-balled by anybody because I know I don't write perfect code 100% of the times and always open to critique and learning.
Here's a thought experiment. Have two teams A and B. A are pair programmers i.e. |A| = 2*|B|.
Now make team A don't do any unit testing, let them purely rely on pairing to ensure quality.
Enforce team B to do TDD to the best of their efforts.
I would be personally very interested in knowing where the consumer of the code and writers of the code have more confidence on code quality and functional completeness.
Open-plan offices are hip and in Europe maybe even a requirement due to space constraints but I'd personally choose a cubicle over higher pay.
> Open-plan offices are hip and in Europe maybe even a requirement due to space constraints but I'd personally choose a cubicle over higher pay.
Yeah. When I was still a high-schooler and later a university student, I used to laugh at cubicles. Oh, the corporate culture, oh, the rat race, etc. Now that I spent some time in this career, I would voluntarily go to cubicle and even take a pay cut. I now understand that cubicles were a good idea of solving the space/cost constraints.
Hell, every now and then I think about building my own cubicle out of pizza boxes, but I'm pretty sure the management wouldn't be happy about it. Over-the-ear headphones suffice for eliminating sound for now, but I still can't focus when I have people walking behind my back.
Service/backend programmers often extol TDD because it is easy when working in a single language where your only IO is strings (i.e. Undergrad CS)
Front-end programmers may use TDD in certain established contexts, but pair may be more important when there are numerous integrations across unsupported toolchains. For example, when was the last time you used TDD with CSS? Oh? Too hard to verify that the CSS for a page makes it so that you can't click a button or print a page? Oh? Market bug isn't your responsibility?
Yeah, I see a lot of defensiveness when TDD is impossibly hard, so it's not "all that". But, use it if it makes sense. Know your tools and use the right tool for the job.
Not sure what you mean. In 25 years, my latest job is the only one that pair, and even then it's only a few teams in a very large organization.
I don't think your AB test makes much sense. The two practices are orthogonal. Complementary, but orthogonal.
If I had to take one over the other, I'd to TDD, or at least strive for excellent test coverage over pairing, but they really don't have anything to do with each other.
Try enlisting services of a consulting company such as ThoughtWorks, Accenture etc. Talk to the people who are already using such consultancies.
Every single job interview I have had this year involved a pair programming session and lists pair-programming as a "culture-fit" requirement.
The very latest one I had to go to, I had already warned the recruiter that that is not my strong point but he still sent me in anyways !
>But they really don't have anything to do with each other
I was trying to propose that maybe enforced pair programming is actually of no real value or very little value it at all and hence pair-programming has nothing to do with anything actually.
I value formal code-reviews. I have ZERO love for being forced into pair-programming be it in a real job or during an interview.
Somehow pair-programming caught on because KoolAid happens.
done well is the tricky bit. You need people who get on ok with it, social skills need to be developed, and you need discipline around not committing code which hasn't at least been reviewed. Social skills is the hardest part IME: knowing you can ask for a break, some time to think, or to split for some research before resuming work as a pair.
Having suffered through a couple code interviews in Java, I now pick Python when allowed to pick a language. It seems like the language for me that allows me the best chance of coding effectively without an IDE.
I just did a phone screen yesterday that was extremely painful. After 45 minutes of fumbling around the thing mercifully ended and I googled for a few solutions. According to what I read on stack overflow, the problem is unsolvable with the constraints given. I expect to get the "no thanks" email from the recruiter soon and I plan on asking him what the proposed solution to the problem was - not out of spite, but because I'm genuinely curious. Maybe I misunderstood the problem statement.
In the 'old days' you had maybe one or two languages and the language was the thing (not vast libraries like Java or .NET .. oh excuse me, the guys who used to get teased for the size of the CMS VAX reference are barely a man page in comparison to modern libraries) -- and now we have a proliferation of languages. Any web dev these days has to know at least 5, maybe more. Any old-timer has had to learn dozens during their career.
So no, I'm not going to use my precious grey-storage to remember an arcane Perl invocation when I can find it easily on SO.
Perhaps I don't meet your definition of 'programmer', but I've seen many junior devs get trapped in SO 'solutions' and wonder how I can possibly glance through 30 solutions and categorize them: "crap, crap, doesn't know what they are talking about, cargo cult, ... ah, there it is.. that's what we should try."
Benefit of experience?
Then again, I don't really use SO anyway - I know somepeople use it daily.
IMHO, what makes you a good software engineer is not an encyclopedic facts in your head but general knowledge about principles, algorithms, patterns, etc. You don't have to remember all methods of all classes of standard library, for details you can check the web or use IDE. But if someone asks you about thread safe objects you have to know what does it mean. What may go wrong if you fail to make it so? Why properties must be immutable? And what the hell immutability is in the first place? All other things you should be able find in the online documentation / language reference.
What makes you a good software engineer is the ability to ask a meaningful questions. We have a powerful tools at our hands these days. Why wouldn't we use them for our benefit? You don't have to remember everything. It is there, within a hand's reach. Sometimes I have a feeling that asking Google the right question to solve your problem should be another skill tested during interviews.
What makes you a good software engineer is the ability to switch the tools and adapt quickly to new environment. Give him Internet, give him StackOverflow and ask him to solve a problem in new language, new framework. If he got the concept well, if he knows principles, that should be more than enough to jump in. Don't get me wrong here. I not going to underestimate guys with years of experience. I am not saying he will write flawless code. But he will solve the problem - that's a good start, isn't it?
Fair enough.
Thank you for your verbose reply - I'll reflect on it.
I think that we just don't believe you. Did you arrive at every job with perfect knowledge in the areas you'd be working on? Have you not had to learn anything on the job?
If that's the case, then you're vastly overqualified. I feel like I'm being utilized as an employee the best when my manager says: "Hey, there's something weird happening in area X" or "I want you to explore feature X" and I get to dive deep into something for a few days. Our strengths are our ability to learn.
I mention it as a copy/paste mechanism as I am surrounded by people who use it for that very purpose.
I also did have the knowledge required for the job I applied for, and whilst I have learned a lot (off my own back) over the last 18+ years, very little of it is actually required, as demonstrated by those others who get given similar tasks that they can complete using Stack Overflow.
I'm definitely in the wrong place, and after a while I sense I lost a lot of perspective due to my peers / surroundings.
Personally, I never use downvotes for things I disagree with, but perhaps some people who are luckier to work in better scenarios believe that situations like mine don't exist - I don't wish anybody to walk in my shoes.
Thank you. I appreciate you taking the time to explain.
There are good places out there, _even_ if they're low-skilled environment. I think it's totally a cultural attitude, to be willing to take the time to learn, and to have a good attitude and effort about it. You can copy/paste from SO and still be using it as a learning tool.
I use Python, Clojure, ClojureScript, C++11, Javascript and bash regularly. I remember all of the standard library functions and third party libraries that I use a lot. But even after 15 years of using Python, I still have to look up, for example, the datetime module or what functions exist in the collections module. I still need to look up how to use the C++11 chrono library. I still haven't memorised the entire Clojure core functions (there are so many of them!) I still need to look up the DOM API for javascript (but then, I don't use JS that much).
I'm sure part of it is the context switch: if I were using one single language 100% of the time, perhaps I could memorise its core libraries. But why bother when it takes 3 seconds to look up and every professional job I've ever had its perfectly fine (expected even) to look up online documentation on the job.
So why not during an interview too? If your interview isn't testing me on what I'll be doing when I'm on the job for real, then whats the point? Testing my memory isn't a good reflection of my skill when working on real tasks.
Now, when I learned to program, I didn't have internet and worked completely from offline documentation (I learned Visual Basic, C++ and Python this way), so its not that I'm completely incapable of it, but it seems like a terrible use of my resources.
Out of curiosity, what do you do professionally? I'm impressed that you have so much domain knowledge memorized that you seldom ever need to look up anything.
Maybe someone who's only ever done web stuffs in javascript and RoR and the one framework that is being used for the task at the company he's applying for, it's not necessary to check facts.
When you've done two dozen programming languages, everything including low level assembly hacking, embedded & kernel development, graphics programming, network services, traditional desktop software, mobile software, video games and front end web stuffs and you've used more libraries and frameworks than you can count, it is quite possible that you're more than competent enough as a programmer. Yet you might need a bit of a refresher on whatever tech you're being asked to use because chances are it's not the only thing you focused on last year, or maybe indeed you haven't used it in a year or three.
Of course it's not black and white; if a company needs a C expert right now, somebody who knows all the pitfalls and can point at security issues & UB, then maybe he should do well on a test for that, without a reference. Although even then I'd let him use a copy of the standard (or its draft).
Example: you have a failing install of a package in a linux version you never used before where the solution would be to increase ulimit but you are applying for a frontend position so you don't even know what ulimit is, but by a quick search on google for the error message there is an article providing a 1 liner for this issue.
The other extreme end is the young students assumption, "you don't need to know everything, just where you can find it", but the ammount of stuff I have to look up repeatedly increases exponentially with, eg, every wiki-page I read and that just takes too long to the point of impossibility.
That is how I expect people to work once they are hired so I expect candidates to be comfortable with it. Many are not, including me 10 years ago. But most are these days.
I don't expect them to solve the problem perfectly during that time, or even at all. The important bit is to test how they try to solve it, and how they communicate during it. Using Dash, google, SO, etc if fine, asking for help is fine. Essentially portray how they would work if hired.
Many times this has quickly highlighted complete blaggers, or cowboys that should be rejected, or people caught like rabbit in head lights that are not right for us now (but maybe later).
Adding a requirement that completely blows most people's initial domain model is good way to see how people respond, even if the requirement is essentially impossible.
When you say "describe and extend your sample app", where does this sample app come from ? Do you give the candidate a chance to work or go through the sample app on his own for some time before you pair ?
If that is the case then yes, I guess that is probably the correct way to do it.
However of late what I have experienced in general is that I have to go in, meet up people for the very first time in my life, they give me a problem and a laptop/computer/pairing station and a written description of the problem which I get maybe five minutes to assess and understand and then... have a go at it, while the strange (sometimes grumpy) developer sits next to me. If I go quiet for 5 minutes I get the "what are you thinking ?" question and then I have to talk while I write code at the same time !
Maybe it works for some or most of the people out there.
Not for me.
In pair programming you're both working together, complementing each other to tackle a particular problem or issue. In a good pair programming session we'll both work together to surpass an obstacle.
In an interview, one person is observing, testing and ultimately judging the other person, while actively ensuring they do not contribute to the solution of the problem so as not to skew results. Only one person is truly working towards the programming goal, and both participants know this.
Yes, but do you love pair programming:
-with a person you met less than 5 minutes ago?
-on a non-standard programming task you've been given less than a minute to think about?
-with a person who is focused on assessing you rather than assisting you?
-with your future job at stake and potentially your ability to ever interview at that company again at stake as well?
I enjoy pair programming too but I still choke on whiteboard programming questions during interviews. And this is after 20 years in the industry.
One thing I've found from technical interviews is that, due to the pressure to speak and communicate and "show my thoughts", I'm more susceptible to falling prey to trick and vague questions that I would otherwise treat more carefully. This is consistent with scientific results on the way people respond the pressure situations[1]. They tend to fall back on what is familiar instead of trying approaches that may be better. Working alone, there is no fear of judgement if I need to try an unusual strategy[2] that may not work or if I need to backtrack. Let me dabble and doodle on my own sheet of paper and iterate rapidly over whatever comes to mind freely.
Furthermore, turning problem solving into a theatrical event unavoidably increases the cognitive load on an interviewee simply due to the fact that you have to make more decisions. You have the technical decisions like data structures and algorithms which are normal and expected, but on top of that, there are all these non-engineering decisions that you have to make because they want you to talk about how you're solving the problem in real time. Should you mention some fancy algorithm or approach that you are considering? You must choose your words carefully since anything you say can be used against you.
I'm not claiming that everyone works better alone or even that I don't like working with others. I just think that the interviewee should have the option of working on the problem alone for up to 30 minutes. As suggested by parent, we can discuss the solution or non-solution after I've become familiar and comfortable with the problem. One of the worst feelings is when I fail to solve a problem during an interview, only to have the solution come easily when I'm able to tackle the problem in a pressure-free environment. Expecting some kind of portal into someone's mind while they solve an unfamiliar and maybe tricky problem strikes me as overindulgent and voyeuristic. It is a flaw in the current technical interview process and the only way to mitigate its effects is to just get more comfortable interviewing.
[1] - https://www.psychologytoday.com/blog/choke/201106/flocking-t...
[2] - For tough problems, tend to use my own idiosyncratic variant of Z-Notation. I have a very high success rate solving algorithm puzzles when I use it but it is easy to forgot to use it under pressure of an interview.
That's a very interesting perspective actually.
If this process prevails then only a certain personality types will be able to work at companies which conduct such interviews.
The danger is that when the biggest and most successful companies (who's founding members didn't go through the same process, on the contrary were most likely solitary coders) conduct such interviews, the smaller and actually very nice places to work tend to copy the same.
I am somewhat in favor of the likes of Google and Facebook who need to filter an ocean full of applicants following this process.
But I have seen smaller companies copying the same process assuming they will find the similar hires as G and Fb. The funniest/hilarious part is when the recruiters start saying "we have a very bar" with so much conviction. Yes, you are a startup, you have a "very high bar" and you need to hire people who will scour the internet and integrate the various bits they find because they need to take the product from A to B YESTERDAY !!!
Although having said that, I think I may be a bit different from what you described. I actually don't mind whiteboarding or coding with pen and paper and I noticed for me it is a lot easier to "talk my thoughts" while whiteboarding. On the contrary, I get seriously annoyed if I have to talk while actually writing code and if it is during a job interview, I just choke.
> Expecting some kind of portal into someone's mind while they solve an unfamiliar and maybe tricky problem strikes me as overindulgent and voyeuristic. It is a flaw in the current technical interview process and the only way to mitigate its effects is to just get more comfortable interviewing.
Yes. We are problem solvers after all. A way shall be found.
Ugh. The worst step of interview process. Remember that guy who wrote a post about Google's hiring process, where they outsourced the asking of technical questions to untrained call-centre staff.
"Standard way to allocate memory in C?"
"malloc"
"Incorrect. Actually there's this cool library that does it for you with additional checks. Sorry, you have failed the interview"
"Wut?"
I've always found that those kind of interview processes put far too much stress on being correct to their "ideals" rather than be consistent with your own and your team's. I.e. I don't care if you name your column "domain_id", but for christ's sake name the other column "account_id".
I didn't get the job, but it was an interesting experience. Certainly selects for a certain kind of engineer, and would not be my preference for how to hire people if I was setting up my own operation.
Hey, I had this experience with Amazon when applying for a software related position based in Europe. For the entry screening I got a call from Indian phone number and a person with Indian accent was clearly going through a list of various quirks an features of a programming language. Wasn't so bad though... the second round with MBA product managers was much worse:)
Also there are many indicators along the way, i.e. which parts the candidate got stuck at, did it take them 1 or 3 hrs to get to the same point (i.e. basic wiring of DB calls). I think all that would be lost if they did it on their own time.
Pressing them hard for information on Monday will expose this.
Every team needs a couple of people who can solve problems on the spot. They're going to fix your production issues. You need a couple of people who want to know why all the time. Someone with a big picture view. Someone with good aesthetics. Someone schedule driven, and a process nerd.
But they don't have to be the same couple of people. Diversity is more than fine, it's essential to avoid echo chamber syndrome. Some of the worst projects I've worked on had little to no dissent in the ranks. The code was self consistent but entirely wrong (bad info architecture, slow, brittle, impossible to learn after the fact).
But everyone tries to hire people that think exactly like they do. If we weren't so bad at determining that we'd probably never make anything interesting at all.
That's definitely a thing, and I believe it applies to some extent to everyone.
But here's the other side of the coin - the ability to operate efficiently when the world is burning. I've been in the industry for a while, and this sort of thing definitely happens. But it's very hard to simulate in an interview. You don't know who the adrenalin junkies are on your team until PagerDuty lights up glowing red across the board.
The only indication I've seen is the correlation with a preference for risk-taking activities, such as sky diving, etc. But it's quite circumstantial. And there's definitely also the true reverse of it: some of the car racers and bull fighters out there are not good team players in times of peace.
Well, if at least a few key players can handle pressure gracefully, I guess you'll be fine, since it's a skill rarely used. You don't need everyone to be able to switch to catastrophe mode at a moment's notice.
Another problem, depending on your philosophy. Specifying the exact technologies someone has to use means that you've closed yourself off to generalists, unless they have time to learn a whole new stack that weekend. Not that it's rocket science to do so, but that turns a two hour project for me in say Angular into an eight hour project in your favored stack. I can always learn your stack on the job, as any good engineer can.
This format also doesn't appear to want full days: it should be an hour or two Friday afternoon, plus a half of Monday. Two part-days isn't really that worse than a full day off (Mileage may obviously vary; particularly the mileage to get to the interview). The initial interview could also very easily be done remotely.
Even if it is a bit higher-impact, I would still prefer it simply for the greater quality of experience, and that's before throwing in the payment.
Luckily not everyone thinks solving leetcode problems is a good way to hire.
If that happens consistently for months at a time, then as said the PM is severely fucking up. And if that wasn't communicated as an expectation before compensation was negotiated, then the company is probably fucking up.
As the quip goes "If a race doesn't have an end, then it's a death march."
B) If you're working a job like this you're working a shitty job and ought to quit. Unless you're maybe hacking the Gibson to stop some oil tankers from going belly up there is hardly a reason in our industry for this kind of shit.
Code now or die isn't usually something most people face in their day job.
The thing is, it's a vastly different kind of pressure. When I am coding as part of a job - by myself or with others - there is no inner monologue in the corner of my brain doing something like this:
"Could do that or... no damn, that's O(n^2), damn I'm taking too long that one looks bored, if I instead put those keys in a hash... no, I'm sure there's a better way, running out of time though, is he looking at his phone now? Okay, what was the profile of merge sort again? ... oh, shit! She felt like she had to offer me a guiding hint, I'm doing it wrong!"
And it never stops going. It picks apart every choice I make, and by five minutes into the interview I come across as a sub-articulate mess.
In no other high-pressure situation does this happen to me. Hell, this past summer I actually spoke at a conference and I didn't have the kind of trouble I do in an interview.
Whiteboarding, paired coding, participating in architecture design - these things come easily. In the context of an interview, though, it goes out the window.
That kind of scenario happens to me daily.
If you learn something from someone, is it that much trouble to give them credit for expanding your horizon?
If this mindset is recursive down your organization's hierarchy, doesn't it create a scenario where a legitimate critique never gets heard because people don't question the decision of a higher ranked employee because there is the impression that they know more than you?
Seems like you and your employer are worse off, but your ego remains intact.
Not all jobs give you all the time in the world on a test server without interruptions.
Performing well in this kind of scenario is much less about algorithms and data structures, and much more about systems thinking and the level of your understanding of your system.
System knowledge and knowing which one bolt to replace or turn is almost always more effective.
PS: Not to mention, if we're getting into algorithmic complexity levels of engineering on a monkey patch to a production system, then so many alarm bells are already ringing.
An interview situation is much more adversarial: They are looking out for reasons not to hire you, you are competing with others for the job and only one can get it! No such consideration on the job, unless the work environment is completely and ridiculously broken. When are you ever in a work situation where several people work on competing solutions and everybody else but the person who made the winning one is fired? At work you are working together, and to solve a problem, not to week you out of the pool.
If there's ever going to be a situation where something costs $100k/hour if it's offline then you have 5 test servers and 2 or 3 load-balanced production servers so a failure doesn't take you offline, and you've written procedures and processes to handle critical problems long before they happen.
You definitely don't try to hotfix things while the foreman complains. That will make things worse.
Also...these situations can be trained for to certain degrees. There's well established precedents in crisis management. If you write code for a customer with a 100k/h downtime risk you should have done a very thorough risk analysis and have mitigation plans for most scenarios. The unknown unknowns that can happen are obviously the hard cases but you should be able to justify that these are going to be expensive no matter what. But even for these cases bricolage is trainable to a certain degree.
If you know there's 100k/h costs of failure you should also very specifically mention the fact that you are looking for these stress coping skills in the job description (and as such the interviewee should be prepared).
But I will not work in a company where that's the norm.
I value my health more than the potential money from that stressful job.
You've got your main client staring over your shoulder, arms crossed saying fix it, and every two minutes asking "is it fixed yet?"
You damn well better be able to code under pressure... And like a freaking ninja
Coding under pressure is quite different from coding on a white-board in an interview, often on an arcane CS problem.
Or a procrastinator ;)
"Hi, one coffee please."
"Sure, just one minute. Ok, first I'll pip install stripe…"
An interview situation is much more adversarial: They are looking out for reasons not to hire you, you are competing with others for the job and only one can get it! No such consideration on the job, unless the work environment is completely and ridiculously broken. When are you ever in a work situation where several people work on competing solutions and everybody else but the person who made the winning one is fired? At work you are working together, and to solve a problem, not to week you out of the pool.
Being able to 'code under the pressure of a deadline' is very different from 'solving a tricky problem with someone staring right at you'
Not everyone needs to be able to work well when the world is burning. Its not part of many jobs. However, if it is part of your job (eg your the guy who has to fix stuff at 4am) then I see nothing wrong with asking candidates to do timed work in an interview. I just think most jobs don't or shouldn't require this.
Congratulations that you can do all these things -- but unless 100% of the engineers in your company need to be like you, you might not want to use your list as hiring criteria.
I have never in 12 professional years been given weeks to test anything. On only a couple of occasions have I been given 3-4 days to extensively test a project with multiple months of development time behind it.
Clearly you're talking about the project where the developer(s) told you very explicitly that it could not be done in less than 3 months. So you just assign the Jira ticket a 3 week deadline anyway. And then flip out when a half-assed project fails to ship. Welcome to the world of software development.
The type of situation you're describing is is why I avoid working for any sort of shop or agency.
For example, I may say please use functional reactive
principals in JS to solve this problem, but the decision
of using Kefir, Bacon, or RX is left up to them.
First, this model alone doesn't scale. Big companies interview hundreds of candidates every week. It's not possible to design and evaluate a weekend project for each one of them. This doesn't mean that they can't have projects, but early filters are needed. That leads to the classic phone and F2F interviews to rule out those who can't code. If you freeze in one of those because of the pressure, you're out.
Second, for simple developer positions, such weekend projects might be very good. But if you are trying to hire a senior developer who is going to solve problems at big scale, then the ability to create a simple movie catalog doesn't tell you anything. And there are no big scale weekend projects. You have to ask about algorithms, data structures and other CS topics. These things are important!
Bottom line, this is a nice hiring technique suitable for small companies at a small scale. But it might not be that useful for big companies.
Which is a bummer. When I interviewed at Amazon a few years ago, it was a day of coding on the whiteboard. Turned out I didn't have the Java experience that group wanted, and I'm allergic to pager duty so that was a non-starter for me.
Can't you just design and evaluate one project that they can all do?
The common approach for big companies is to declare that false positives are way worse than false negatives, but if the problem is that you can't find enough developers to fill your roles, it may be better to have false positives coupled with a trial period or something similar.
"You should just find that out upfront" I hear you saying. Well that is not always possible. Recruiters either don't know the range, or lie about not knowing, or you have to submit a big packet of info (cover letter, history, resume, homework) before you can even talk to a human about salary.
Another thing that companies could do better is think really hard about "required" and "desired" skills, and be truthful. Lots of companies list things as required which they're probably willing to bend on for the right candidate. All this does is scare off potential candidates (odd when you're complaining about that mythical shortage of engineers). On the other hand, recent events have encouraged me to not be afraid to apply for jobs that I'm unqualified for :) so maybe it's not such a big deal.
> There's so much song and dance that has to go on
> before you get to the meat of things
I had to say no late in the process to several companies which offered me the contracting-to-maybe-hire dance.
Never again will I do contract-to-maybe-hire.
> but companies need to start specifying their salary
> range up front
> I've been having a lot of time wasted lately by
> getting through Tech Interview #1 or Homework #1, only
> to find out their max salary is laughable for San
> Francisco.
"I am interested in roles paying above my current salary of $xxx,xxx only, and am looking for an improvement on that to reflect my increased experience and the risk of moving roles"
> Recruiters either don't know the range
> or lie about not knowing
> you have to submit a big packet of info (cover letter,
> history, resume, homework) before you can even talk to
> a human about salary
That's funny, I thought this was a employee's market... Better get with the program, bub.
I'm actually in favor of 100% transparent salaries. I've yet to hear a compelling argument against it that wasn't simply "it feels icky."
> I'm actually in favor of 100% transparent salaries.
> I've yet to hear a compelling argument against it
> that wasn't simply "it feels icky."
Are you really advocating that people disclose their salary to a recruiter, voluntarily?
What terrible advice. "How to immediately throw away your best bargaining chip 101".
> Are you really advocating that people disclose their
> salary to a recruiter
> What terrible advice
> "How to immediately throw away your best bargaining
> chip 101"
If they're an external recruiter, they have no control at all and are incentivized to get you paid as much as they can, as they're paid on commission.
Not only is it obnoxious to read on mobile, as horizontal scrolling is necessary, but it also fundamentally doesn't make sense as you should be able to copy/paste whole sentences.
One might expect that as a recruiter/(ex-?)developer you'd realize how painful it is when you use this formatting.
I agree with photogrammetry in that it's essentially unreadable on mobile portrait as you have to horizontally scroll each quote you want to read (then scroll back and again if it's multi-line).
Shorter >'s with italics are my preference.
That might be my salary. It might be my floor for moving to a new, more expensive, area. It might be what I think my current salary should be.
Depending on how things like bonuses are done, base salary alone isn't super relevant anyway. There's a potentially very big difference between a company that says "we give bonuses of up to 20k per year" and "our bonus target for good performers at this level is 25%."
Maybe 9 times out of 10 the recruiter says "sorry, that's not going to happen." But you only want to find the ones who say "we can work with that" anyway.
This may not be the best way to absolutely maximize your total, but if you pick the right floor, you can save a lot of time up front. And the way to really maximize your offer isn't to hide your current salary, anyway. It's to take that first offer to another other company you're interviewing with, and see if they can beat it. That's your best bargaining chip.
> State it upfront when you submit your application, rather than "find that out"
As an employee, I would say the costs of specifying my expected salary range upfront outweigh the benefits.
> As an employee, I would say the costs of specifying
> my expected salary range upfront outweigh the benefits
Got an awful offer to relocate to an area where 1/4 to 1/2 the house costs twice as much and I would no longer be able to work from home. I would also lose a significant portion of my life commuting. The offer was comparable to what I made working remote from Boston.
I don't mean comparable in the sense that they tacked on a CA bump to handle the cost. Comparable as in same money, but I would have to live in CA.
Ended up working remote for more somewhere else. Everyone's time wasted. I don't get it at all. It's not like they didn't know it was low.
Seattle is not even close to being as bad as silicon valley in housing costs.
But don't worry, they'll make it up with about $10k/yr of stock grants which you don't get until you've worked there 4 years.
I picked another lesser-known company and am making - no shit - about $70k/yr more than Facebook was offering. With stock grants that also might be worth about $50k after 4 years, though admittedly, the stocks are a much bigger gamble in this case.
If you are in the higher range of salary, you gotta find out which local employers/industries are worth talking to and limit your application to them. There probably aren't many.
• What 2 hour project can you assign that would let the candidate go into enough depth for you to judge their code, design decisions, architecture skills?
• If you give them more time (e.g. a weekend like in the article), is it still financially viable to the company, especially at $100/hr??
• How do you filter out the candidates that aren't serious about the job but just want to make a quick buck?
• What about all the candidates out there who you can't legally pay (e.g. workers on a visa)?
This works when you are a small company that needs to pick 2 out of 6 total applicants, but otherwise completely unfeasible.
Where I work, our take-home is a simplified version of a task that's common enough (in our field) that we built tooling for it. It's generic, it tests understanding of a few things, and we've gone through three iterations of our own tooling so we know there are multiple viable approaches.
As for judging decisions, that's why the interview that goes with it is a collaborative code review. You treat it the same as if you were reviewing code a co-worker had written; ask questions, discuss tradeoffs, etc. etc.
Maybe it would be better to explicitly state that the results of the work are to be released under CC-BY?
> "Create a single page app that lets me enter movies in my home movie collection, store them in offline storage and search through them. I want to search by Genre, Title & Actors."
Significantly more than 2 hours, unless you are looking for a quick hack and for the candidate to repeatedly explain "I would have done it this other way if I had more time..."
Part of software development is tailoring a solution to the scope of the problem, and this includes taking into account the time and effort available to come up with a solution. Someone who looks at that problem statement and designs something that will take a week has missed a very important requirement.
You could reasonably scale up any problem to take an arbitrarily long amount of time, by adding onerous requirements (must have 7 9's of reliability). What I've found more rare and precious is the developer who can limit the scope so that items that are not up for negotiation like time (2 hours) & resources (1 developer) are respected.
I fully expect a developer to state their assumptions up front, so that I can judge their design based on the stated assumptions and not some unstated imaginary assumptions. It also lets me see how or if they prioritize.
What whiteboard interview would?
Let me answer that with an example. In my previous job, I had to implement a simple username/password based authentication API in the first interview. The problem was explained to me. A two page document with a written specification of the problem was also provided to me. After I read the document, I was asked how long I would need to solve the problem. I asked for 40 minutes. Then the interviewer left me alone in the room with a laptop. After 40 minutes, the interviewer returned and went over the entire code I had written, discussed the API design decisions I had taken such as the corner cases that were handled or not handled, exceptions that were thrown, how empty or null usernames or passwords were handled, what error messages were generated, how concurrency was handled, what performance problems could occur if thousands of requests invoke the API per second per thread, how the password was secured, etc.
After I got the job, I learnt that my colleagues had also gone through a similar coding interview with different problems of similar size and scope. Some people asked for 30 minutes while some asked for 90 minutes. Therefore, I believe that it is possible to have a tiny 30 minute to 90 minute project that lets a candidate demonstrate his or her software engineering skills.
I felt this method was very effective because it evaluated a couple of key skills that were important to do the work that we were doing there: time and effort estimation, API design, writing clean and correct code, robust error handling, and design decisions from performance and security perspective.
I didn't get paid for that 40 minutes but it was part of their half a day onsite interview process, so I did not have to spend any extra time apart from the time I had already scheduled for visiting their office. Also, they did provide me free lunch, snacks and drinks, so I didn't complain. I wish more employers did this.
Well, the movie project suggested in the article was a pretty reasonable one.
That feels like premature optimization to me. I can see how it'd find generally good web developers, but I can also see how it'd produce just as many false negatives as Google's interviews do. So is it really that much of a step forward?
I dunno...I've worked with a lot of people who could code a simple app but completely fail when it comes to serious programming.
Write a program that reads a text file, and output each unique word with the number of times it appeared.
The program must be done in a programming language among: C, C++, C#, Go, Java, PHP, Python, Ruby.
print wordmap
This is on the level of a pretty simple first phone screen problem.
A couple of knowledgeable friends can do wonders, and if you also prep with them a little you can probably answer all relevant questions and beyond about the code on the Monday on-site interview.
Sure, that's cheating. And even if it wasn't, I would never do it for the wrong expectations it would set upon me if I actually got the job. But the ease of "embellishing" one's CV always comes up in these threads/posts, and while I wouldn't do that either, I kind of fail to see how this home project approach solves that problem.
Anyone able to ELI5?
That sounds actually brilliant. While some might argue that this comes back to the "whiteboarding under pressure", honestly if you have already written the code from scratch you definitely should be in a much better position than with the usual "implement a red-black tree insert, you have 15 minutes" whiteboarding tasks.
Seriously though, I guessed that many companies probably must have this kind of laid-back attitude on the matter, since it would be stupid to presume they are ignorant of my original point. Still, I think it might be a little unfair to the people going by the rules-as-written, which somewhat diminishes the consensus of awesomeness of this approach.
That is, if the interview process is not actively trying to choose for hustle... :)
But I'd consider that fairly unlikely. Plus, starting from a position where you don't even trust someone enough to behave like a mature human being isn't a great way to begin a professional relationship.
This makes sure that if someone has not solved home assignment with other's help.
The interview problem should only serve to be a conversation catalyst. If they got some help with the problem, outsourced the problem or simply googled prepackaged solutions to the problem then they are going to have a really hard time talking through the context of how they solved the problem.
What you're testing for here is communication skills and, if possible, a really basic datapoint that may indicate their technical ability. The latter is very subjective and incredibly difficult to test for but the former is arguably a lot more important if they are going to be working as a part of a team.
Communication skills seem to be held in much lower regard despite them being the key thing for effective day to day work. I can manage / handle working with someone who needs more time than a "rockstar" or hasn't memorized TAOCP if they'are able to communicate effectively. Outside of very specific roles, non-trivial software development is a team effort. It's usually extremely obvious within 15 minutes if a candidate genuinely gets what they've coded or is winging it.
> Outside of very specific roles, non-trivial software development is a team effort.
Couldn't agree more, and I have to say I'm rather baffled that so few recruitment processes are fine-tuned for this. Or maybe I should say that the image one easily gets is so. I myself have had pretty well balanced interviews that also tried to assess things like this, but I've always kind of considered myself just lucky.
When one takes your points into consideration, I can definitely see how having the take-home assignments offer value above the usual whiteboarding+bullshit bingo.
Nothing, and that's fine. In fact we encourage candidates to use all resources on our take home interview test. I would probably be impressed if someone setup a hackathon to solve a rather simple take home problem.
A take home problem is simply another signal, and a jumping off point for discussions. If someone used some novel approach we'll discuss it and find out if they actually understand what they did.
The real surprising part is how many take home problems we have received that either do not compile, do not pass the test data we give, or look like the instructions were completely ignored. It ends up as a low effort FizzBuzz on our side.
This sounds great. In my response to another comment I was worried if accepting-but-not-encouraging this kind of behavior would put the interviewees on uneven ground. But if you tell them in advance that basically anything goes, it's fair for all parties.
> The real surprising part is how many take home problems we have received that either do not compile, do not pass the test data we give, or look like the instructions were completely ignored.
Sounds horrible. Though I guess that's only a good thing. Seems like the assignment is doing its job in those cases pretty well :)
Like Fizzbuzz, the point is a quick way filter out candidates.
Cool, can they do that for all of their work from now on? Heck, I'll hire all of them if they're that cohesive a team.
This is true for nearly anyone with a standard SV job. So if you're willing to restrict your pool to the currently unemployed, this can work, but I think you'd rather not do that.
By that interpretation, you can never legally take part in a whiteboard coding session during an interview either since the code you whiteboarded also belongs to your current employer.
There is such a thing as being too literal.
> and definetly apply in this situation
I have received legal advice in my jurisdiction on this specific issue such that I was part of a team that decided not to offer pay in situations due to this advice.
In this case, the hiring company just seems to want to say "This isn't an actual problem we have, but we use it as a problem set, and you should be paid for your time" because this goes above and beyond an in-person interview.
Your (existing) employer (in California) wouldn't begrudge you for taking part in a user-research survey that resulted in a $100 Amazon gift card. This could be done the same way, the difference is that you're writing code (which is what you do for a living).
Again, I am not a lawyer ;).
I'm probably in that pool. Really, I'm not creating anything of massive value by doing a little example project. If my employer wants to sue me over it they can go ahead. I will probably lose, but they will have a rep for suing people for taking interviews.
Some people negotiate over such things or have them struck out from whatever HR paperwork. Consultants definitely have such crap taken out from contracts and replace with more straight forward and standard non-disclosure agreements (or charge a lot more - clients will pay if that stuff is important).
In other words, employer/client can put whatever crap they want and you need to always read and understand contracts you sign, as an employee or contractor.
Really? I've found blanket IP ownership to be more common than noncompetes, at least in US outside CA. My experience, is that noncompetes are more likely to be limited to key personnel, whereas the company IP is sacred and must not be tainted, and the employee is not to be trusted to come up with their own material on their own time, at least not without sign-off.
Blanket IP is really back-door antimoonlighting anyway. You can't very well moonlight as a software developer if your employer is encumbering your IP.
I'm sure the California startup world is different from my experience.
If you think they would, you're probably overvaluing what you can accomplish in 2 hours, and you're underestimating the cost of a lawsuit to them. I can't imagine a situation in which they would actually pay any of their legal staff to even look at something like this, given all the rest of the things the lawyers have to do there.
I worked for Google for 11 years and wouldn't have hesitated to take a paid interview as long as it was all done in good faith (i.e. for the purposes of getting another job).
Google is trying to hire good engineers and retain them with interesting work and good compensation/perks, not shackle them there by legal means.
In the country I come from, the master/PhD students working at a company are under a special set of contracts & laws (slightly different from the usual employment rules). Legally speaking, the company cannot forbid them to perform what is mandated for their school & necessary for their grade.
> DON’T use a real problem because of tribe knowledge needed to fix.
I do not see how this is different from a whiteboard problem, literally speaking. It's a "problem", not a "product", and presumably the solution is throw-away, or at least not used for commercial purposes.
If the author was talking about tasking potential hires with a small, real consulting task, then that would be an issue, whether it was paid or unpaid (and by industry norms, using applicants to do unpaid work is extremely unprofessional, though not unheard of).
> California ... has a law on the books that generally prohibits employers, on public policy grounds, from making claims to IP generated by employees working on their own time and using their own resources.
But I am afraid that they just slipped it in and you were too worried to ask it taken out. This is definitely something one should look in the contracts and ask to be redacted (IANAL).
> I legally can't take part. I have a job now, which
> claims ownership over any tech-related IP I create
> ... This is true for nearly anyone with a standard
> SV job.
Is there an omitted qualifier to this, e.g. ...while on company time; for IP with sufficient correlation to your primary role?
I'm having a difficult time swallowing the legal enforceability of "all-day-everyday-any-tech-related-IP" outside of military personnel subject to the UCMJ. Would that suggest that you technically couldn't contribute to an open source project?
(xoogler, with some knowledge of this area)
Also, fuck Google and their ridiculously limiting anti-entrepreneurial IP agreement
Use that paid time off, or "flex" hours.
And regarding the paper trail, payments under $600 need not be reported to the IRS. 1099's only get filed after you have been paid over $600. Ask for cash.
There are laws that are really there to be followed correctly, like the ones about theft, murder, etc. But there are laws that basically just enable people who you piss off to get back at you. And the latter ones are a risk you must take when you want to improve your life.
Sorry for ranting, I'm just pissed off at how an industry that brags about how they pamper their employees can treat them like garbage during the interview process.
If you apply for say 4 jobs and they all want 2 hour work samples you've just taken on an entire extra work day to apply for jobs. I can't even just take a day off work and batch them because of the way applying for jobs works.
Honestly, I don't know the right answer. From the other side of the table I like this approach because it lets me assess how someone thinks about solving problems and writing code, but its just reinforcing a culture where only young people without any real responsibilities outside of work can get into the industry.
What am I going to do? "Hey boss, I need to book another 2 hours off as holiday so I can apply for jobs elsewhere."
When you want to hire people who gets the job done it is no harder than:
1. Please tell me about a project you have done, a sw book you read, a hard bug you solved, really anything you like about computing etc.
2.a (This guy seems smart) Let's discuss compensation and you can start your probation period asap.
2.b (This guy is not smart nor passionate) Thank you for coming in, but we don't want to hire you.
edit: I used this method with great success, imo programming exercises are just an indirection that only adds noise and loosely correlates with on the job performance anyway
I see a lot of comments here about how they would never waste their time on an exercise like this, but a small project for the purpose of an interview can probably be done in less time than any of your public projects.
However, it would be totally unworkable if most companies were to adopt it. Try finishing a dozen homework programs during a typical job search.
Even a simple phone screen is going to eat up 2 hours of my day, much less the full day interviews that most big tech companies do.
Personally, I'd rather be given a weekend to work on a smaller project rather than be tested on algorithms that are available by common libraries of most programming languages.
On the article, Amir wrote: "They refuse to do the project because “someone will hire me without it”.
On book "Smart and Gets Things Done: Joel Spolsky's Concise Guide to Finding the Best Technical Talent" Joel said that the "the best developers usually have lots of proposals, so they don't spend time doing tests for interviews".
I think Joel and Amir are right.
I think Joel means "unpaid tests". I routinely turn interviews with unpaid projects down, because each interview process gets a number of hours of my time for free (depending on how much I want to work for that company) and unpaid tests/projects usually way way overrun that time.
Or maybe that's what you said and I misunderstood?
I want to move to this style of hiring; however, I fear that because it's against the grain I may miss out on good candidates. Is that unfounded?
It hurts the size of our funnel, but I think the results we get are pretty good.
When I was looking for a job last year, I liked that it was something simple and low-pressure (compared to typical coding-interview practice) which approximated how programmers actually work. It was OK to look something up in the documentation if I didn't have an API memorized off the top of my head. It was OK to change my mind partway through and realize I could do it more cleanly another way. It was OK to pause for fifteen minutes and get something to drink.
Now that I'm on the other side of it, I like that it's a more natural thing. I like that I'm not having to loom over someone or try to fill awkward silence while they live-code. I like that I can read their solution before I go into the interview and have some prepared comments or questions to get discussion going. I like that it emphasizes the collaborative nature of programming in a team.
Good candidates are beginning to revolt against traditional coding interviews. Take-home exercises are one of the alternatives that people actually seem to like on both sides of the interviewing equation. So you're not going to miss out on good people by switching your coding interview to a take-home; if anything, you're more likely to start attracting them.
I've interviewed with companies that gave all kinds of exercises. I have no problem with take-home exercises, if they're implemented like the article suggests. A few times I've gotten "do this exercise and we can maybe pay you a bit for your time", followed by them never paying and me never asking. If you decide to pay someone for exercising, tell them how much beforehand, and pay them as soon as they hand it in.
The best interviewing experiences have been with paid onsites for companies, even if I didn't end up getting hired in the end (I got to meet the team, work with them, they took me out to dinner, etc, it was nice).
When I was first looking for a software job I was a researcher who'd written a lot of his own tools but I didn't yet have an open source portfolio or commercial development experience. Solving a challenge was a great way to break the chicken-and-egg dilemma of needing experience to get hired and needing to be hired to get experience.
If I had to switch jobs again I'd also love to find employers screening with weekend challenges instead of live coding. Paying me to complete challenges would just be icing on the cake. I've done well at every company I've worked at, but asking me to write heapsort or whatever while you're peering over my shoulder and the clock is ticking is extremely stressful and unrepresentative.
I'm one of the people screening hires at my current job and I don't want to subject candidates to stressful live coding. But neither would I want to hire someone who can't write decent code and challenges look like a good way to prevent that. Surprisingly, even some people who include a github link in their CV are sharing code that's quite poor. (Like turning O(1) operations in an inner loop into O(N) because they picked the wrong collection type, in a language that offers perfectly suited collections in the standard library.) Maybe they think that nobody will actually read it, and having a github account at all will mark them as savvy?
However, before they had time to review it and get back to me, another company interviewed me and offered me the job and I took it.
From the other side of that, we now give these assignments when hiring. It has virtually eliminated the bad candidates, and we've hired a few good junior programmers for a change.
However, neither of those 2 situations paid anything. They were throw-away projects that the company couldn't actually use to avoid any legal issues. If we were going for experienced programmers, I think I'd recommend the payment route. But junior programmers? We got too many applicants to send it to all of them, and so we'd end up pre-filtering them like we used to do, which makes the process less valuable.
I might be more ok with it if it was meant to replace the in-person interview but otherwise it took up significantly more of my time. It was nice that they gave me 2-3 $100 amazon gift cards but, like other posters, I'd rather have my time than extra compensation.
The first assignment was supposed to take 4 hours but took me over 6. The second was supposed to be a whole day thing but I could only put in about 4 hours.
It was useful to gain some insight into their culture. I'm still mixed about what I think of it. The short takeway is that they are continually learning and updating their practices (good) but they were koolaide drinkers (bad) and throwing around buzzwords from specific individuals' blogs as if that was everything without showing a deep understanding of those buzzwords (causing them to completely miss that I exercised those same principles, even if I had no clue what they were talking about).
The longer version. They originally were going to turn me down for not using enough <buzzword>. They were also disappointed that I didn't use <buzzword>. Originally, they weren't even going to explain more than that but I pushed on it to see what I could do better and through the conversation they warmed back up to me. Later I looked up their buzzwords.
The first was SOLID which appears to have limited use in the industry. First, they couldn't even break up which part of SOLID I was violating but instead were throwing the acronym of acronyms around like that was all that needed to be said. When I dug into their feedback more, I found out it was my main function that "wasn't SOLID enough" which did argument parsing and delegated the rest of the logic to other code that was so well designed that they had to throw out the main follow up question.
I literally could only find one software development subcommunity that used the second buzzword and that was mostly limited to one consultant's blog. They also showed a lack of depth in it because I was actually using it, just at a different layer in the stack, solving the same problem they thought it would address.
f.e. I really like my current job, so I would benefit from getting a better look at your workflow and interact with the team to decide if I really want to work for you.
Hopefully the situation will improve in the near future, but I am so taken aback with common practice that I currently see no other way but to start my own company, I just cannot imagine working for such stubborn and close minded people. And for darn sure when I hire, this will be the process.
A bunch of place do this in some form or another, usually without paying the person, and as part of a series of interviews on an interview day.
I myself don't like homework questions, mostly because I still have to do an onsite, and I only have so much time to do these things. They are usually much longer than 2 hours too. If you communicate upfront that it should only take 2 hours and would remove the need for an all day onsite, then I would definitely be pretty excited. It would probably let you interview a segment of employed engineers who don't want to sacrifice a vacation day to do interviews.
I really like coding exercises as a general interview type although, I feel like it's the most important & real part of an interview.
The paid version doesn't work for people on a visa or are worried about IP assignment agreements, but I like the charity idea brought up by another person to solve this.
a) Creating a frontend form to submit movies b) Creating the database to store movies c) The 'search to find the movies' feature
Maybe I could do it in two hours, but it'd be pretty bad, and there's no point showcasing poor quality code.
My favourite was when they asked me to produce examples of things I had coded. Like any other creative industry, a portfolio is normal.
CoC (Convention Over Configuration) solves most of the problem for you, since it's basically CRUD, and that is boilerplate stuff in pretty much any language.
That having been said, most of my value is writing sane, readable, cohesive code that can easily be extended years later, with a minimum of refactoring. That's what has been most valuable to the businesses I've worked with, and employers love the fact that my team is the one with the lowest implementation times for new features.
I agree with your second paragraph completely.
This wouldn't cut it at the companies you mention, I agree. I think the main thing is that the actually valuable skills (being able to figure things out on your own, having an intuitive sense for when a solution is suboptimal even when you don't know the optimal solution, knowing design patterns, etc) are transferrable, so a good developer could work on either thing.
It probably wouldn't take two hours to become acclimated, but after a short ramp-up period, they'd be pretty good at it.
I have a long way to go it seems! It would definitely take me more than 2 hours to do that to a standard I'd be happy with. And there's no way I'd submit "what I had" in 2 hours if I could sneak off and spend more hours of the weekend doing it & return something polished.
Do you penalize people who give something "too good" or query them on whether the _really_ only spent 2 hours doing it?
Usually when someone makes something 'too good', it usually means they've leveraged libraries that they already know. If they make a lot of code, I know they probably spent more than 2 hours. By making it a weekend project, maybe they are subtly hinting that better candidates will spend more time on it, which I think is shitty and one of the reasons why I don't like homework questions.
I'm assuming for all of the things you've just said, it means you are using libraries that make a lot of those things one liners. Like read/write local storage, search a data struct, reactive UX changes, etc.
That sounds like something a bad tech hire would say.
Not as catchy as "fire fast", but infinitely more sustainable.
The first one was a pretty egregious experience, honestly. I never spoke with anyone other than a recruiter. The first was a 2 hour Java test. The next was a take-home that I was supposed to spend 5-7 hours on. I did it and sent it in. I knew it wasn't great, but I put a hard stop at 7 hours. It took almost a month for me to hear back, from the recruiter, "we've decided not to continue with your application at this time..."
For a while, I said never again. However, after refusing (and passing on what might have been good opportunities), I did another. Circumstances were very different. I had two extensive interviews with developers prior to the take-home, and I was interested in the job. Furthermore, they did reply with feedback, though it was through a recruiter.
The feedback was mixed, some things were good, others were bad. Unfortunately, though, it was a hard stop. I didn't get and presumably will never get a chance to defend my choices. Although I understand the company probably doesn't want to open up a debate, they've moved on, it rankles that someone got to say those things about my code and I never got a chance to reply.
So, am I at the point where I'd never do it again? Almost. I think I would insist on an opportunity to defend my code base, at least one back and forth, before calling it quits.
It sounds like this company (mattermark) does this, which is a positive thing. You do get a chance to argue (though maybe they do cut some people off when the project is terrible, and who knows, maybe the people who rejected mine (no connection to mattermark) were just taking it a little easy on me, but were genuinely not interested after seeing it).
I don't actually care about getting paid for the project time, if it's a good opportunity. I have to spend as much time re-studying my data structures and algorithms book for whiteboard interviews anyway, and I actually my learn something from the take-home.
I'll probably talk to a buddy I consider to be a really good programmer who works in this area, and see what his take is on my project, asking him to be pretty brutally honest about his assessment.
From what you typed, it seems like there wasn't a lot of communication prior to doing the coding exercise. I'm also not aware if you had channels to ask questions or discuss technical design options. If available, those are often "part of the test", to see if you make use of such resources.
When I'm reviewing code in such scenarios I'm often looking for other indicators aside from the code's correctness. Things like: naming conventions, structure of the code, patterns used, how over-engineered it is. Just as important is seeing if the candidate picked up on less explicit instructions or hints in the coding exercise. Finally, I look for code that follows the conventional patterns and standards for that programming language; I expect a candidate to discuss with me the deviation from those standards prior to handing in the code.
Anyways, I wouldn't completely give up. This might be an opportunity to grow and learn, but you'll never know unless you follow through on asking your buddy. Likewise, I've seen people here on HN volunteer to review code and provide feedback.
Your instincts are good here - one cited problem was indeed the departure from standard conventions. I had my reasons, not saying they were good ones, but I didn't get a chance to explain them either.
I'd like to, but it appears I won't get that chance. I gotta say, that is the part of this all I don't really like. It was vastly better than my only previous experience with a take-home project, where I didn't hear back for a month and got a one-line brush off from a recruiter. In this case, I am certain that they looked over the code reasonably carefully - they did provide a short bit of feedback, though it was delivered by a recruiter, and there was no back and forth here. I'm ok with not getting a job, but I would have liked an actual follow-up where we reviewed the code after that level of effort and time invested.
Lastly, the job was fairly senior, so it may have been a high bar, and that wasn't the only problem with the code. Then again, how much are you really supposed to work on a sample project?
Overall? I'd be hesitant to do this again. I might ask in advance what the policy is on a review. Maybe a 30 minute conversation is a requirement for me to do this. Still mulling it over.
Regarding the "algorithms trivia via whiteboard" style interviews: I think this type of interview is, unfortunately, the best large companies can do given their constraints. These constraints include hiring lots of new graduates, interviewing pipelines that include technically illiterate recruiters and engineers who lack the executive function to accurately assess candidates' viability. Lastly, the people involved in hiring at large companies are clueless about what the new hire will actually work on. It seems so obvious that tailoring the interviewing process to the actual fucking work that the person will be doing is a good approach. Once again, the contraints in a large company prevent this from happening in general.
Please, no. There are plenty of companies that respect experience and the wisdom that comes with age. It may be challenging to find them, but don't go management just because "you're getting too old." If you love programming, keep doing it, and find a place to work that suits you. Signed, a gray-haired 47 year old programmer.
However, if you're experienced and skilled there will always be work available. You'll probably do OK. Overcoming your social anxiety would probably open some doors, though, and not just in management positions.
1. A phone interview to validate the applicants resume.
2. An in-person interview where the applicant is given an hour to complete level 10 of Blockly Maze (https://blockly-games.appspot.com/maze?level=10) after being briefed on the wall follower rule (if they are unaware).
3. Completion of an outside example project in the applicants strongest language, for which they are not paid to avoid non-compete clauses.
This usually is enough to evaluate whether the applicant is embellishing on their resume, capable of analytical thought in a completely alien language, and their ability to deliver a deployable project given specific requirements.
I thought I'd try your step #2: I have no idea of the wall follower rule, and no prior experience in maze solving, but did bring it home in 18 minutes.
That included figuring out this particular Blockly UI (have admittedly seen similar UI with Dash and Dot, only at a much simpler level of doing non-branching, sample tutorials with my 6 year old boy), trying to design a simple program, reaching some dead ends / observing effects, and then realising what needed to be fixed.
I did use all the blocks for 17 lines of JS though, so perhaps better ways to do it, but thanks for that interesting little challenge at any rate. :-)
I'm guessing really smart people would have it done in just a few minutes.. Also monitoring son's cough (night time in Australia) to make sure it didn't get worse, for a bit of distraction.
(Edit: Brought it down to 1 spare block / 16 lines having a 2nd look after writing this comment.)
You might be interested in a solution that uses less blocks (i would be). I might have been lucky, took me 2-3 minutes to find this.
https://blockly-games.appspot.com/maze?level=10#taj4p4
Does anyone know of a better one?
Edit: just as i posted this i saw that you dont even need the first "Move Forward", so it is 10 lines total with 4 blocks left
EDIT: Nice. I was doing it slightly different to yours, https://blockly-games.appspot.com/maze?level=10#foif9x , two if statements instead would have fixed that, and the pattern becomes perfectly clear. Cheers. :)
But that's code bingo - fewer lines, but less efficient for the mouse.
My ex-wife is a concert-level pianist who can remember texts (words) after only hearing them once. She's also, astonishingly bad at spatial tasks (gender stereotypes aside).
So unless you're hiring for computer graphics, or visualization or some-such, which has a clear spatial component, you're implicitly biasing against less-spatial people.
When did we stop smart people who may not have fit precisely into the perfect little niche we've carved for them but showed a good ethic, and an ability to get the job done and learn new things as technologies evolve? We're really screwing up this industry in favor of getting it done fast, today.
Probation is also important; I've used it when I don't like companies and I think companies should be willing to say to people it's not going to work out.
Three months is enough to know; people can look amazing on month 1 and lose their way. It is also important to never underestimate a manager who actually thinks through what they want from their staff and asks them to do it :-)
For example, I set up the project for my team in 7 different languages (Ruby, C, Python, Perl, C++, Java, Go), with the appropriate project structure, etc, for each. This took considerable work on my part, but the end result is that I can judge the output based on whatever the candidate is most comfortable in.
Granted, that's all assuming my skills in each language doesn't completely suck (hopefully).
Not sure. The whole idea of the test is to get around people that talk well vs do well.
And assuming the friend told them about how/why they did something the interviewee can talk about this solution as if they did it. How would you know? See I'm assuming the 'friend' adds value to the solution to make it amazing, and the person who is interviewing still has average knowledge so its not like they are completely winging it.
> are you just optimising for a problem that could potentially happen?
Absolutely. You should optimise for problems that could happen if they have reasonable likelihood and will become significant issues if they do happen. Not going to rule out all issues. Thought this seems a likely risk over a reasonable number of candidates.
I mean, I like the idea of proving what I can do, but software development - as in most jobs - are far more than technical ability.
You want me to prove my worth? Fine, you can pay me for it and get something useful out of it at the end, as a senior developer I really shouldn't have to "Write a palindrome detector" to prove that I can code, I have a degree, we did that shit repeatedly!
Nice to see other people are coming around to this way of thinking
I like this approach of merging consulting and interviewing because it gets interviewer and interviewee closer to the source of truth about how a real work relationship will exist.
We work remotely, so is our hiring process. While it would be possible to give a programming task to a candidate, we won't have any control over whether they did it themselves, or how much time they actually spend on it. Well, the former is less of a problem, once you discuss the solution you see pretty quickly whether they did it themselves and understand how their solution works. However, candidates that want the job will try hard to do as best as possible, so some candidates that you give a 2h task to, tend to spend much more time on it. Others might not have as much time to spend on the task, since they usually still have a full-time job. So the results are not comparable.
Besides that, I find it quite difficult to find a task that can be done in 2h, but gives a sufficient insight of their skills, however you cannot expect somebody to take on a more elaborate task, whether paid or not, because most candidates are in a full-time job when they apply.
Also, this might be different in the US, but here in Germany, if the candidate isn't a freelancer/contractor at the moment, but permanently hired by an other company (this is normal), there is no way to pay them officially.
The solution to this I feel is a bit of a prisoner dilemma because the entity that can make this easier is the company the candidate is a planning to leave (!). If they have flexible working, make it easy to take vacation on short notice etc. then that ironically helps.
Having lived for the last several years in a very rural area of the midwest I would say that not only do you need the right mechanisms in place to sort through / vet candidates, but you also need to be willing to constantly venture outside your comfort zone. Why are you always recruiting locally? Why aren't you willing to look in all locations throughout the country? To be frank, I would guess that almost never do you find a "diamond in the rough" outside of large metropolitan areas, but my personal experience is that as soon as I work with a team in a metropolitan area I realize I'm almost always substantially ahead of the game in terms of best practices / capacity, etc. I would probably argue that I'm an exception to the rule, but (not to be political) the recent US election leads me to believe that the "norm" is not the "norm" any more. People are thinking outside the box and we (as recruiters / employers) need to realize this and not instantly discount a person just because of where they live (btw I'm pretty strongly Dem living in a mostly Rep world).
I don't need or expect payment, either. If I'm legitimately interested in the company, I am happy to spend a couple of hours showing them my stuff. Hell I'm proud to do it. I do enjoy programming, after all.
Closet thing I've been offered ironically was rather tough sys administration tasks for no pay for a few gigs (5 hours of work) or asked to solve an obscure ntp issue on a Linux server they couldn't figure out.
For programming jobs? Never.
This is wrong in certain cases. The candidate may have significant contribution to an open source project or may have shipped an open source project as an owner.
Why should that candidate not refuse? You already have the code in front of you. If you as an employer cannot take the time out to go through that and evaluate what you would otherwise with your take home assignment, then something is not right.
This one size fits all rigid process won't work. Given a process, the employers must learn to be flexible, all the while expecting employees to do so, is insane. So certain qualified candidates saying 'someone else will hire me' is not arrogance, but fact.
Now employers don't want to do that most likely I assume is because no one wants to go through the pain of figuring out the details of GH code. OTOH, the employer is expecting the candidate to understand the problem assigned to him and come up with a solution!
It was the best interview I've ever done. I rode a huge wave of emotions the entire time and learned A TON.
Technical minutiae interviews like The one I did at Google really suck
1. What is a reasonably sized project? I.e. often times two hours is not enough to produce something of substance/complexity, yet asking the applicant to do more becomes prohibitive. 2. I see a bunch of people asking candidates to do greenfield applications, while virtually all work most future workers will do is refactor and maintain. The latter however requires additional time investment to get to know an existing system/codebase.
How are you guys handling those issues, and trade-off time complexity against insight value?
And you need to produce such problems relatively often, right?
The take home version might be useful beyond that, but I've found that solving some fun toy problem like, "let's build a little app for X" in a couple hours you absolutely know if the candidate can code and have a good feel for what it would be like to work with them.
If for some reason they don't have a portable dev environment I guess we could provide it but that does hamper the process somewhat because it's not really "in their element" anymore.
I don't know how you really prepare for such an interview, other than actually having the portable dev environment and being well rested.
The only thing we use the whiteboard for is understanding the problem statement and sketching out the steps to creating the solution, as in, things a whiteboard is actually meant to be used for.
Not to say you shouldn't do this but it's very important to remember a lot of these kind of things when it comes to recruiting involve tradeoffs and you rarely get an unalloyed good.
I suppose it's nice that this is paid. A lot of companies give me HW problems... and frankly it doesn't scale.
https://danielmiessler.com/blog/why-you-should-hire-using-ev...
I, like others stated above, tend to perform much better in my own space and time than under eyes on a whiteboard..
Getting paid could provide the ideal constraints for the problem at hand.
Exactly like Google, who tell applicants that their solution is wrong when it isn't. Most people complain about how bad their application process is and that interviewers don't know anything. Nope, they just weed out smart asses like you that are a pain to work with.
Sometimes a candidate is a good coder but a horrible communicator or human :|
This is the end of the process final wash
And so you probably won't do it with those candidates that fail your normal process. If you are biased against old, black, women you won't give them this test
It reduces your false positive rate. But won't ever help your false negative
> Since the candidate is getting paid, the candidate treats it as a legit consulting session and therefore gives their best effort in solving the problem.
If a candidate has applied for a job then it is in his interest to treat that application as his career depends on it. Best effort is wagered not on the fact that he is paid but on the premise that his life will become better by getting this job.
> It gives the candidate a real life sense of how you interact with people on the team.
Where is this coming from ? This is just another assumption the author is making.
> The biggest issue I have seen with tech hires is that they can become very defensive over their solution. I will purposely challenge their solution to see how they react.
You are asking us to be non-judgemental in the paragraphs above and then being judgemental on the candidate becoming defensive.
> Pay them immediately on Monday
Why ?
> Where is this coming from ? This is just another assumption the author is making.
It’s coming from the fact that you’re giving someone a project and discussing the result with them.
> > The biggest issue I have seen with tech hires is that they can become very defensive over their solution. I will purposely challenge their solution to see how they react.
> You are asking us to be non-judgemental in the paragraphs above and then being judgemental on the candidate becoming defensive.
That’s not being “judgemental”. It’s just using regular judgement – evaluating a person’s willingness to listen to criticism. The earlier points were about hearing rationale for choices without previous criticism.
> > Pay them immediately on Monday
> Why ?
It shows that you’re well-organized and keep your word.
I think most of your arguments were coloured by the mention of a paid product.
But even if it wasn't, a company of any significant size (enough to pay even 1 person's salary, say) should have a low effort and cheap way of tracking contractor payments and issuing 1099s.
Which I mean sure has nothing to do with why this idea is being marketed so hard.
If a company were to give me a take home assignment I'd see it as them not valuing my time.
When you do a traditional interview the company has skin in the game. They have to "spend" the same amount of development time as I do in the interview. That person interviewing me could be working, if they're interviewing me instead, that's the company signaling that they think I'm worth the time. You don't have that in a take-home problem. It's asymmetrical. I'm investing time on this and they aren't. That's not how I want to start a business relationship.
Then there's the marketplace. I have a lot of options for employment. I'm confident that if I quit my job tomorrow I could have another one, just as good, by Thanksgiving. The job interview goes both ways. I'm interviewing you as well. I want to talk to your current developers. I want to get a sense of your office culture. And I don't want to complete a take-home assignment before I get to do that.
Finally the people I like to work with are like me. They're busy, driven, and good at what they do. I can't imagine they would work for a place that used take-home work for interviews. So even if I did go through with it and get the job; what type of people am I going to be working with?
It will be good for candidates, good for employers and good for the economy.
Then again, everyone is different.
To implement any new method , system infrastructure is very important. According to me paid interview is not that much beneficial for a mid-level company as it will be for some MNC or IT-giants because the number of openings matters. The article suggests to bring a group together on Monday to discuss the solution to a particular problem given. In that case the number of opening & scheduling has to be huge in order to arrange such group for the G.D which is not possible in a mid level company where you have only 1-2 openings at a time and the number of candidates who apply to take the interview is also very limited i.e within 5-8 in a week out of which only 2-3 appear for the interview in a week.
According to the Indian mentality Paying against just appearing for the interview might not be that fruitful in absorbing the right talents in the company . If such concept of interview is out in the market people will take this as a scope of earning rather than joining. We might not get genuine candidates who actually wants to join the company. Candidates will just appear only to earn that money and will leave. Here what we can do is we can only provide the money to the candidates who gets selected and then joins.
Here we are allowing them to write the problem down and take them home. Here comes the question that what if the candidate is taking someone else's help to solve the problem and how much honesty he is maintaining to solve the problem. If the candidate is appearing for the interview in the office premise such factors won't be an issue as he will be monitored constantly.
If we are following such process of interview then we constantly need a team who needs to change the question for every session of interview as the question is going out therefore we will always be in requirement of new set of problems/question which is again tough for a mid-level company to maintain due to resource crisis and that to only for 2-3 openings at a time.
We are providing the candidates constant supervision and guidance to address the question, also we are providing them no time limitations and if they clear the test a group is there to judge his perception and the result instantly on a one to one basis which is less time taking and prompt as compared to the process suggested in the article.
Pay people to complete a real life project and then bring them in to talk about it.
Why is this so hard for most tech companies to understand?
I would not rely solely on this as no process is perfect. However, I do weigh this part of the process the highest for my hiring.
The student that wrote the code can always explain not only how it works but why they made certain decisions along the way.
The student clueless enough to copy someone else's code usually can't even explain its function, much less explain tradeoffs.
Giving someone a toy project unrelated to the business tells you nothing about how they will perform in the real environment. Letting the candidate take the work home to work on it in isolation tells you nothing about how they will work with your team, or within the company.
In his study "People and methodologies in software development" Alistair Cockburn found that “People’s characteristics, which vary from person to person and even from moment to moment, form a first-order driver of the team’s behavior and results." (http://alistair.cockburn.us/People+and+methodologies+in+soft...). Other studies have come to similar conclusions.
In a technical job, actual skills and experience are necessary but not sufficient for success. Programmers fail in their jobs because they can't work with or fit in with the team, and the larger organization. Any competent programmer can learn a new language, framework, or technique like TDD in a week or two. But people who don't pay attention to the business environment and requirements, and can't work with their team, group, etc. will fail, often costing a lot of money.
Presumably you have weeded out the unqualified before offering to pay them to work for a few hours, but this seems like offering an unnecessary incentive to learn very little about the candidate's likely on-the-job performance.
I don't know any scientific way to measure how someone will fit in, and I don't think anyone else does either. That's why having several people from different parts of the organization interview is the norm -- the interviewers compare notes and reach a consensus because everyone understands they are going mostly by their subconscious (gut) feelings.
I spend time describing the business requirements and then ask candidates how they might solve an actual business problem. I don't expect a complete or deep answer; what I'm after is finding out if the candidate can think about a business problem at all. Examples: Our inventory system frequently doesn't match what's in the warehouse. We get a lot of people abandoning their cart at checkout. We have customers gaming our promotion codes.
Testing for things like "functional reactive principles" is an example of ignoring the business environment and favoring technical trivia. No business ever had the problem "We need 2,000 lines of Ruby code by next week," or "We need a system to organize a small movie library, you have two hours." What I find is that far too many programmers don't even hear business requirements and can't do basic analysis or think of the right questions to ask. They want to start writing code. Having expensive people writing code that doesn't address business requirements is costly, and very common. If a candidate isn't asking questions about the business and the team that's a big red flag for me. Programmers who just want to be left alone to do their own thing are easy to find but they aren't necessarily a good fit, or productive.
I also spend time getting as many people in the organization as I can to chat with the candidate, so we can compare our impressions. That is just recognizing human nature -- we all tend to form opinions of other peole within a few seconds, and then look for things that reinforce our impression. Of course I want to see evidence of actual technical skill, but I can do that with a few actual programming problems -- even something on the level of FizzBuzz will tell you if the person can solve a problem in code. And I want to see how they solve the problem, and what kinds of questions they ask.
If someone is uncomfortable working alongside another programmer, or solving problems on a whiteboard, or talking to a group, I get it, but those are all necessary skills too. Just like learning a new framework, a motivated person can learn to work in a group, listen and ask questions, speak to a room. They may be a genius when left alone in a corner not interacting with anyone but they probably aren't going to be a good fit in most organizations. I often ask candidates to tell me how they would go about hiring someone for the same job they are interviewing for -- suppose we have two positions open and you get to hire a peer. That can tell me a lot about what they value in a co-worker.
One guy came to us from a recruiter highly-recommended. He had a few years experience, but he was widely respected online via his community contributions. He was also out of a job due to redundancies. It was enough to get him an interview, and he could talk well, so my bosses got the dev manager to set up some basic problems on a client website to solve.
He seemed like a nice guy, and he had a lot of questions, which was a good thing. He had a nice chat with some guys over lunch, telling them that he'd been unemployed for several months, and he was down to his last savings. He was able to fix the problem purely through looking on some forums for the answer, and that answer helped him fix the problem. The dev manager was happy, and the PM's were happy. The devs that worked with him were brought into a room to see his solution, and everyone in there said he should be hired. We had a new dev!
He was instantly given a project from the backlog, which was billed out, and would take around six months. He clearly knew what he was doing, so he was left to his own devices. To cut a long story short, the code he was writing was absolutely shocking, and he'd engineered a solution you'd need a PhD to understand. He was also finishing tasks really quickly, but they were bounced right back to him because his code was full of bugs. He stayed with the company for a while longer, but was ultimately let go. That project went over budget by six figures. It turned out that he was let go from his last two places for similar reasons. He was a junior developer that believed himself to be senior, and as such he disregarded any advice from anyone but managers. He had claimed open source contributions, but he was widely considered because he'd made thousands of posts on an open-source project forum, and due to their community contribution rules these posts deemed him a candidate for MVP. Despite thousands of posts, he'd never really solved a problem for anyone.
I looked through the code he had written during his trial, and it was largely the same mess, but obviously smaller and self-contained. It wasn't an optimal solution, and he'd written 500 lines of code for what should be a 20 liner. The dev manager used to be a programmer, but hadn't written code in a while, and didn't see a huge problem with it. When the rest of the devs were asked about it, they all said that they thought his code was messy/crazy, but the managers wanted him, and the guy really needed a job, so they said yes.
I'm not saying that this solution isn't a good one. If you can do it, I'd argue that it's the best way for a typical company to hire a developer. What I would add is the following:
1. Management should not publicly offer their opinion on a candidate before technical advice is given. If the managers weren't praising him, our devs might have had the confidence to say no.
2. Any technical test should be fully QA'd before a "Yes/No" is given. You're not just looking for someone that can write code. You're looking for someone with domain knowledge of your tool-set that can offer solutions to problems. If you're shown a bunch of code with little context. Funny enough, the code that he submitted was actually fixed by another dev on the following Monday.
3. Make sure they're not working on something that actually needs to be delivered in the next week. This guy was brought into the office on a weekday because he was unemployed, but it turns out most of the pressure for this was from a PM who said that the problem he needed to work on was actually required in that sprint.
@throwaway4job, you appear to be hellbanned for no reason I can discern. :(
> throwaway4job 8 minutes ago [dead] [-]
> This isn't a viable approach for much of the hiring pool, as a majority of working engineers have contracts which explicitly forbid work for contract for other employers. As such, if you hire any of those, you're hiring somebody with a proven willingness to ignore their contract, which is a strong anti-pattern.
That said, that was a completely defensible and evidence-based thing to say in context (the founder threw a fit about regulators who wanted to know what the chances are that people would die), so unless HN is cracking down on the casual use of "sociopath" by people who aren't making professional medical diagnoses (which, to be fair, I do find objectionable, but not to the point of hellbanning), it seems like it was for no sensible reason.
When a banned account's comment gets vouched for and rescued, we look at the account history and unban it if we see that we banned it unnecessarily (we try not to do that but it happens) or if the commenter has reformed. Alternatively, there may be a good reason for the account to stay banned, but if it posts good comments, HN users will vouch for and unkill those. This system has proven to work well, even better than we hoped when we introduced it.
Please don't do this. We created vouching for just this case and it works much better. To vouch for a [dead] comment (assuming your account has > 30 karma), click on its timestamp to go to its page, then click 'vouch' at the top.
Besides being distractingly off-topic, making a big deal out of [dead]ness confuses the thread when the comment does get vouched for and rescued, as typically happens for good but [dead] comments. For example, in this case the replies to throwaway4job's comment are now split across multiple places in the thread. We'll move them.
We detached this subthread from https://news.ycombinator.com/item?id=12916816 and marked it off-topic.
So given that Amir of June is the only person doing this quirky interview style (where you actually pay them for their time), the interviewer still has absolutely no idea of this psychological gimmick Amir has baked into his process. Right at the very end!
Amir Yasin admits that he is elevating them into a contracting gig status, and contractors ARE THE EXPERTS! They have to be the expert! They have to make up stuff on the fly to defend their implementation, "its a feature".
But for Amir's warped interview process, of which the candidate is being bussed around on a literal auction block at other companies before having the luxury of getting his groundbreaking interview style, he still has this one random assessment.
Look, Amir is the one that elevated his tiny sample size of engineers into gospel. There likely was a literal 2 people that formed his and his company's entire opinion about Rule #4.
So I think it is worthwhile to point that out.
What is your favorite 4 letter word.
I usually ask at the beginning of interview and expect an answer at the end.
Answers like fuck and shit === no hire.
The other ones tell you a lot about the person
For example, if I reply "hire", would that impact my application positively or negatively?
If you get all pissed off that I asked you such a question, perhaps I should not hire you since you will get pissed off at silly things that happen at work.
but you cant say that because idiot hiring managers who ask idiot questions would insist that you do.
sorry, but if you ask shit questions, dont expect non-shit candidates.