How Firebase Interviewed Software Engineers
startupandrew.com
startupandrew.com
This is the part that didn't ring true, at least for me. I believe that there are some candidates would not complain. But if I am good engineer, odds are that I already have a job that works well for me. Maybe not the perfect job, but working well enough that you'll have to give more incentive than just a job being available before I'd spend a day on a take-home, unpaid project. So according to the quote above, I'm not the strongest candidate because I'm not willing to devote a fair amount of time to landing the right job.
Maybe from the hiring manager's point of view, that is even true. But I've already landed the right job. I work there, every day. If you want to poach folk like me from a job that is already right, 6 hour take-home tests aren't the answer. Give us something quicker. Maybe not easier... I believe a challenging interview process is fair. But quicker.
What’s so compelling about your company that people are willing to put in 5x the time commitment to apply as for most other companies? I can literally do 4 rounds of “phone screen -> on-site” with other companies in that time. If I’m working, why would I want to blow 1/3 of my yearly vacation to go through one interview process?
Having had interviews at companies where it was NOT as advertised during the interview phase, I'd value that.
If I was already interviewing for another job outside of my existing one, I'd gladly take a week off to do it.
But if they were trying to get me to leave a job I already liked, I wouldn't commit that much time for it.
Simplisticly, it might boil down to something like a preference for quality* interviewing over quantity.
* not saying our process is definitively better than others, just that some candidates want fewer interviews that they feel are a better potential fit for them.
Paying for the interview is compelling. Almost nobody does that, so it's a differentiator. My first assumption on hearing that is "it sounds like they respect their engineers as professionals more than most companies do".
They don't hire people who have jobs because no one would/could do this.
Most devs with existing jobs have good jobs with decent vacation/PTO policies. So they have days to invest if they want to.
Don't be so quick to assume the worst. :)
But you do use the work to build your backend/office applications. In the end you are paying someone $250 for a 15 hour project + two days of in person development time.
Great idea. Many with follow.
"The costs of travel and lodging are compensated at this stage, but we are not paying them hourly. The work they do for us is not customer based work and does not generate revenue for us."
I don't like complaining without a suggestion, so... consider letting a candidate code on their own, and then ask them to walk you through what they did. You won't be disturbing the process by staring at them, and you'll get to see a bit about how they work independently & articulate what they've done. Unless you're a pairing shop, that's really what you're looking for anyway, yes?
The right hiring process for one position may be completely wrong for another. And that’s OK.
I don't think that necessarily applies to real-time pairing exercises though. The interviewer has significant latitude to interactively question and prompt to get at the skills and values a candidate possesses.
I showed him, we talked about it, he asked me to add a simple feature, I think I taught him something about the tech stack that he didn't know. It worked well for both of us.
If you're already happy with your current job, and we can't convince you that our company is potentially a much better opportunity for you, then this is working as intended.
Interviews are expensive for the company too, so it only makes sense to invest time in someone who really is excited about switching.
The unasked question is where you believe the engineers are who can make your project a success? Are they out looking for work? Or are they happily ensconced in other work?
There was a prevailing belief, at least a number of years back, that the best engineers were not found on the open market, so you had to actively seek them out. Maybe that isn't the way hiring managers approach the field anymore, though.
My blog post goes through the process for inbound candidates, which accounted for the majority of our hiring. We did actively try to poach folks as well. I spent literally 4 years trying to recruit one of my friends out of Microsoft.
For these folks, you need to sell, sell, sell your vision, product and team. They're not going to switch unless they're really pumped. All of this work to get them excited happens before the interview process even begins.
I think for these people the difficulty of the interview process is a small issue relative to the risk of switching away from a job they already like.
I mean in the article you say "The strongest candidates take their job search seriously and are willing to devote a fair amount of time to landing the right job." the times when this has been true of me has been times when I do not have a job. If I am being headhunted for a job and I am already in a good job, I have a hard time envisioning a scenario where the job being offered would be so enticing that I would be happy to make a significant investment of my time although I might do it grudgingly if the offer is good enough.
When I took the job at a small agency I'm in now I had stopped a consultant task that had run out of funding. I was in a hiring process with four other companies, the small company in the interview asked me how much I wanted I told them my price they said "that's a lot but we expected it would be something like that so it's ok" and then they said something that really cemented my interest in working there than any of the other places that I was interviewing with - they said "we generally have a coding task to do but in your case I think we can skip that".
thank you for saving 4-6 hours out of my week! My family thanks you too!
In fact now that I think of it this testing is another reason why I give people a significant raise as a requirement for moving from my current job. I am earning pretty high up for my market, the market is like many tech places a sellers market. When recruiters ask me to interview with the companies they're selling I name a 18% increase of my current wage and equivalent perks as a requirement to consider moving. Maybe I would only want 10% if I knew that trying for the job wouldn't require me doing a bunch of unpaid work.
Is this necessarily the case? The Clippers had to bend over backwards to get Kawaii Leonard on their team. They almost certainly would never have gotten an NBA Finals MVP if their philosophy was only go after people who were already excited.
The point is there some tradeoff in the design space. It's nice to have people who are excited to work for you, but chances are the superstars that will take your team to the next level, already have a dozen suitors. Kawaii Leonard isn't doing a six hour take home tests.
I'd suggest that maybe Firebase's experience here is not representative. It was one of the hottest startups in the Valley at the time. So maybe it could put up a lot of onerous requirements and still have talented engineers beating down the doors. If you're the Lakers, Lebron James comes to you. But for the 99% of hiring managers that are at the equivalent of the Clippers, maybe the same approach doesn't work.
I'm not against projects in concept and I've never complained about them to my point of contact, but they make the process slower (your process takes about twice as much of my time as everyone else's) and less exciting. Every other step of the recruiting process is really fun - talking to people about their company and the work they do, but a project is just homework that I have to do alone after work or on the weekend. I've really liked some companies that gave me a project, but I struggled to find time and then I got to the offer stage much faster with other companies that I also liked, at which point its hard to find the motivation to complete the project knowing that it isn't even the final step.
While you say you never lost a candidate due to the project, I would challenge you to consider whether you lost candidates due to the perception (fair or not) that your process was less enjoyable and more time-consuming than your competitors' and so candidates received and accepted other offers first.
Wanting me to jump through a bunch of performative hoops makes me assume you are in the "call you on the day of your daughters birth" camp.
Speaking from the candidate's perspective, as someone with performance anxiety, I abhor live coding sessions. Best case, I'm thinking at an impaired level due to the anxiety. Worst case, I can't think at all and freeze up completely.
Therefore, I absolutely love it when companies offer a take home project. And you're right: I'm probably not going to do it unless I'm really interested in the company.
I'm actually going to be doing one this weekend, and I don't mind spending 4-6 hours on it, because at this point in the interview process, I'm really excited about potentially working for this company.
I brought in my half finished side project. We talked about it, he asked me some questions, I showed him a couple of things about the tech stack that he didn't know. He asked me to add a basic feature. It worked well for both of us. If I had been faking it I think it would have shown.
Do also spare a thought for the people who don't have side projects, or any code that they can legitimately share. There are plenty of talented engineers in that position, and it'd be a shame to have a hiring process that leaves them out in the cold.
"We don't want the best if they are happy in their current job".
Because you're NOT affording THEM the courtesy. You're doing it for US as a community of software engineers.
I read a lot of complaints on here about companies do x, y, or z in their hiring processes and we don't like it.
Fine: tell them.
You know why? Because companies often listen to this feedback. Sure, some don't, but plenty do, and we certainly do.
If you are hearing the same thing over and over again from great candidates about why they don't want to go through your hiring process[1], and you're not able to hire engineers in the calibre and numbers you need, you'd be pretty obtuse not to do something about it. Even if you are able to hire the people you need, you might choose to make changes so that even unsuccessful candidates have a good experience and are more likely to speak well of you, as well as to increase your pool of better candidates. Sometimes those changes will take an awfully long time to occur, but companies do listen, and they do make changes.
(There are of course some companies that have so many people who want to work for them that they can afford to have what many of us would perceive to be an unduly arduous selection process and they're not going to change however many people complain about it, because they have no need. They have a different problem that they're optimising for.)
You might complain that this is not your responsibility, and that's fine: it's certainly your choice. But, if you want to see a net improvement in hiring practices across industries that hire engineers, you may want to take some very small portion of the responsibility that we can all choose to take for making that happen.
Let me be clear: it is much more effective for you to give companies with poor practices feedback directly (and politely, even if through gritted teeth!) than to complain on a substantially anonymous internet forum.
With all of the above said I'm now going to move into perhaps more controversial territory.
Not everything is about how good an engineer you are. In fact, your technical skills, whilst absolutely essential for the job, are not the most important factor.
We care about character. We care about how well you're going to work with others. We care about attitude. We care about maturity. We care about entitlement (or rather we have a strong preference for the lack thereof). We care about listening skills. We care about humility. We want people with initiative and skill, but we don't want divas.
Sadly this sometimes feels like we're spending more effort on weeding out negative traits than selecting for positive ones. Still, as I've said these character attributes are quite a bit more important than candidates' technical skill.
Fundamentally, we want to work with people who are easy to work with because in a team context you want everyone to rub along well together. We want people to treat eachother courteously. We want to avoid unnecessary drama.
[1] I've chosen that phrase carefully: I'm deliberately not discussing people who don't want to work for you, because there can be many reasons for that that have nothing to do with hiring process, culture, salary, or whether you're actually a good place to work (location, product, sector, etc.).
[2] Note: character, not personality. I don't mind what your personality type is (extroverted, introverted, whatever). We can work with anyone, as long as they're not an asshole.
And also I've worked at a company that was linked to a callcenter, I can say with 100% certainty, how someone behaves during an interview is never an accurate portrait of how they behave on the job
A lot of people get angry if you tell them the truth. If you try to soften the blow then they misunderstand and clearing up the misunderstanding then leads to them getting angry.
Weirdest interview experience ever.
I once had a founder offer to intro me to other companies if I wasn’t interested in his, which I thought was a pretty brilliant move that ended up saving us both time.
Prior to having a family, I'd spend 12 hours on a test but claim it was only 6, now I'd be pushed to spend 3.
People who give their firm everything they have generally get rewarded for it. Those that don’t in some industries (like finance) are quickly fired. In others, it just means you get passed up for promotion and make less.
Just realize that everything has an opportunity cost and that nothing comes for free (both spending time with your family and spending time at work). Also that people have different priorities so don’t make blanket statements about what people want.
The first job I got out of college, there was a sea of Solution Architects. They all worked around 60 hours a week, if not more. If a boss said jump, they would always ask how high. If a boss told them they needed to take a red-eye tonight, they always did it. Yet every time I went into the main office, the group was entirely different.
That's an example of a group of people who drank the Kool aid, who were "generally" supposed to be rewarded.
> Just realize that everything has an opportunity cost and that nothing comes for free (both spending time with your family and spending time at work).
It's a sign of a deeply sick culture when everything has a transactional value.
I'm aware, I have a degree in economics. I'm talking about cultural values that force constant optimization onto individuals as being deeply sick.
Regardless of if every action has some minmax strategy, it's not healthy to have individuals constantly running Bayesian analysis into if they should spend time with their family, or take a second job.
1 day: 100% 2 days: 50% 3 days: 33% 4 days: 25% 5 days: 20% 6 days: 16% 7 days: 0%
As you can see, the less everyone else works, the more value your extra work is. I’m not saying that the equilibrium is gonna be 7 days. But it’s not gonna be less than 4, at the very least. This is also the reason why the 20 hour work week will never catch on.
Wow TIL.
> As you can see, the less everyone else works, the more value your extra work is.
It's surprising someone like yourself subscribes to the Labor Theory of Value.
As such, we can view working more than anyone else in supply and demand terms. People who produce the most are always in high demand, and the less of them there are, the more value a high productivity individual provides.
How can you fit in _another_ project (albeit small) in a week’s time (heck, make it two) _and_ manage any two of the above at the same time, when setting aside an hour (or more, if it also entails on-site) is already a very significant investment on your part?
you guys are forgetting that this is a partnership, you also have to sell me your company.
if you're ok with that, I think you're missing out some good candidates.
That being said I also haven't needed a job for a very long time so...
What kind of gets me thinking is that, in my experience, people will take a look at multiple options before committing to a career change. This is where this concept seems to fray a bit.
Still I think it is a valid concept to minimize your exposure to false hires so it’s ok.
At my shop we opted for a similar approach but exchanged the project with a pair programing task that candidates get up front (and yes, we created reference code bases in many, many languages).
One other departure is that we don’t subscribe to the hard comparisons mantra. Candidates get to code with someone from their future team(or two sometimes) and it’s just a thumbs up or thumbs down signal. Thumbs down would usually mean that the process also gets cut short.
Sorry, you are describing a candidate who's working life is hell and sounds to me they would be willing to do whatever it takes to get out and get on.
Now, consider if they instead said, we don't need you to come onsite for 4 hours at all, just take this 6 hour at home test at your own leisure at your own time, and we will give you an offer based on it. That would speak volumes about the type of company and culture that they are.
I volunteer teach, I am president of my condo board, help run a local meetup, have a demanding job, oh and a family that needs love too- my free time is precious.
I started with their personality test (Calipers). It actually took 3 hours over 45 minute sessions that took the better part of a week. Fitting that in was extremely stressful and took away pretty much all of my personal time. When it came down to blocking out 3 hours for their coding test, I just could not be bothered- this would require me to more or less have to lock myself in a room for a good part of a saturday or sunday, toiling away still to only have a chance for a new job that might work out better than my current (I quite like my current job, this was just a potentially bigger role), but maybe not.
And I wasn't even in the market- Imagine now I am doing this for 5-10 (if not more) different companies. Its completely unreasonable IMHO.
Finally, we would all vote again, and this time, a “hire” decision needed to be unanimous to move forward.
Maybe they better start with the social part first then, not wasting your time for in case in the end someone happens to not like your face.
I've gone through rounds of interviews and take-home assignments and calls just to be given a very low offer. Start with that, save time for us both!
I think this is decently respectful of the candidate's time, while giving us the opportunity to review some actual code written by them. And yet, most of them don't even bother... Well, I get that you are busy, but I don't feel comfortable investing time in an on-site interview without seeing any code first.
Disclaimer: If they provide a link to their github with some solid projects or OSS contributions, we skip the take-home altogether.
But, I would only do this for companies I am desperate to get into (Ironically FAANGs for 'now' as they are the ones who are least likely to pull a WeWork/Uber/Theranos on me) and if I was pitched this by someone puritanically Id show them a finger. I do take my job search seriously and any hint of a company devaluing my personal time is a huge affront to me (and my family).
Edit: I wonder how much was luck. I definitely aimed for Firebase. I used it once after seeing on HN and fell in love with it immediately. It felt like the future, so it wasn't dumb luck, but then, it worked out so very well, I wonder if I saw a similar opportunity I would be able to leap at it again. Would it work out again? I guess time will tell.
The assignment was to read a file containing a list of numbers (some formatted incorrectly, so there was some very simple parsing logic involved), call an API using each correctly formatted number as a parameter, and store what the API returned to a file. I am to this day stunned that 60% of people who passed a phone screen could not solve this task. Note that we gave them the input file, so it wasn't a matter of an edge case tripping them up or them getting one input file but the test input file having some other edge case.
My point here is that it may be possible to get the same screening value with much less investment from the candidate.
Applications: 5000 candidates; ~20% pass rate (vs unknown)
Phone Screen: 1000 candidates; ~50% pass rate (vs 40%)
Technical Test: 500 candidates; ~40% pass rate (vs 25%)
On-site interview & reference checks: 200 candidates; ~50% pass rate (vs 40%)
Offer: 100 candidates; ~80% hire rate (vs 60%)
Hired: 80
So by some arguments you could say Firebase was 2.5x as selective (40 offers vs 100). With a funnel like this, even small changes to the percentages end up having a larger overall effect.
Unfortunately, we don't have the Applications number from the blog post, though he says "we considered a great deal more applicants than that [1000] on paper." I suppose a "great deal" could be anywhere from double to 10x...
That makes sense to me, because a smaller company needs to filter out as many people who couldn't possibly get hired at earlier stages, since the later stages are even more time intensive than code reviewing the technical test.
It consists of a small chunk of code in Java/C#/Go or otherwise that has some obvious and other not so obvious mistakes. The candidate is asked to point out any issues they see in the code.
It takes them 15 minutes to do and about 15 minutes to review the response, which I feel values time on both sides.
It feels like it tests something different than writing code, though both may be a proxy for "quality candidate".
I am sure half of these things are never thought through. In Python setting up a new project and downloading dependencies may involve needing to install a load of other crap and often takes more than two hours. Some libraries are incompatible with others.
If you are making assumptions that the test will take two hours, make sure that it involves minimal dependencies on third party stuff.
I'm sorry you've been burned, but that doesn't mean there aren't tests that actually take < 2 hours. I can't speak to every language, but what modern toolset can't open an input file, make an http(s) call, and write to a file?
I also don't understand why we shouldn't figure out how long something takes before administering it. Several people took the test and the time ranged from 15 minutes to an hour and a half, depending on language and experience level. I will say that if someone couldn't do it in 2 hours, they wouldn't have been a good fit for the team. If several team members took it, of course we're going to make an assumption about how long it takes.
Furthermore, since we didn't prescribe a specific language, there's no reason why someone wouldn't have all of the tools pre-installed. Even so, if you had to install your favorite development environment, you'd have been fine. That also wouldn't have been part of the two hour time frame (which wasn't a limit, BTW, just how long it ended up taking competent developers).
> We treated this code review like a real production code review, and we did it whether or not we planned to move forward with the candidate.
First, thank you for actually investing time in giving feedback to candidates regardless of their performance. Second, did you tell candidates that they would get detailed feedback or a code review as part of the instructions or phone screen? If yes, I figure that this would greatly affect how willing people were to doing the assignment.
I've always hated the thought of take home assignments because I have never been given detailed feedback and only rarely been given any feedback. Requests for feedback are always ignored regardless of performance. I sense that part of why people have greatly soured on take home assignments is because of how lopsided the time investment has become.
I have a list of checks to run through, some general (Sass variables used, reusable code/components used if possible, variables have meaningful names), others more specific to our codebase conventions. I add to the list as new conventions get prioritized and check it. Code review is more about maintaining codebase consistency than suggesting alternate solutions to problems for me.
Measure and adjust: Find a source of high-quality code reviews; the easiest source may be the high-level engineers at your employer, but if that's not possible/helpful, I'm sure there's free/open sources. Make your own code review before reading others, then compare and contrast. What did they flag that you didn't notice? Can you generalize to categories you could prioritize more? What did you flag that they didn't? When you both flagged the same thing, whose phrasing was more helpful?
More personally, here's my approach, which may or may not be helpful for others:
* First, skim the entire PR. On a raw, visual level, get a sense of what it looks like. A couple big edits, or a bunch of small updates (like changing a variable/method name, changing an imported library, re-ordering some interaction, etc)? A bunch of indentation changes without much else? An extension of an interface? This is to familiarize myself with the general question of Just What's Going On Here, basic context to use as the foundation for the real review. This should take just a minute or two.
* Second, perform a close reading of the diff. Don't worry about high-level conflicts like ordering of operations or other semantic considerations; stick to syntactical-level concerns and context-free mistakes. Are the variables and other symbols well-named? Is exception handling done properly, is this library not initialized correctly? At the "fit and finish"/polishing phase, this might be the majority of the effort. At the early-draft stage, it may be counter-productive to actually note all the possible issues. Either way, to use an analogy to compiling source code, the goal should be to build an AST of the diff, so you can begin to build a model to manipulate through the rest of the review.
* Lastly, address the higher-level qualities. Depending on whether you prefer a bottom-up or top-down approach to design, you might order this section differently. I tend to try to identify where on the abstraction scale the highest-leverage feedback resides, and then focus on that level. Depending on context, that could mean discussing which component needs to change, which repository/module this change should live in, which classes should be changed/created, what's the behavior of specific methods, etc.
Whatever your feedback, consider what you hope to accomplish with the comment. Are you trying to educate someone about an idiom they don't know about? Ensuring correctness of a complicated interaction? Maintaining a common code style or test coverage threshold? Make maintenance/debugging simpler? Is this something that you view as blocking, or is it more of an FYI? If the latter, it's pretty important to communicate that clearly.
Way too many companies have leetcode down the line after the candidates have invested considerable time in their process.
A few days later I was called back for a phone interview with the CTO who wanted to do a screenshare leetcode type thing. He had no idea I’d even done the take home task, and wanted me to solve a stupid graph traversal problem requiring the same algorithm I’d used in the take home project.
The comments I personally got back were infuriating enough that convinced me of that.
It always seems weird to me. I barely know you- why would I be excited? Ask me at the end of the onsite after I've talked to a number of people across a variety of roles, seen your office, heard about the work, explored the culture, etc etc.
During the initial screen, I really only know about your business, which is not the biggest factor for me.
I wonder if "excitedness about the company early in the interview process" is in any way indicative of on the job performance later on.
If you were a top-tier developer, there was a >0% chance that you would be super excited about the product because you realized the pain point that it would solve for you if you were building a real-time product at another company. And that excitement or lack thereof was a useful litmus test for Firebase specifically.
I agree that "excited about our company" is not something you get up front if you are interviewing at a company that is building an on-demand AI marketplace for ML-optimized scooter rental office spaces.
For such a small team, and an early-stage company, I believe it's critical to hire people who are absolutely excited about the mission. They're not simply looking for "an employee," but a partner who believes in what they're doing, coupled with having the ability to help them do it.
It does seem odd. Unless I'm begging for a job, you should be telling me why I'd want to work there.
... only for the 1-2 top companies they are interviewing for.
I have discontinued multiple interviews because they demanded 4-5 evenings of work for free on short notice, if I am going to spend 10-20 hours per company I have to take two weeks off from my current work to do it. On the other hand I have done the required work if I really want to join a company, but they are only top 5%.
Another problem is, usually what is expected and how the project will be judged is not mentioned beforehand;
- so do I write unit tests?
- do you want docs for running it?
- should it include deployment instructions/code?
- does have to support 5000 req/s?
- how important is clean code for this?
each add more time and these kind of question are never ending.
So unless you are Stripe, Airbnb, Netflix, etc. (Very strong Engineering and better than average culture), I will not bother, will say thank you and be on my way.
Finally, let's say you have done the project and they reject you, they never put a fraction of the time you spent to give you feedback. because that means a lot of time they will spend on someone that will not be hired. Also might open the way for litigation for discrimination in some countries etc.
On the other hand; Once I was managing hiring process of a new grad where we asked her to design a FE+BE system, after receiving it I spent a couple of hours reviewing it, we decided to hire her, she was very enthusiastic and smart, I also sent her our notes and how she can improve the project. Her response was along the lines of "even if you don't hire me I am very grateful for doing the project, did learn a lot" She joined and quickly became a very productive colleague.
Edit: I noticed that the author mentions they keep in touch during the project and provide a code review, had not seen that yet while writing the comment. In that case it makes sense, but the candidate should know they will receive feedback even if they fail which I believe is hard for the team to accomplish but kudos if they did manage to do it.
This is also a double-edged sword, if the experience is horrible, you can be sure none of their classmates will apply to you or even their school if you blew it exceptionally.
There is an article by Joel Spolsky about Hiring great engineers that touches on this point also: https://www.joelonsoftware.com/2006/09/06/finding-great-deve...
> Even though only about one in three applicants who make it to the in-person interview stage passes all our interviews, it’s really important that the ones that do pass have a positive experience. Even the ones that don’t make it go back to campus thinking we’re a classy employer and tell all their friends how much fun they had staying in a luxury hotel in the Big Apple, which makes their friends apply for an internship the next summer, if only for the chance at the trip.
I actually agree with your thesis! You're right that candidates are only willing to put in this level of effort if they are really interested and consider your company a "top 5%" big opportunity.
So it's critical that you convince them that you are in that top 5%! You don't want an employee to join you because their top choices turned them down -- you want to be their first choice.
We regularly won candidates away from Google, Facebook, Uber, Twitter, etc. If you want to build a great team, you're going to need to be able to do this too.
Re: code reviews -- yes, we gave everyone a thorough code review.
Why?
Unless you're offering FAANG levels of compensation, isn't it pretty arrogant to think you're anyone's top choice?
Candidates can easily lie to you about this. In fact in my last job hunt I was instructed to lie about this by recruiters who are actually being paid by the company I'm lying to.
I told every company I interviewed at they were my top choice.
Your intention is to get candidates to fake excitement or drop out?
It's not arrogant. It's simply creating a process where you can filter out for folks who are truly excited to build what you're building and believe in the potential future impact. It might also turn out to be a big payday, or not.
Not everyone is simply chasing that 400k TC every year. If you lie about your excitement, a panel of folks who have went through the same process will easily sniff it out.
Honest question: Would you feel differently if these companies had 1-2 days of onsite technical interviewing instead of take-home problems?
In practice, I rarely get pushback from candidates for our 4-hour take-home problem. We provide clear instructions about expectations, constraints, and boundaries. All candidates have thanked us for allowing them to work on their own time in their own coding environments.
- Unit tests? Yes. It will increase your standing.
- Docs? Absolutely. A must have. You will not be considered really otherwise.
- Deployment? See above.
- Support 5k requests? Not unless that is what the test is testing for, pay attention during phone interview about the role and it's requirements, the test will look for this and other generalizations.
- Clean code? Very, very important. Don't try to show how smart you are, show how audience aware you are. Most likely you will be working on a team, a team doesn't want a cowboy programmer.
Something you didn't mention but bodes really well for anything with a front-end interface: hosting. Executives hate running things themselves, so a hosted version will put you at the top of the list.
It's crazy how many coding tests are poorly designed and waste a candidate's time. I'm putting together a list of good practice for test design here: https://candidatecode.com/articles/coding_challenge_best_pra...
Feedback and contributions welcome
That's a problem even if you have reasonable expectations yourself, because the candidate probably has experienced some tests where the expectations haven't been reasonable. For a (real) example: marking candidates down for not having an analytics integration.
How does your candidate know that's not important to you, even though it was important to the last company?
> testing your experience, not your literal skill in pumping out code
Past a certain point, it gets much more efficient to discuss experience in an interview setting. Good coding tests should be designed to test the skills you care about most to produce a strong hiring signal for the next phase - it's difficult to showcase a full decade's worth of experience in 4-8 hours' worth of coding challenge.
> Anyone can pump out code
I've found that to be depressingly untrue amongst a significant percentage of engineering applicants.
I cannot take this article seriously. And "a history of working on challenging projects with real deliverables" sounds ridiculous, when you know they hired people straight from bootcamps as engineers.
I really do want to hire people “that I like and wanted to work with” though. I would hope most managers do.
FWIW we never technically interviewed internal transfers inside Google (we counted on Google’s normal interview process to maintain the bar). We were just looking for team fit in those discussions.
I’m disappointed you would belittle bootcamp graduates. Many of them are awesome, and we were lucky enough to hire one amazing one. And yes he did have a “history of working on challenging problems with real deliverables”, just not the CS kind in this case.
I agree that 4 hours is a good target, and that the coding challenge must be fun and in an language of the candidate's choice.
> One common objection I’ve heard to take-home tests is that they are too time consuming, and candidates won’t be willing to do them.
As a candidate, I encountered this situation twice, but it was because the coding challenge was poorly designed.
In one case, the coding challenge basically required me to learn two major Java frameworks. As I had minimal Java experience and never touched the frameworks, I estimated that it would take me 2-3 days just to understand what someone else could probably do in 2-3 hours. I have no desire to be a framework jock, so I walked away.
Another time, I did a challenge for an open-source company. They had a list of some bugs, and I picked what I thought was easiest. The language wasn't one that I did any work in. No one responded to my questions, so again, I walked away.
Yikes.
I would entertain the idea later in the interview process, especially if there were some token payment to show that the potential employer is just as invested as I.
This is a rough estimate for the startup provided in this blog post, you can do this by taking the salaries, getting their assumed hourly rate, and then applying it to the amount spent interviewing.
See https://blog.usejournal.com/it-costs-50k-to-hire-a-software-...
> Please note that this post is intended to help with vetting of software engineering candidates at early-stage startups with significant technical complexity.
candidates at early stage start ups need to sacrifice time, energy, and effort in a manner that's not based on hourly compensation for the company to hockey stick.
it's critical to filter out people who aren't willing to do this. the proof is in the pudding: all the people volunteering time/effort, freely, to get better scores on the gold mine game... after they're hired.
not to say those aren't redeeming qualities for talented software engineers. they're just using this curiosity/drive as a signal for "willing to work on cool stuff above and beyond normal salaried expectations, and not ask too many questions about how they will benefit".
there's an obvious tie in to age here as well (younger people are less committed to external causes e.g. relationships, have weaker boundaries generally, more able to be starry eyed, etc.)
And before someone says, "you should be sharp on your skills regardless of an upcoming interview", I challenge you to try a mock interview for a similar company.
Big yikes, Firebase.
Although yeah, I spend my evenings with my partner, and I was working extra hours at my real job, and it was a test you had only a few days to complete, so I had to pull an all-nighter for it.
Amen. I've known people who got through Google's interview process, but even then the response is "Great job, we'll let you know within a few months if there's an opening on a team you can join."
Not a great response when you're graduating soon and you've got concrete offers from other companies that will expire.
Pre-Screening to get to an onsite interview taking 6 hours? I'm sorry, but this is the exact toxic interview culture that is poisoning the job hiring process in tech startups.
>Some of our employees enjoyed the test so much, in fact, that they continued working on their solutions after they were hired
Jee, I wonder if that had anything to do with the fact that they were, you know, new employees? As in, trying to put their best side forward and make a great impression? Did it even occur to you that you are inadvertently increasing their workload with your downright torturous screening process?
If you do this, make sure to treat candidates with respect and humanity. Give them some feedback at the end of the process, so they know their efforts aren't wasted.
1 - take-home task of "only" 6 hours. that's on top of the onsite (~6 hours) and a few hours for phone call and followups. that's too much for an unpaid job.
2 - they try to find people they know to sniff around about the candidate. that's usually bad as you can jeopardize the candidate's current job (usually people interview while still working at another place). overall this post made me want to work at firebase even less now.
Things are a bit different when I'm some schmuck off the street begging for work. If you're trying to build a team and you want my skills, well, in this market I'm not putting a lot of energy into it.
Case in point, I've had a company trying to hire me for over a year now. They're in consumer electronics and it is a fickle market so I kept turning them down, but I had a bad week at work and figured I'd go for it. I'd be an absolute perfect fit given my experience and what they're looking for, so it seemed like things were going to work out great.
I accept an on-site, do well on a whiteboard coding test and a general review of my resume, and then one of their senior folks throws me an off-the-wall algorithm question(random sample a stream of data with uniform probability). Now, I'm an embedded guy, with an EE degree, and I specialize in "software engineering". They have many problems I'm perfectly suited to solving. They really, really need a guy like me(to design and fix drivers, abstraction layers, facilitate porting to new devices and enable third party clients). But here's this guy, asking me an algorithm question, and he didn't even explain it right. I googled the answer(reservoir sampling) when I was finished and understood the solution in a couple minutes, but it was too late. I was flagged for 2 more algorithm-heavy, 'cracking the coding' interviews the following Monday. I really didn't feel like spending my weekend reviewing bullshit I never use in my job, so I turned down the offer. It's too bad, I was a good fit for the team, and I was a great fit for the problems they were trying to solve.
I'd rather spend those hours on a personal project which I can use for demonstrating to potential employers or perhaps use for a side-income.
1. They signal from the first point that all your time belongs to the company. Taking a test of 6 hours is not something I'd have time for.
2. Their company value "Give 110%, 33% of the Time" creeps me out. If someone is not satisfied when you give 100%, run away.
If what you have to offer is above average (compensation, growth opportunities, reputation, culture) people will be willing to make an extra effort. If you are offering the same as everyone else, candidates will deprioritize your interviews.
I am absolutely sure that this is something A LOT of companies say and think about their current setup. Impossible to verify, just something that sounds great and makes the 24-people group feel special.
If you get a request for a proposal where the potential customer / hiring committee clearly just copy/pasted a set of items that don't even fit their company type or area, or that ask for unreasonable stuff from the start, then you will save yourself a TON of time by saying thanks but no thanks.
If someone has an interesting set of question or a clean and focused RFP, fantastic. GO for it, you will enjoy working with them.
Another item - if your customer will need to do some type of work, have a TINY bit of that in your engagement process. Ie, let's say client will need to write a lot of text, have them briefly respond to a question in the process. If they will need to fill out worksheets, have them do a tiny one.
Another one - if they won't tell you what they need or who they are - not worth it.
Startup idea: take away home assignments for a fee. You send us your code assignment we make it for you. We also provide you with code highlights so you can understand the code and architecture faster and have something to talk about on the onsite.
Most of the work is not done by us but by "volunteers" .
Who are these "volunteers" ?
Desperate engineers looking for a role, who are thinking that our fake company is really wanting to hire them.
We also get in touch with hiring managers and convince them to send take away assignments in order to increase our market.
(/rant)
I don't like code assignments because they can easily be abused by both companies and candidates and regardless of my opinion on it I think Firebase is awesome. I was an early adopter before you got adquired by Google.
I figure it hasn't been used for recruiting in a long time, so there shouldn't be any downside of that nature to it being so. But there is the invested time in making it available, of course.
I hardly think I would be the only one thriled to try it just for fun. It would probably provide a great deal of goodwill for you guys, as well as further the industry's understanding of what constitutes a good take-home problem.
…is excited about our company.
Wow, I had no idea who came up with that question first. (see, e.g., https://github.com/alex/what-happens-when ).
When was the first time it was asked? Where?
We're lucky to have this in the software industry as standard practice. Has anyone had horror stories otherwise?
Yeah, maybe if they're interviewing to work for Facebook, Google, Amazon, IBM, Microsoft or other big-ticket dream companies a lot of developers want to work for. But, Andrew was talking about hiring for an early-stage startup, before Google even acquired them.
As a startup, you're asking others to take a risk by working for you. Some engineers have families, debts and commitments to worry about. Working for an early-stage startup that might run out of money or collapse is a very real and likely risk.
It's hard to argue that Firebase was not a roaring success for Andrew and others. But, expecting a candidate to do a six-hour test for a startup that statistically might not have succeeded, and furthermore, listing out the benefits of a take-home test being less stressful and then proceeding to describe what sounds like micromanagement under the guise of helping (by calling candidates at the start and during of the technical test).
And we keep hearing there is a talented engineer shortage. Maybe the problem isn't a shortage, it's companies expecting engineers to be put through arduous and time-consuming interview processes like they're trying to get a job at NASA building human-payload rockets. For every company wanting to take hours and days of your time, there is another company that won't put you through a BS hiring process and make it faster.
I work for a smaller company, I had two interviews for a front-end engineering position. I came in, met with the engineering lead and another senior. They didn't make me do a technical test, they opened up my GitHub profile on a large monitor, handed me a keyboard and mouse and asked me to pick one of my repositories and run them through my code and decisions that I had made. I chose a Javascript library I had built, they asked me things about the architecture, if I had tests, my thoughts on Webpack and other bundlers. It was all related to the library and all answers I could easily provide because I built it. No BS, just a pressure-free interview.
The second and final interview was a coffee with the general manager and the engineering lead. They gave me more detail about why they're looking for someone, explained it was a new position and where they saw me fitting. I ended up having lunch with them and a couple of other team members who joined. It was all casual and pressure free.
This company offered me $50k more than I was currently earning. It was a MASSIVE pay bump for me, and I didn't have to sacrifice hours of my family time just to be told I didn't pass or get the job. It's also a completely remote position, no commute whatsoever. No wonder this company has been around thirty years and many of the original hires are still here.
Sounds neat until one realizes you can be replaced in an instant by one of the ever growing number of new "graduates" every year.
Hi all - I hope you find this useful!
If you have questions or want to know more I’m hosting a live stream Q&A today on Twitter at 1 PM Pacific. Tune in and bring your interviewing and hiring questions! I'm @startupandrew on Twitter.
For example, “Later that candidate did X, which would have been great for Firebase.”
How often did you take subordinates that weren’t fresh college graduates or former/current FAANG engineers?
We definitely hired non-new-grad non-FAANG engineers. In fact, new grads and FAANG alums were a minority for us.
I don't actually buy this. Environment makes a difference, sure, but if someone goes off and succeeds technically[1] at a another company in a big way they would have succeeded at yours as well. If that person can't succeed with you, then there is something fundamentally wrong with your organization. My experience, though, is that such organizations have much lower hiring bars.
[1] I say "technically" because it's quite possible that the person is not very good technically, but landed at a place where that didn't matter and some combination of non-technical skills led to their success.
The total time spent is higher than most interview processes I think, which can be a struggle for some folks. But I do think it's the right tradeoff.
Regarding why time taken is an important data point -- we found a strong correlation between people who were able to get good solutions working quickly and people who were able to effectively defend their solutions during the "GoldMine review" in their on-site interview.
That's not the only alternative though. Why not use a more humane interview process involving conversation about previous work, ideas about the future, strengths and weaknesses?
> The total time spent is higher than most interview processes I think, which can be a struggle for some folks. But I do think it's the right tradeoff.
I don't agree but I'm glad you thought about it. I personally think making your interview process more inclusive allows you to get better candidates, as opposed to structuring the process in such a way to exclude those that have different priorities and life choices.
> Regarding why time taken is an important data point -- we found a strong correlation between people who were able to get good solutions working quickly and people who were able to effectively defend their solutions during the "GoldMine review" in their on-site interview.
Correlation doesn't equal causation though. I also want to point out that if you are expecting someone to "defend" themselves, that means you are on the offensive. Interviews don't need to be adversarial.
Edit: I do want to put in one good thing I got from your article to offset my rather negative responses. You mention that even if you reject a candidate you gave them the full code review and feedback about why you rejected them. The culture of secrecy surrounding rejections is something that needs to die, and I'm glad that although I think your process as a whole is unfair, you took the step to change that toxic part of our industry's interview culture.
Because I'm hiring someone to build technology, not to schmooze about it. I need proof of ability.
It's easy for candidates to exaggerate or misrepresent their accomplishments. This is especially true when they worked on large team projects. For example, if they say "I was part of the team that built Google Search", that could be a really impressive thing if they played a key leadership or technical role, or it could be a really unimpressive thing if they hung out while others did the heavy lifting, and it's very hard to tell the difference.
In addition, there are levels of technical chats that become very difficult to fake, further reducing the pool of lying interviewees people claim to be defending against.
That's a pretty idealistic view of people. Most people exaggerate on their resume. The most reliable tool (at the moment) to cut through and establish some common metrics is the technical test. How do you compare technical "chats" between individuals? Duration of call? Gut feel? How do you ensure that the person on the phone has the breadth and depth of experience to thoroughly discuss any and all projects on the phone with the interviewee? Is it the same person on all of these calls? It can't be, so how would you compare between two interviewers' subjective feedback?
> This is not without cost to the employer...
This is a HUGE cost, especially at a startup. A mis-hire at a startup can sink the entire ship.
I do think that your approach has merit, and I personally dislike technical interviewing, but for a startup the OP's approach is, IMO, the best tradeoff.
There are tradeoffs, but I found it to be an overall net positive in addition to the technical assessment. Doing this effectively requires that the interviewer have a pretty broad exposure to lots of different technology, otherwise you can't distinguish between someone BSing their role vs something you just don't know a lot about / can't ask intelligent questions about. It's also incredibly subjective, which would make me hesitant to rely on it without more objective criteria.
It's the "eventually" part that's problematic, since it can be long, uncomfortable, and expensive to the company.
Seems reasonable to me.
Which means the candidate will be under incredible stress still, no matter what the supposed target time is.
This process is so weak, when I heard about it in 2013 I was like...why would I ever want to work at firebase?