How to Interview Entry Level Software Engineers
technology.cloverhealth.com
technology.cloverhealth.com
This is then done for every possible combination of approaches (homework questions, whiteboard interview, pair programming, fizzbuzz questions, algorithmic questions, work-history questions, career-progression questions, mini-project, magical incantations, and so on...)
It may sound cynical but at this point it feels like no one has any new ideas about it and the same content keeps getting recycled in the form of yet another article and yet another comments section.
Half of these articles are from company blogs which exist largely as just a marketing tool.
Since there is no actual new thought, everyone just cargo-cults the Google approach.
It's a gamble, no matter how in depth or complicated your hiring process is (from personality interviews to whiteboarding to take homes). Being interviewed and doing well is a skill set and it's very different from a full time work-place skill set.
All of these interviewing processes that get refined and more lengthy just create a new class of people who are good at being interviewed.
I don't know what the solution is, but making candidates jump through 3-4 interview rounds, memorize algorithms just for the sake of it, and go through 4 to 8 hour on sites probably isn't it. Even if it happens to give you a better-than-average employee, what are you sacrificing (as a business) in terms of man-hours, productivity, and focus?
Personally I'd like to see more companies institute a temp-to-hire process. Bring in new candidates for a two or three week contract, see how they fit in, pay for the their time (at a discount), and then go from there. If it doesn't work out, you get some work done for your company, and they get some relevant experience for their resume.
Writing the perfect resume and re-reading CTCI for the 10th time won't tell you how well the candidate will perform, but seeing them actually work might.
As to your point about marketing - I agree. Not one of these approaches are really novel or helpful to anyone but the company themselves, and is just a glorified "look at how we took a chance" tactic.
In the real world, people generally don't put in their notice until they've accepted an offer at another company.
The going rate is what the market sets eg for developers in London its £400-500pd for grunt devs in Java or Perl
Actually your downside is a side benefit to employers: you don't risk hard feeling by hiring somebody away from their old job if you latter need to work with them. (these shouldn't happen, but they do)
I wonder if there are statistics on contract to hire positions that don't get hired, and out of those that don't, is it because the person really didn't work out or was there really no intent to hire to begin with?
As a point of reference, where I work they almost always to contract to hire. I've seen a few contractors leave (they found other work), some didn't get converted to full time (those weren't actually to-hire though, and specifically short-term, although in one case a contractor was brought back twice to fill in). But in almost every other case the person was hired eventually (even if it took a couple extra months to push it through HR).
It would be great if the only reason people could be let go were legitimate, but that's hardly the case.
If you devise an interview framework or a hiring profile, no matter how many traits you want to evaluate(as the article points out), they will lump those together and measure in some scale of how likely they are to hire the candidate(sometimes that is even just binary), if a candidate can emulate their desired profile he gets a high "score" and gets an offer, the bigger the financial incentives to do this, more people will game the system(prepare for specific interviews), think, if you can play the part for a couple of 1-hour interviews, you can get a 6-figure salary at a tech startup, that is a high ROI.
Most companies can't stop that, there isn't limitless budget for interviewing so the interview process has limited resources to evaluate people, which leads to false positives. There are also a lot of talented people that don't practice their interviewing skills enough and end up as false negatives.
It's human nature, people need to evaluate things on a single criterion and you can game that. The same thing happens with school, you can work your way through school material to maximize your grades without learning much.
Also, every person you are interviewing has the option to interview with other companies. With that in mind, you should always be interviewing even if you aren't actually looking to hire at that moment. Sometimes you might find a really good candidate but, most importantly, you will have options yourself.
Last, I really like your idea of a paid temp position to assess fit. This will benefit everyone tremendously.
If it was the company's goal to crank out gig workers then there are some really big problems happening below the surface at that company and it was in the candidate's best interest not to get a full time offer.
Is it the high salaries? In that case, do you see such testing in law, or finance?
Is it the technical difficulty? In a previous life (read: up to a few months ago) I worked as a mechanical engineer, in a pretty technical field (numerical simulation of structures). Additionally to demanding reasonably deep knowledge in a few areas, the job came with the responsibility of making sure that your designs do not kill or injure people.
I had never had a technical test in an interview before I interviewed for my current IT position. Nor have I heard of colleagues having such a test. Is it because most engineers in Canada are licensed (myself included), and most jobs require you be licensed? If that's the case, it's pretty funny, because at least in my province, the test you have to pass for the license is entirely about ethics and professionalism, nothing technical.
Part of the code of ethics though, is that it's on you as a professional to ensure you have the necessary competence to do the work.
The barrier to entry for software engineer jobs is all at the interview. That's why it's so hard, tedious, and complicated.
I'm aware it's a regulated profession, but imagine if it wasn't? Whats there to stop management from actually fast tracking the backup pilots as that?
That is, in fact, how every Air Force in the world recruits pilots.
Tricky brainteaser questions are popular in quant finance interviewing. Example[1]. There are also discussion forums dedicated to it. Example[2].
Some electrical engineering firms will ask you to analyze complicated circuits in the interview. Some even have "take home tests" where the candidates design a circuit according to the interviewer's specification.
[1] http://www.streetofwalls.com/finance-training-courses/quanti...
1. Pay a lot. Barring trading, you'd be hard pressed to find a job that routinely pays 200K+ to 25-year olds
2. Are, at least most of them, not fundamentally difficult for a generally smart person, despite whatever people convince themselves of. I am not talking of the so-called 10x engineers here, who you are unlikely to find through standard interview techniques.
3. Pride themselves on NOT being credentialist (at least on the surface). You can't wake up tomorrow wanting to be a lawyer, read some books, and get a good job offer in a couple months. It's possible for CS jobs.
Routinely? Hahaha.
Those salaries are pretty much exclusive to a few locations, primarily Bay Area and maybe NYC.
If I see a bridge, I don't need to be an engineer to gauge if it is likely to support my weight. But if the bridge is invisible, I probably want significant assurance that no one is pulling a fast one here, so to speak.
In a field like mechanical engineering, I assume an employer can take for granted that a new grad has a certain set of skills (general physics, structures, fluids, etc.). That can't be assumed in quite the same way in software engineering.
In that case, should we conclude that universities are the failing piece here, and that is why even with a 4 year degree, employers feel the need to test candidates for their knowledge?
Medicine, law, pharmacy, actuarial science, accounting...
For most upper class jobs it's the norm.
I was thinking of the complicated interview process, which seems rather exclusive to software, at least as far as I know.
Medicine, pharmacy, actuarial are all licensed professions. A lot (most?) law and accounting practitioners are also licensed.
So, is the difference for software that it doesn't require a professional license, so employers are forced to sort of invent their own "licensing" in the form of these very difficult interviews?
Not that I'm advocating for licensing by the way, it comes with its own set of problems (notably rent seeking behavior). Just trying to understand why software as a field seems to stand out here.
When I interview at AmaGoogFaceSoft, they can't take my word for it that I know how to code my way out of a cardboard box. So my interviews tend to be fairly technical, making me prove that I know what I know, and might actually be a useful asset to the company.
When my wife (a CPA) interviews, they know that she knows a whole bunch of stuff, by virtue of being able to go the accreditation board. So when she interviews, it tends to be a super mile high sanity check (have you done accounts receivable/payable before, yes? no? If no that's ok too...)
I can't speak for medical, but I have family in law and it tends to lean a lot more towards the accounting style for interviewing (barring maybe some patent law or other specializations). E.g. one had an interview for a corporate banking regulatory position that went roughly "Oh you have no experience in banking or regulatory law? Well that's ok, welcome aboard anyways."
All the accreditation board knows is that she knew enough at one time to pass the tests. It says nothing about whether she still knows that stuff today.
It's the same thing with a PE license. In fact, with a PE the advice is to take the test as soon after graduation as you are able, because the further out of school you are the harder it is to pass. So, it says you knew that stuff once. It cannot be used as a measure of current competency.
In my (canadian) province, the actual license test has no technical component, but there is a continuing education requirement, plus the deontology/ethics requirement I mentioned in the GP.
This is why there are programming candidates failing fizzbuzz.
It’s the equivalent of someone applying for a pharmacist position and not knowing what antibiotics are, but it happens, because anyone can call themselves a programmer.
It actually shook me a little after the 2nd time, because I thought I was doing the right thing not dismissing people just because of gut feeling.
Make the process a conversation rather than an interview. Get to meet the person, have lunch with them, talk about what motivates them and what they're passionate about, discuss tech stuff, and at some point during that hour or two both you and them will have an idea whether you want to turn it into a steady relationship. :)
I like to add a code review stage to the mix, where we show them a small chunk of intentionally badly written code that might come up during a code review, which has both obvious and more obscure issues related to syntax, clarity, efficiency, etc., and you judge how they approach it and what issues they bring up. I've found this to be quite a useful metric that mostly replicates a real world scenario, but in a more comfortable setting than asking them to write code on a whiteboard.
And you try your hardest to block any biases for/against them you might have when making your decision.
By volunteering along side the company doing the software build(which was also volunteering, mostly) we were able to get the project on track, and update the client's expectations to a more realistic view of the timeframe and outcome.
By volunteering alongside them, for another org that we both were happy to support, I didn't feel like I was doing free work for them, and they weren't struggling to pay full salary and onboarding cost to find out the same.
Just a thought for an approach to get a sample of each other without one party doing free work for the other.
I also work at the company this article is about, and the article was written by one of my colleagues. While no process is perfect (including ours -- both the one detailed in this article, and the one we use for general engineers), one of the things that I like about working where I do is the emphasis on putting thought into the interview process and questioning a lot of things that get assumed without a second glance by interview processes elsewhere.
For an entry level engineer, this removes a bunch of variance and also because it gets you another scenario which happens way often in actual work.
You show up with the code and it gets reviewed & discussed on how/why of the implementation.
The most talented entry level people I've worked with have struggled with criticism instead of grading (i.e "grade the code" through a bunch of test-cases is different from "why did you do this and explain").
I recently had an old boss try to bring me on to a new job with him, but wasn't able to get me through the interview phase, despite knowing I'm capable, because my interviews with the rest of the team went so bad.
I've been a software engineer over ten years. All of the positions that I have been offered have come from especially kind/empathetic teams, and the interviews usually focused more on conversation than quizzing.
It requires an unfortunate amount of rejection before coming across my more selective criteria, but I guess that is what it is.
How do you think that would have been with regards to your anxiety?
On the other hand, I’ve had take home assignments that were total wastes of time because all they really cared about was whether you coded in their particular OOP fashion. I made a functional demo which performed well but they neglected to mention that they were looking for OOP design paradigms. Mention should be given of current methods they use so that you don’t spend a week going down a path that won’t tell them how you will fit into the team.
Some things that help me in that situation are when there's some light/friendly conversation preceding the test, or when there's a limited the us-vs-them size, like only one or two interviewers at a time.
* I'm under pressure to impress, but also be my true self. I've had little contact with you so I'm not sure how to gauge all the feedback when I have increased sensitivity. * Sometimes don't have a real calibration for the level of the test. I've done the recommended prep, but it doesn't feel indicative of the test, or the interviewer inserts and additional level in before we've achieved the basics of the test. * What if I want to rough something out, but feel I'm exposing what I know would be a weakness from my pre-test assessment? I've just gone and made a hash of something I knew I needed to know.
I'd say "how you collaborate" would be the biggest factor. If you start out with some code and fix up something, have me fix up something, and progressively raise the difficulty, then I'm much less likely to freeze up. I don't believe I'm particularly unique, but I don't find many tests delivered in this manner. It's more a case of "here you go for your practical test, show me what you can do kiddo" which just demonstrates how little programmers know about assessment, interviewing, and psychology.
I'm going to state the following because I think the question was directed to one individual who identified as having a particular disorder. I don't believe I have any specific disorder that would cause me to freeze up. I'm an introvert, tested under Myers-Briggs as an INTP (if you consider that any value), and am OK in networking events. In jobs I've always tended to end up in regular conversations with skip-level managers etc.
Problems: the existing employee ends with even more latitude to wreck the interview. You need to ensure that the interviewee is taking the lead. You probably also need to record the whole thing for objective review by someone else, which you'll have to disclose, which could make it awkward again.
Ed: I've also heard of processes where you put multiple interviewees on a task together. That has some similar potential benefits.
In most cases you are not looking for the people with the most level of CS knowledge, but rather someone that can think logically, that learns/ramps fast and have at least some explanation, even if wrong, for his choices and is able to take the observations/review as a learning opportunity.
While I can't be absolutely sure that they've done it themselves, a big part of the in person interview is based on the homework, asking them how they would extend it, add specific features, potential performance problems they see, etc.
Those answers are hard to fake if you didn't do it yourself
We had one case where a candidate used some open source code snippets in his solution, not declaring and removing the comments and license headers. Still it was very obvious in the difference of coding style.
The task was super simple - basically just duplicating some of the configuration and renaming some values to create another job, but at the time I just froze and couldn't see it. I managed to get some sort of answer out, but wasn't happy with my response. They claimed that they were meant to give this to me in advance as a homework question and somewhere the ball got dropped, but I had a feeling they wanted it to be a more on the spot thing considering the problem was so simple.
Thankfully I had the wherewithal to ask if I could take the problem home with me after the interview. When I got home and looked at the problem with fresh eyes I facepalmed so hard. I wrote up the correct answer and emailed it over to the company recruiter, hoping she would pass it along to my interviewers, but didn't get my hopes up.
I guess she did, since I ended up getting the job. I still cringe a bit about how I just blanked on that stupid question. Thankfully I got ramped up really quickly once I actually started the job and had no problem figuring things out after that.
1. Unemployment can be extremely stressful and there is a very good chance the person needs a job NOW. Every little thing you do to dilly-dally instead of making an offer is another hurdle that may send the applicant elsewhere. Act as soon as possible when you see a résumé you like (get on the phone, find out what you want to know, make a decision; if you bring them to interview and they’re struggling, give feedback immediately; don’t have a system where you wait 3 weeks to tell them yes/no).
2. Any effort you expect an applicant to spend will be greatly multiplied. That person is not talking only to you, they’re looking at LOTS of job postings!! If every potential employer starts assigning “homework”, it becomes a major hurdle and you can bet they’ll look a lot closer at an employer that might provide a paycheck in a couple weeks instead of “homework”.
3. You don’t need complex problems to understand if somebody can code. Let them use whatever language their résumé claims they know, give them something to do for a few minutes and see what happens. If they get stuck, give them a solid hint and again, see what happens. You’re trying to assess things like (a) if they appear to know what their résumé claims, (b) if they are the kind of person you can interact with to get somewhere on a problem.
What do we think the median interval from "established intent to interview" to "offer" is at medium-to-large software companies in SFBA? If I had to guess, it's on the order of a month. Nobody's hiring homework process puts a dent in that timeline. On the other hand, when done well, "homework" makes it easier for engineering teams to make confident decisions, which expedites offers.
How much effort do you think it takes a typical candidate to get through an interview gauntlet at a tech company with more than (say) 50 employees? It's at least multiple full days worth of aggregate effort, almost always including one actual, on-site full day. To the extent that homework offsets on-site interviews, it reduces the amount of effort candidates need to put in.
You basically already know what an entry-level hire will know: they’ll remember most of what the last year of their degree taught them, and have a fuzzy memory of anything earlier. They’ll be about as good at problem-solving as you can see on a whiteboard. You’re not paying ultra-senior salaries out of the gate; you’re just getting what you pay for so bring in somebody promising, start them on a project and let them begin a career with you.
Like I said, put yourself in the “new hire” position. They should be treated better given their likely circumstances:
You have to survive, you need rent and food and whatever else. You have debts. You’re probably working some non-ideal job (or 2 jobs) in the meantime for enough cash, enjoying all the no-free-time-left that gives you. If you’re really lucky, only a handful of companies have given you some stupid assignment to do with your spare hours, and since it’s “homework” and you have “more time”, they probably dreamed up a more involved problem than the average whiteboard question would be. Then you feel extra pressure because you really want to get it right but in reality you probably ended up rushing the thing and making some silly mistakes that you won’t see.
Never mind that “homework” is just the latest example of unnecessarily-long-winded hiring practices. Once you add up every little thing that every company wants, you’ve spent a lot of time that you don’t really have. (Some want you to create a new web account and profile that redundantly re-specifies 90% of your résumé. Some require you to specify your résumé as a file, then they scan it and scatter it into the wrong fields and discard the rest, requiring you to fix it all. Some have other forms for you to fill out. Some want multiple phone interview half-hours of your time, multiple on-sites, etc. and still can’t get back to you in under 3 weeks. Now do that 10 or 20 more times and don’t miss your next shift at the coffee shop.)
Who needs a full 1 day of interviews plus time for phone interviews for an entry level job.
That's why we started doing take-homes: because we kept having the experience of interviewing people with impressive resumes and sterling references who did not deliver at the caliber of the rest of our team. But even though we started doing homework challenges as a way of screening out bozos who interviewed well, we quickly learned something much more important, which is that for every talented developer you'll talk to with a strong resume, there are many --- probably at least 10's --- of talented developers who are underemployed because they don't have strong resumes.
You also assume a job ad that basically says "We're the world's leading cat sharing company. Wanted: senior developer. Requirements: 15 years Python experience, must do homework problems". I don't blame you for assuming that because for some reason not a single recruiting team in SFBA seems to be able to write a decent job req, but that's not what our job reqs said, and we had zero problems.
Finally: what's a company that hires software developers at any rank with less than a day's worth of interviews? I think you'll have trouble naming one. Even if you can, I don't think you'll have rebutted my point about the amount of energy it takes to complete a typical interview-based process.
Also, I work for the company which published the article under discussion here, and did the homework exercise when I interviewed there. It was refreshing to be given something that was an actual open-ended real-world problem, to be solved in real-world conditions.
If our expectations are higher, then what I look for is code sensibility. Do they cut and paste the code without thinking "hey, this is kind of dumb to do"? Do they see repeated code and think "this can be collapsed into something more succinct"? This is the hardest thing to train for, and the most time consuming from a mentor point of view, so if we have a high bar for entry level programmers, then code sensibility is one of the factors on top of the 3 above that I look for.
I've rejected interns that worked for me from a permanent position because after 4 months, they still had terrible code sensibility. I had to pour through their code with a fine tooth comb because it was riddled with tiny bugs, and it needed constant rewrites because they just didn't improve over the course of the internship. I've had other interns that just "got it" and wrote great code from the start, or only needed a few reviews and then "got it". Those are the ones that will be productive quickly and won't hamper the rest of the team with lengthy reviews.
I like the homework idea. I don't know about "coming back" but maybe send it to them a couple days prior to the tech interview.
I've personally been in the situation of receiving a list of 10-12 required features, with a stack that the recruiters know I am unfamiliar with, and then been told to only spend four hours on it.
While this was an extreme case, I find that the unreasonable time expectation is not uncommon. That is quite frustrating as an applicant.
I am an experienced dev, but couldn't tell you my birthday in a technical interview. All the rest of the interview is fine, I only 'freeze' up at technical questions.
Sucks.
I'm just underselling myself, shooting too low. At this point do I need to be applying for management?
So 1000 gets flipped to 0001 ?
uint32_t flip_bit_endianess(uint32_t word) {
return word;
}
And like I said, this is for an experienced embedded software position. Ironically enough I care less about showing that you can code in junior positions. Those are more "does it seem like you can learn?"
Where would this be used in the real world?
To be real, I'm not even looking for a particularly efficient solution; just "can you make the tests all turn green with ors and shifts" is what I'm looking for.
There is a O(log(N)) N is the number of bits solution where you swap each pair of bits, and then swap pair of pairs, and then each pair of pair of pairs, and so on. I want to say that we clocked it to 15 cycles for a 32 bit word on a semimodern superscalar core that can do each half of each pair in parallel until you or them back together.
All of that being said, I'm really just looking for a for each bit, shift and or it into the right place, O(N) solution. It's mind boggling how many people with wonderful resumes can't do that in an interview time block. Hence why I don't have a lot of patience for the whole "well I'm senior and have a great resume, so you shouldn't have to make me do a stupid coding thing that's beneath me" mindset.
> Where would this be used in the real world?
Yeah the problem itself is a little contrived, but it's a good use of the sort of logic and putting the pieces in the right place at the right time sort of skills you use in embedded programming, and more so HDL, which we require of our software engineers too. And I have seen it in the real world a few times on a serial line that has the wrong bit endianess.
[edit] Note this is obviously for 16-bit integers.
int flip_endian_helper(int num, int i) {
if (i == 16) {
return 0;
}
int shift = 15 - 2*i;
int workingBit = num & (1 << i);
int partial = shift > 0 ? workingBit << shift : workingBit >> (shift * (-1));
return partial | flip_endian(num, i + 1);int flip_endian(int num) {
return flip_endian_helper(num, 0);
}In theory, I like the approach they've taken to hiring associates, but at the end of the day, what really matters is: are the people they end up hiring successful at their jobs? It might take a year or more to really answer that question.
I don't feel like we're very good at this. As 'kinkrtyavimoodh points out in another comment, we see a lot of "how to interview engineer" type posts. But we never find out if, in the end, they were effective. This article claims that everything went really well because they had (or believed they had) such high-quality candidates that it was difficult to choose at the end, but they don't actually know that. They won't find this out after evaluating actual on-the-job performance. It might have been hard to pick from their candidate pool because their process failed to tease out flaws that are only going to become apparent later. We just don't know, and likely will never know.
Well...
> Some of the qualities we looked for in our associates in particular were:
> Resilience: Learning on the job is hard and we assumed that the associates would make mistakes and struggle through difficult concepts. We needed people who could endure these struggles and bounce back ready for the next challenge.
> Willingness to learn and the initiative to do so: Clover would assist the Associates in their growth, and provide teachers and mentors to help along the way, but any incoming Associate would need to be responsible for their own growth.
> Humility: This is an important trait for all Clover engineers, but we paid special attention to it in our Associates. They would have to learn from those around them, be respectful of others, and be able to take difficult feedback with grace.
> ...
> Deciding the technical skills to evaluate was a long process. We expected to teach our Associates most of the technical skills they would need to do their job, but we couldn’t accept candidates that were completely blank slates.
It really sounds like this is measuring the candidate's "potential".
I would argue they defined "potential" as "resilience, willingness to learn, and humility", which they feel is an atypical definition.
(Personally, I think that's a decent set of criteria to look for in entry-level engineers.)
with experienced people they always think they are at or near the top of the scale, but dont understand that you have a different scale entirely.
with a junior person, ask them to describe a project they worked on. if they 'i dunno, school stuff', then you're done.
if you can engage with them meaningfully about their school work, or if they've done personal projects, have developed any kind of insight, show a spark of excitement or creativity then you're done - hired.
just keep an eye on them for a while to make sure they're engaged and absorbing culture. invest a good amount of time in their success (mentorship, pair debugging, whiteboard talks). if they dont start contributing more than they are costing in time after a few months, find them another role or fire them.
Even the company that eventually hired me didn't look at my website.
edit: oh, maybe i wasn't clear. school projects are fine grist, you just have to be able to say something interesting about them
I would assume you'd be asking about how the address and data busses work which varies by cpu and architecture (von Neumann vs Harvard) presumably you'd be happy if some one explained how PIC memory access works ?
If they asked the subjects to modify the homework slightly during the debriefing, that might be enough.