How we interview engineers at CircleCI
circleci.com
circleci.com
> Our ultimate goal is to have the best people here, doing their best work.
I feel like everyone is so afraid of hiring a 0x or -1x programmer that they don’t want to take a chance on a mere 1x programmer.
If I ran a company (I don’t!) on the scale of hundreds or thousands of developers I’d be very choosy about who I let into senior/leadership type roles, but be rather accepting of individual contributors, the hope being that many of them would grow and develop their skills.
Am I just vastly underestimating the overal strength of a random sample of 1000 candidates? Are there really only 3 worthwhile developers?
Past a certain level of skill, it really does come down to specific experience, priorities, communication styles, etc. There are many worthwhile devs that simply have a less ideal package for that specific role, at that specific company, at that specific time.
It still seems high, but I can imagine these average numbers being inflated by some important 'Lead Engineer' type of position, for which the extreme vetting is reasonable.
A few companies do manage to do it, but as far as I can tell you have to set up your corporate culture, HR department etc. from the outset to be prepared and ready for firing people frequently, and that's not something many startups (other than maybe Netflix?) were/are really well prepared for...
Firing doesn't seem hard but I think your point does stand that the cost of having to do so is kind of great. Find, hire, onboard and train a new employee, one that you hope does better than the bad hire.
Basically, there's a ton of completely unqualified (or even under-qualified) applicants who apply to just about every job opening out there. If you've ever been on the other end of the table, posting a job opening and sifting through applicants, the situation becomes clear very quickly.
I'm actually surprised 250 make it past their initial screen. The rest of the numbers seem reasonable to me though.
We have benchmarks to understand how well a company is converting at each step in their funnel. CircleCI is converting well in 2 of their funnels, but the other 2 are underperforming significantly per my other comment in this thread.
This process is never perfect, but I think when you don't hire someone it is always nice to provide something solid the interviewee can go off of. Even if the response is simply that we felt we had stronger candidates for whatever reason.
https://www.quora.com/Why-dont-companies-give-interview-feed...
That top answer there leaves a lot to be desired. Suggesting that a candidate offers "no value" to a company if they are not hired is very short sighted.
But that's a fair point about waivers not being legally binding.
I've found most of the time, unfortunately there's no good in giving details. I realize people genuinely want to improve, but most of the time the reasons we don't hire someone aren't actionable.
And, parent had a good point to. The one time I asked for feedback and received it, I did react the same way (Them: We didn't see this; Me: But I have done that!). I figured while I had their attention to give it a shot. I was already rejected so what's the worst that could happen? Anyway, I tried to be extremely thankful and polite about getting the feedback. Should've done the Glassdoor thing since they also actually formally rejected me and did so in a timely manner.
It seems to me like if there was some standard of proving technical skills, both parties could save a ton of time and go back to regular culture-fit style interviews.
Of course, for the highest paying/most demanding jobs interviews would have to be tailored - but I don't really see why junior/mid level roles need interviews that last 4-8 hours on site plus any pre-on-site screening that needs to happen.
The PE exam is traditionally taken by people in the physical engineering disciplines; mechanical, structural, civil, and the like. By passing this exam you earn the right to use the 'Professional Engineer' title. Many (all?) permitting jurisdictions require non-trivial construction projects to be reviewed and approved by licensed PE's - this is not a joke certification.
In the physical engineering world (structural, mechanical, civil) the PE exam is seen as a mark of a competent engineer, and usually comes along with a significant pay increase, even if the engineer is not using their license to review and approve projects.
Seems to me like the PE exam would be an appropriate exam filter for software hiring.
Mechanical engineering has that because it's been going on for a very long time, and has had time to converge on such a canon, as have many other professional disciplines. But programming hasn't.
This is why the few successful certifications in our industry are issued by a body which controls a specific canon - Cisco can certify you as a network associate on its switches, Microsoft can certify you as a solutions expert on its development platform, etc.
https://ncees.org/ncees-discontinuing-pe-software-engineerin...
I've overheard some other fields and the rigueur they go through is laughable in comparison. Heard a Marketing guy brought in for an interview in the next room go "I'm the right guy for the job!" basically, his only credentials being one previous job in the field and a Marketing degree, and my boss comes back and goes "Hmm, he seems very confident! Maybe we should hire him!"
Meanwhile in our field, having 15+ years of direct experience means basically nothing other than they will set up a phone screen. The inevitable technical interview seems to be designed to rob us of our confidence as soon as possible, putting us on the defensive when we dare to get one question wrong because we couldn't remember or study for 100% of everything that exists throughout all of computer science and technology and our domain specialty, etc.
Even my S.O., who's in proposal writing, only has to make sure she has her existing portfolio of past work updated and printed, but otherwise does very little to prepare for interviews. She doesn't try to study every possible question in that could possibly be asked about technical writing, marketing, graphic design, RFPs, sales, and whatever else her field touches. She generally only gets background and personality questions, that's about it. Yet she's directly responsible for her companies winning millions of dollars in business in her job, per proposal.
The default of distrust in programmer abilities amongst companies is so bad I've been very tempted to leave the field at times, because I'm sick to death of going through the endless hoops and basically having to refresh my entire degree when I want to start looking for a new job (which I'm overdue to start doing again and I've been dragging my feet partly because I don't have the time right now to refresh the skills that haven't been needed at my job the past few years).
If some one had done an "engineering" degree an done a year working in industry you could expect some level of competency.
Are they compensated for this work?
If I were betting on this I would wager that something like 95% of the time, the existing employee would be able to solve the project or problem much faster without the candidate's input at all, but the point is to try and approximate a more realistic work experience and environment.
I think that's a reasonable goal and I don't think CircleCI should be vilified for it.
Not at all.
> Do you believe that CircleCI has some nefarious plot to increase engineering team productivity by mining spare workcycles out of interview candidates?
Your tone suggests that you think this is impossible, but it happens all the time: https://news.ycombinator.com/item?id=16661338 That was on the front page three days ago.
The GP's concern is legitimate; job interviews should not be an excuse for unpaid labor.
That's fair. To be clear, I don't think it's impossible, and especially given what I've heard from friends who've worked in freelance web development there's a huge long tail of scummy opportunities out there.
I more meant to suggest that it was unlikely this was CircleCI's motivation.
Also an important distinction between these two situations is that in the example you point out, someone is being assigned a specific task to produce some new IP which the company can then simply steal, whereas in the CircleCI case I am assuming the "problem/projects" they have you pair on are something in the neighborhood of e.g. "let's go analyze some page load performance data together and see if we can fix some low hanging fruit" or "let's figure out how to set up an alert in our Slack channel for this signal we have in some logs".
Yes it's still true that a candidate could come up with some brilliant insight that saves CircleCI real money in this scenario, but it just seems unlikely and not something they could profitably build a development strategy around.
How should companies interview people in a manner that assesses their technical skills but doesn't "exploit" them? Personally I think the algorithm based interviews are great but also see value in these modified approaches of a short take-home assignment.
In this specific case, that is not what is happening. Rather than assessing candidates on some arbitrary algorithms question (which people are constantly complaining about), they are giving them a real problem the team is currently tackling. It's your time to shine and show how well you work as an engineer on real problems. I'm baffled that companies have finally heard the non-stop complaints about algorithms questions and are treated to new complaints about not being compensated for time spent interviewing. What an amazing time to be alive
"What an amazing time to be alive"
My assumption would be that if a candidate's code is good enough to get released, then the candidate is, by definition, good enough for the company.
I also assume that most of the time spent in this "interview" in a live codebase is the employee asking about how the candidate thinks and decides on things, not for the pair to actually produce working code.
I also assume that the code used within the codebase is non-business-critical stuff like an internal tool.
Anything else is "too much work" on the candidate.
A take-home? "I don't have time to work on unpaid stuff!"
A pairing session? "I don't have time to work on unpaid stuff!"
Software engineering trivia? "Why are they asking me to build a linked list when I'd just use stdlib in real life?"
Asking about stuff on the resume? "They've never worked on ${thing}; how could they judge me on it?"
1. Hiring a new employee into an existing team permanently affects the team's dynamic and morale. Bringing in a bad fit can have undesired cascade effects. Bad team members can cause other good team members to leave earlier than intended, for example. You can counteract this by having the entire team interview the candidate, but this doesn't scale for teams larger than, say, three candidates, doesn't work at all for distributed teams and takes their time away from work that needs to get done.
2. Hiring an employee is also a legal risk. Firing people in "protected" classes is touchy and can take a while, for example: https://www.entrepreneur.com/article/62846. You're also opening yourself up to unnecessary lawsuits that just add more of a headache for Legal to deal with...if you have legal. (Early-stage startups typically don't have legal teams.)
3. Hiring engineers is expensive beyond their base salaries. Firing bad engineers can be a huge waste of money.
If you're an early stage startup, you're better off hiring contractors for point work and taking time to find the best hires that you can. Later-stage or profitable companies can afford to not hire "false positives". At worst, a potentially-good candidate you're on the fence about can come back in six or twelve months and try again.
Yet you have to tailor your ad language so more women apply because they get scared by words like "objectives"?
I don't even know where to start here.
I think it's great that Circle is trying to do their part in being as inclusive-minded as the can be from the research they've collected.
If at the end of the day that is somewhat misguided, I'm sure they'll readjust, given that they have already proven they will, where many others have either explicitly not cared, or never even thought to.
By trying this hard to become "inclusive" for women they are potentially discriminating against others.
> Studies have shown that that women are less likely to respond to an ad that has overly masculine or aggressive language (despite being qualified), whereas men will apply regardless.
Having a job description more gender friendly encourages more women to apply and men will apply regardless.
You should read the article liked to this article - https://hbr.org/2014/08/why-women-dont-apply-for-jobs-unless...
Women apply to jobs only if they are 100% qualified whereas men doesn't get deterred by lack of qualification.
I don't see where the article shows that men will apply at the same rate regardless of aggressive language.
The trade-off of less aggressive language resulting in both more applicants but also a higher percentage of under-qualified applicants will eventually reach some inflection point where it becomes unproductive, as you're almost certain to already have a well-qualified fit in the batch of interviewees.
What you miss here is that men are often overconfident and women are under-confident. It has nothing to do with qualification. It has everything to do with confidence.
And yes, women are often less confident because tech is an industry often marketed for men.
You mean:
- leading
- competitive
- objectives
- driven
- independently
And this is what is considered "masculine". Not only that sounds incredibly sexist and stereotypical, it is demeaning for women who would consider themselves possessing one or more of the above traits.
For reference, I am a woman but I am often discouraged to show leadership skills or be competitive with others, especially with guys at work. I can feel the air change when I exhibit these traits. People start to distant themselves from me because I don't fit the culture norm. I feel I can do a much better job if I showed more feminine traits like listening more and being empathetic and showing I'm a team player.
So yeah, it is easy for a guy to be critical because he has no experience in the matter!
If you want to know more about how women are treated differently here's an article - https://www.fastcompany.com/40456604/these-women-entrepreneu...
Men are not immune to this either. I have been successful in many work endeavors in the past and drew praise from management, to discover soon after that half the team would subtly dislike me because I'd make their work look bad.
Also, competitive people with leadership skills are an asset to any company. I'd advise you and anyone else move on if you are not fairly valued, regardless of gender.
This is generally a lousy experience. I've been places that want me to build a 'simple' (single entity) fullstack CRUD app (angular w/ASP.NET core backend.)
So what is it that they want prioritized? UI skills? Backend dev skills? I can do both, but I'm not going to spend a week making it enterprise-production-ready. The hiring engineers all have things they are looking for and these things are never well defined for you up-front, so no matter what you do (unless you spend way too much time on it,) someone may use it to disqualify you (without you ever knowing it was important.)
By the way, "Objectives" and "driven" are neither feminine or masculine, they're just words. Can you imagine if every word in the Oxford English Dictionary had a gender classification?
How did you come up upon this conclusion?
If it's not true, then I would agree that this is probably a waste of time.
There's at least one peer-reviewed study (Gaucher et al, 2011) which supports the idea that these words make jobs sound less appealing to women. Do you know of a study which challenges these findings?
And it would seem self-evident to me that not all of their 100-250 employees were hired at that 3/1000 rate, as number of applicants will grow over time due to increased brand recognition, increased spending on sourcing/outreach, etc.
Policies tend to just get stricter and stricter over time, especially with every bad experience they have (only takes one or two usually before these people go off the deep end trying to correct for it).
This seems like it'll bias to those who can quickly ramp up on a new code base in an unfamiliar environment. Which, in my mind, is a distinct skill from being able to contribute quality code to a code base you already know. I wonder if this is known and acknowledged bias or not at CircleCI.
Finding friends is hard. Finding a significant other is (ideally, anyway) a once-in-a-lifetime challenge. Finding people to hire is even harder: you're still trying to establish chemistry, only now there's a lot of money involved and the actors are big groups of people with complicated internal dynamics.
So I think what's wrong with this post is that CircleCI seem so self-satisfied. They seem to think they have a handle on this problem, when anyone who's been through a job interview in the last year or so will tell you it's a shitty, capricious, random experience even when they take you.
It does seem pretty strict, but without some sort of baseline this might be rather normal. I've never worked in recruiting or seen hard numbers on this sort of thing before so I have nothing to compare it to.
For CircleCI, their new candidate to screen is 25%. This is higher than the 17% average. Their screen stage looks considerably larger than I've seen at most companies: it's 4 different steps. From the 250 candidates, it looks like 7 go into an onsite. That's about 3% which is significantly below the 32% benchmark. Of those 7, 3 will receive offers (it's not mentioned how many accept those offers). So the onsite to offer is 42%, which is higher than the 31% average. We can't determine the offer acceptance from the data.
I think it's great they're looking at the funnel and conversion rates. Most recruiters aren't able to tell you these figures which is a red flag. The only way to improve those metrics is to know them first and then measure how changes impact each stage.
[1] https://medium.com/@jmtame/surprising-insights-from-talking-...