It could be adding a feature to a side project, starting something new, or contributing to open source. (Or, we have some stock ideas if people need inspiration.) Rather than asking them to solve a problem they had never heard of before or use a codebase they're not familiar with, I get so much more out of watching them work in their own environment.
I can ask them questions about why they're doing something, and they tend to have much more detailed answers because they've been thinking about it for weeks. They're solving a problem they care about in a codebase they know, which mimics how working with them will be a few months in.
We can talk about tradeoffs and design decision, and I get a real sense for how they think. Plus, I've found most people are so much more comfortable and excited.
(If you're interested, I wrote a blog post about all the ways I've designed how we interview, to give everyone the chance to show off the best version of themselves: https://blog.readme.com/designing-a-candidate-focused-interv... And of course, I'm hiring!)
For people like this, though, I’ve seen many use it as an opportunity to work on something they’ve been wanting to for a while and never had the time! A Hello World in a new language, merging in PRs on an old repo, writing a script to automate something or even a take home coding exam for another company.
One that we tend to do a lot is creating an image gallery using Flickr or Unsplashed's API. It's vague enough that people can take it in whatever direction they want, and simple enough that you can get pretty far in an hour.
I ask people to send me code they've written via email or links to projects and go through that.
I have a gigantic open-source portfolio[0]. I also go to great lengths to make sure that I’m a “known quantity” (like posting frequent exposition on fora like this one). I don’t feel that I have anything to hide. Everyone that has worked with me has found the experience enormously fulfilling.
It’s been quite jarring to see this ignored and treated with disbelief. That’s pretty much exactly the opposite reaction from the one that I would have, but I guess I’m out of touch with the way things are done, these days. I even talked to someone who told me about a friend that had invented some kind of machine learning candidate evaluation system, and relied on that, as opposed to human vetting.
One of my goals is to discourage people that wouldn’t want to work with me. I was a hiring manager for a fairly high-functioning, high-stakes team, and am quite aware of the value of making good hiring choices, which go well beyond simple expertise. Team dynamics are also very important.
I feel that getting a job as a result of misrepresentation is a big mistake, and a recipe for misery. So I don't hide my age, don’t pad my résumé, and make every effort to be as open as possible. If you don't want to work with me; fine. Let's not waste each others' time.
I have always sought to make it easy for people to figure me out. Since I live a fairly good life, and know my way around the playground, I feel that I'm a pretty good bet, but I have no problems if others think otherwise.
One of my favorite Twain quotes is "Let us so live that when we come to die, even the undertaker will be sorry."
I think I just threw up in my mouth a little.
Literally everyone? Ok :)
As such, I won't post stuff like you just did. It's uncouth and unprofessional. I consider this a professional environment. If you choose to look at it as a personal entertainment medium, then knock yourself out.
I'm a reformed troll. I used to be a right bastard. A great deal of my approach, these days, is an act of atonement. In fact, a great deal of my personal philosophy is based upon becoming a much more productive member of society, than I have been in the past.
I'm every bit as cynical as the next chap; more so, I'll bet. I regularly deal with some truly scary people, in my personal work on NPO stuff. I doubt there's a thing that you could say that would shock me.
It's also quite possible for very good people to exist in this world. I try to be one, myself. I know of many, many others. They are my heroes, and my models.
Please don't make the error of mistaking optimism, kindness, or politeness for weakness or stupidity. It's my experience that this can lead to...costly mistakes.
My post had nothing to do with any of this.
All I was pointing out was that being excessively boastful and claiming that everyone, without exception, finds working with you "enormously fulfilling" is just unbecoming. I feel like many will find that distasteful and you should leave other people to rate their interactions with you. I doubt there's a single person in the world who has a 100% success rate with regards to positive social/professional experiences with others...
I could have definitely worded it more politely and honestly if this account was tied to my personal name I probably would have worded it more politely, so you've got me there :)
I posted incorrectly. It is not 100% universal, but I'll bet it's a lot more than you might think. I've spent the last 40 years, getting along with some of the most difficult people on Earth. In fact, many of them are ones with a history of violence, and it's always a good idea to be polite in that kind of company.
Geeks are easy, after that.
That said, I suspect that many companies have been exploring this. It's pretty obvious that many places are already using AS (Artificial Stupidity) systems to triage applicants.
My guess is most people are covered by some sort of IP clause for their main work, and the most visible OSS projects are still not run by very many people, so most devs are not gonna be able to show you much.
Rather, the engineers on the team are usually building something new, and figuring out how to accomplish it is their responsibility. This way of interviewing mimics that a lot closer.
If someone can't explain to me a project they're working on, then they probably won't be able to do the same while we're working together. And that's just as important of a skill as writing code.
If I'm expecting interviewees (who are nervous!) to learn and understand my codebase/problem, why can't I do the same for theirs?
Or they tune out. Or they dismiss any variation as "wrong" rather than giving it a chance.
What happens in practice is the interviewer compares candidates answers to each other, not to his/her own way of solving it. It doesn't matter what whether the solution is "obvious" to the interviewer. If I hired someone who only did OK on this question and they ended up being a great engineer (and that happened multiple times) that lowers the bar for the test (or tells me it's not a very good test etc)
Whether it works out to be a harder or easier interview over time because of these changes probably depends on the individual interviewer.
I think that's _ideally_ what happens in practice. But as sibling comments mention, is that what happens in practice?
Interview fatigue is very real. Many interviewers are not positively incentivized to put in the substantial amount of effort to conduct a quality interview. It's something that is simply thrown on top of their usual workload. Come promotion time, there's no reward for doing it well. But, there is punishment for noticeably "screwing it up". Often, that means that interviewers put in the bare minimum effort with interviewing (as with other tasks) to not get punished.
I've seen competent leaders avoid this in the past by emphasizing that interviewing is the most highly leveraged thing you can do for your organization and your own career, because you are step-function increasing the effectiveness and capability of your own team if you do it right. But in order for that to happen, there needs to be a real culture of working as a team and not just a disembodied group of ICs: a culture which rewards increasing the effectiveness of the team (instead of merely focusing on ones own tasks) and doesn't just pay lip service to it. It's tricky. But I think the subthread GP's approach is a clever and creative kind of a lateral thinking that shows how to do just that.
do you (or others on team) ever make suggestions on how the code should be written? I think this would be an interesting way to pre-teach the hire on how you like things structured.
on edit - I guess since you ask them questions that implies making suggestions as well, but just wondering how it works.
We sometimes might make suggestions, but rarely for the sake of code quality. We'll once in a while ask for something unique just to get them out of a rut, and see if they can improvise. Otherwise, we're here to learn and listen, not teach :)
I signal a lot of things like testing by asking them. It gives room for the candidate to ask those sorts of questions back about how we do things.
Imagine you’re taking a class on a topic you don’t know. You can tell pretty quickly from how the instructor talks and explains something if they know what they’re talking about. It’s very similar! I’ve certainly interviewed people much more skilled than me in languages I’ve never used, and still got a pretty great sense of their expertise.
It’s an advantage we have as a small company! We can rate people based on whatever criteria gives them the best chance, and we can find a ton of great candidates that maybe wouldn’t make it through a traditional hiring interview at a larger company.
Like YC said, "Do things that don't scale."
If you asked me to work on something, it would be a toy project. Because I can't contribute to open source and if I build anything substantial it will be taken from me.
You shouldn't punish people in this situation. It's extremely common, maybe even the majority of coders outside California.
That being said, I've interviewed hundreds of candidates over the past 5 years, and not a single one has cited IP assignment as a hesitation.
of course they do, you're a knowledge worker for them. they need to own what you create.
a standard phrasing includes "related to the employer's business". see for yourself: https://www.lawinsider.com/clause/intellectual-property-assi...
"during the period of his employment relating to the business of the Company". If your contract doesn't have the words "relating to the business of the Company" then add those words in yourself, signing and dating the change and telling the company of your change.
you're not their slave you know. they don't get stuff you make in your free time. however, not replicating the company's business is a fair requirement, since they might share enough information with you to do just that. I think it's fair for them to get everything related to their business.
It's a great proxy for empathy and understanding. If you're curious, it means you want to learn the why behind why other people do things.
"Less certainty, more inquiry." --Erik Seidel
If we sense they're doing something they've rehearsed a bit too much, we might push them to do it a bit differently. "What if you tried to do this by _____", just to make sure they can think on their feet.
Honestly, though, the best candidates for us are the ones who wouldn't try to game it. They hear the prompt, and are just so excited to show and talk about a project they're building that it's not a problem!
By letting the interviewee pick, we're letting them decide which of their "superpowers" to show off. If they can explain why they chose to do something the way they're doing it, and show they understand the tradeoffs, that's good enough for me.
It sounds like you're putting that system of bias under the rug.
You can find all my past articles[1], and I'm happy to find you specific ones. I think, given that you can't switch away from algorithmic interviews, at least make the interviews more collaborative, define your expectations up front, and generally find ways to help the candidate show their strengths.
[0] https://hiringfor.tech/2020/02/10/false-positives-and-false-...
That means that regardless of the format, interviews are always going to be inherently unfair. Even the slightest hint of incompetence will nuke your chances. Assessing a person you’ve never met is just an inherently noisy process, and there’s always going to be false negatives. And almost every company in the world would much rather be safe than fair. You probably are having a bad day or misunderstood the question, but employers can’t afford to take that risk.
For the interviewees part, the process is just plain soul draining. You’re literally subjecting your life’s work to the harsh, quick and unfair judgement of people you’ve never met. You’re putting yourself in a position of extreme vulnerability, and it’s easy to feel like a piece of meat.
I don’t really see how twiddling with the format will change the fundamental dynamics. The only thing I can say is that both sides should have more understanding and sympathy for the other.
Candidates, don’t take any single rejection too personally. Remember the interviewer is a person, who also can be having a shitty day or make mistakes. Interviewers be kind and positive and give constructive feedback, even for the clear no’s.
If you really hate interviewing, then spend more time networking. Hiring personal contacts and recommendations is much less noisy/risky, and therefore an overall less adversarial process.
When I searched for anew job I had some slow time at my existing place and had all the time in the world to work on a take home assignment, while I could see the the candidate I recently interviewed had a hard time expanding their assignment beyond the basics due to lack of time.
Another variant of this challenge is; when you have multiple interviewers and some are positive some negative about the candidate. Generally companies will always side toward rejection unless there is a specific reason not to (for example, you need someone super technical but they struggled on the soft-skills side and you have the right team to help them grow in).
Surely, it depends on the situation of the specific employer. So the trick for a candidate would be to figure out if there exists a single argument (e.g. a certain skill) which makes it impossible to not hire you.
Coming back to the original question: You can be honest to the candidate; figure out what your organization is desperate for and tell them. Of course, this gives up some negotiation strength.
We provide a candidate a piece of production code with some slight tweaks made that will break it in (common) edge cases. Further, we give a test harness that already supplies some basic tests for which it will work. What we're looking for is to see if the candidate can grok the provided code (< 30 lines for a 2hr interview) and reason through ways it could break. Writing the tests insures that the candidate can _actually_ write the code and shows any areas they might be less focused on or miss.
Compared to our old procedure of writing the (provided) code snippet this has been a far more successful approach. In 2 hours I learn more about an applicant than through the entirety of the previous process. An added benefit: this workflow is representative of real work - you'll need to track down code you likely didn't write and implement new functionality, or fix something that's broken. Doing this in an interview is a near direct translation to the job (minus understanding the whole codebase).
I'd strongly recommend this unit testing based process and it can definitely be tweaked to fit your own scenario!
This. The stereotypical algo interview assumes a greenfield project, but there's almost always "legacy code" you'll need to grok first.
“Indeed, the ratio of time spent reading versus writing is well over 10 to 1.” ― Robert C. Martin
It's just bizarre to me.
Part of the deal is the 'humiliation' of being stuck following someone else's stylistic/design/naming-scheme choices. Another aspect is the curse of knowledge, leading to useless documentation. Plus, it's easy to underestimate the amount of effort required to reinvent the wheel: all the necessary complexity looks like incidental complexity at first glance. "Why should I be forced to learn all this gobblety-guck when I should just be able to re-implement from scratch in an afternoon? [... three weeks later] Oh, now I see why."
Do you know what I do when I see a problem for the first time that I don't immediately know the answer to? I Google it. Because A) it's faster B) many minds greater than mine might have already come up with an optimal solution that I'm unlikely to self-produce in 2 hours.
What if a developer hasn't been deep in debugging production code for a while? Those edge cases might not be top of mind.
Maybe they're doing documentation or planning or writing Jira tickets and remembering all the ways to nullcheck a JavaScript variable isn't their current life goal. Instead they might try running the code locally or write some tests for it or research to see what others have done in similar patterns.
Dreaming up magical solutions while under the stressful glare of people judging your life's work may sound reasonable to you because presumably it matches up to some internal signals you value in each other, but every workplace is different -- different codebases, different problems, edge cases prevalent in one may not be prevalent in another.
You're better off doing a traditional Q&A about their experience and a take home mini-project (or asking them about theirs). If they don't want to spend time on that, then and only them give them some timed test you feel is applicable.
Of course there's Q&A and their reasoning/explanation is just as valuable as the code they right. This is just the technical portion of the interview. However, I think we've really hit the sweet spot with this approach.
Take logging. Super important - I spend a lot of time evangelizing logging to younger devs and how to do it well. But filtering? How? In what way? For what values? I don't have context but the expectation I should be able to filter for specific things you have secret preferences for strikes me as a signal that's not clear or universal.
Checking HTTP codes. In what context? There are plenty of times when it's not helpful -- for example there are many page not founds that DON'T return as 404 (even tho they should). So this a priori belief that all devs should automatically have this built-in preference for checking HTTP codes as some universal signal of quality in programming is, I think, a larger assumption than you might realize.
I don't think there is a perfect way to interview, but I do think if you're going to quiz developers it helps to give them some advance warning of what areas you prefer to focus on so it's not such a random, out-of-left field line of questioning.
You may not consider your questions abnormal -- but that's the problem, neither do the people who ask obscure algorithm questions! It's very hard to validate your assumptions.
You also don't give a second chance if they fail this one assignment.
Provide candidates a small sample "project" of 50-100 lines, a Flask webapp perhaps. Work with them to get it running on whatever interview machine they're using. Have them extend it with some fairly trivial feature, around FizzBuzz complexity but with a bit of a design component. These questions should be carefully designed and standardized across interviews.
Key point: completely strip out identifying information from their solution and pass it to someone else for grading. The person "giving" the question is a proctor, not a judge, and they're officially on the candidate's side. Possibly add a side dish of talky conceptual, design, and behavioral questions, with as much blinding in the assessment of the answers as possible, though that's harder if it's a dialogue... Hmm.
A good interview is fun. You get a nice bite-sized problem to solve with someone to bounce ideas off of, and maybe learn something in the process. I'd suggest almost a game level-design approach to coding problems. Make interviewing fun again.
It had a few benefits:
First, this is much closer to the type of work that we'd ask a new employee to do, as compared to the typical industry interview question: "did you pay attention in computer science? You did? Now prove it by solving this problem that's a thinly-veiled algorithm exercise. Using a computer? Oh heck no, you're going to use a marker! There's no trick to it, we don't ask interview questions that have tricks, but if it leaks externally, it's 100% useless and we cannot ever ask it again"
On that topic, who cares if the premise leaks? If the code somehow got out, we'd just write another one.
We get to hear them talk, out loud, about their expectations and how the app's organization differs from them. This isn't interesting by itself, but the candidate gets a chance to articulate their experiences and how well they notice patterns. For more senior engineers, you get to see how quickly they can pattern match and explain the structure of the client and server, and explain how it could evolve under different circumstances. "You're in a startup that just got a ton of VC based on this hackathon project. Your friends hire you and give you this codebase. How do you spend your next 2 months?"
Also, you'd be surprised how many people disqualify themselves on attitude. People will read sections of the toy app and say out loud, "who the hell wrote this? This is terrible." And you'll ask them, "What do you mean by that?" and they'll rant about some section of it, and you're like, "wow, I really don't want to work with this person." If these engineers can't even keep it together during an EXERCISE whose PREMISE is that the code doesn't work well and can be improved, it's a negative signal that they're going to demonstrate any empathy when they're working with you or any of your colleagues over the years.
As an interviewee, get a strict proctor that doesn't want to give out information? Well you're not going to appear as competent as the interviewee who got the proctor that wanted the interview to be fun
I would say it's still an improvement over leaving the whole thing up to the whims it the interviewer.
My favorite interview question I ever had was to design an elevator system. There's no right answer. It wasn't even mostly code, it was mostly talking. But every so often, they'd say "write that as a method" and let me imagine whatever api I wanted. I had so much fun doing that.
It would be an interesting idea just to have a candidate go to the site and screenshare to watch them do some coding.
All excuses for why this won't work can be answered with: then make a better test.
As a species, across all cultures, we tell ourselves that the human we're after -- the student, the employee, the promotee -- can't possibly be selected by test alone.
When in fact, if we can't write a test to select the qualified humans, then either we're too lazy to write one, or, more likely, we actually want to leave plenty of room for human bias to do the actual selecting.
And this is ok because we have a special power: we can judge the value of every human, and its future likelihood for success, with a single conversation. If we weren't in a tech company, we could make a very good living reading palms.
We never hire people who don't pass our wonderfully fuzzy exams, so we have no evidence that we're selecting the best people or not. No worries, though, our palm reading is very, very accurate.
The way we look at it is like this. We make a test no one can pass. We always have one more question or one more "level" that there's not enough time for. Because when all candidates fail, we have to fall back on our palm reading, which is just how we like it.
Power and privilege are precious resources to us. We give them out to those most likely to reciprocate. That's what we're poking for with our "culture fit".
In the future, students of our culture will look back on our hiring practices and say, "It was illegal to hire based on race, age, sex, and a million other things, but not beauty??? They didn't start with beauty? And they never realized that beauty needed to be in the mix? I don't understand."
But we understand. It makes perfect sense.
The same reasons whiteboard interviews are problematic will apply to any test done at scale.
if we can't write a test to select the qualified humans, then either we're too lazy to write one, or, more likely, we actually want to leave plenty of room for human bias to do the actual selecting.
Creating such an exam is an unsolved problem.
Why can't we write a test to see if she can code in Python at an appropriate level of expertise without mixing it in with algorithm gotchas? What are the algorithms she needs to know? Why can't we provide her with a list of those algorithms and show her how we will test for mastery of them?
>Why can't we provide her with a list of those algorithms and show her how we will test for mastery of them?
That's exactly what is done, it's called leetcode.com, recruiters send you a link to it and tell you to study. I don't see how anything you're describing isn't covered by the existing whiteboard interview dynamic.
leetcode is an open-ended ocean of knowledge you have study, it's not a test.
So it's all standard questions that have answers you can study for ahead of time.
leetcode as used by FAANG is a hurdle that you can't ever be sure you actually cleared. it's not a test that returns a pass fail score.
My ideal interview questions are as follows:
* No trick question. Reaching a solution shouldn't involve obscure knowledge or an "aha!" moment. * Only basic data structures and algorithms, the kind that you will realistically use in most day jobs. * Start with a very basic problem, and ask follow up questions with increasing complexity. * Extend the problem in different dimension to probe the candidate in different axis. For example, ask the candidate to design an API for the code, or to run it at massive scale.
It's hard to come up with problems that cover all these points, but you can get close. I find these kind of questions allow me to assess candidate without making the process unfair or overly stressful for them.
Technical talks are never about coding, but more a conversation about previous project and perhaps a little side track on how to tackle large projects.
The style of interviews discused here on HN show an alarmingly low level of trust in the people you want to hire. That’s a bad way to start a relationship.
But there's also a bunch of other dynamics going on. 'Geek machismo' is a very non-trivial one. Other's are akin to hazing. One friend of mine described one value of very high bar hiring is that people in the bubble can (go back to) assume that the colleague is smart, etc. rather than assuming people are stupid/incompetent until proven otherwise--and that can be a good thing from the culture/sociology standpoint.
I suspect there are technical tests in developer interviews because there can be rather than because they're actually a useful way to appraise someone.
The incentive to bullshit your way in the door is way higher than most other fields. Imagine if you could get hired as a doctor, without having to graduate high school.
It also seems weird to lie in your CV, what are you going to do if you actually get the job? You’d fail horribly and get fired.
My point is that in some parts of the world you can actually trust resumes. If companies start making outrages demands, like 15 years of experience in a 10 year old technology, then of cause desperate people will lie.
And I don't think people typically lie on a resume. I do think there's a category of people that makes things look nicer than they are. I do think there's another category of people that just have a different perception of "mastery" than I do.
I have seen small programming task presented to candidates at a previous employeer, after I was hired, but that sort of failed. All the candidate actually submitted perfectly good solution and the process just reverted to just talking to each candidate in a relaxing setting.
Edit: I actually once interviewed for a job that required a personality test, because HR need to justify their existance I assume. I will never apply at a company which uses personality tests again. Not only was it wrong, it’s also kinda traumatic. But this was after actually getting the job, the test was sort of a “before we actually sign, please take this test”
So my starting salary was close to what typical software engineers earn as a seasoned senior, and then I got to twice that within a few years. That is why I care about technical interviews.
Writing some algorithm No why should I? If I need an algorithm that must be fast I will look up a peace of code that is used and tested by other people.
No I will not live google in front of you watching me. How is this a real world scenario? How came up with this?
Like stupid questions like: How many log entries should a service have each hour? Yes this was a real question in a job interview.
IT Job interviews are broken in so many ways.
I refuse to live code or have this you now have X minutes to solve a problem stuff. Again this is not how I work and also I have never seen people work like that in the real world. Only in shitty teams with shitty organization and where they were thinking that they were agile because the sprint ends in 5 minutes...
Here is the only way of interview I tolerate these days: 1) (optional) Have a call with HR
2) Talk with people from the team in an open way. Talk about real challenges and how the applicant can help in the challenges of the team. Check if the applicant will fit your team! Good team chemistry eats technical knowledge by 1000x. If you only take people because they know some algorithm you will most likely end up with something in Germany they call "fachidioten". They will solve the problem on a technical level but will bring with them 1000 other social, product and long term problems with them.
3) If you still not convinced that this person is good enough for your team then. Give that person a small challenge he/she can do at at any time she/he wants.
4) Let the applicant send you the solution and if you think it is okay then talk about the solution of the applicant and see how he/she reacts to criticism. Again a team that has good social skills and likes each other will eat every team were you have rock stars that think they are the best.
I agree that the in person time pressured performative problem solving & coding engineering interview process is often a poor approximation of the work.
If at least two people who worked closely with someone like them enough to give a referral, they are probably worth hiring.
I had quite a few “aha” moments where I was “ah wait, I totally didn’t do that, have I?”. Didn’t get the job, however neither did I like them.
We had one case were you could clearly see that the person had no clue what he is talking about it.
Its quit easy to see.
Just ask them to change something small.
volume if you need to hire a lot of people? volume of applicants?
I have hired dozens of engineers and have never failed do find great hires.
What I mostly do is just a conversation, which I call “geek conversation”, just like when you meet people at meet-ups or tech conventions.
With a simple talk I can first make the person relax, this allows me to have a great impression about her/his personality, passions, interests, personal roadmap, attitude towards problem solving and etc. Whenever we ponder on a more technical subject I throw some questions that might help me measure their technical level a bit, but what I care to measure is mostly awareness on things, nowadays software engineers need to be, above all, great at researching, for most roles they must have a broader overall knowledge and be able to creatively join the dots with a ton of pragmatism. I don’t care if they are able to implementing Dijkstra's or to modify a Trie in front of me, although having awareness of what problem these things solve is a good indication of some seniority.
These interviews take a lot more from me, I have to properly learn about the candidate and the role before I enter the interview and that requires people skills and real interest in the product, and of course I am not able to interview 40 candidates to find just “the best one” (which is ridiculous and something I actually don’t believe actually exists).
Yes there are roles which require the extreme algorithm coding skill, but those are a minority and it is unfortunate if your company is treating every candidate as if they were going to do that.
Knowledge is usually the easiest problem to fix. It’s a google search away. All the bad hires I’ve seen have had masters or phds and the problem has been poor work ethic, focus, and practical application of knowledge towards solving business problems. Interviews, in my opinion, can easily test for basic dev knowledge but should focus more on the way the candidate will conduct themself as an employee.
My "FizzBuzz" coding question that I ask on phone screens is "Write a function that takes a string and returns a list containing the most frequent character." It has a few clarifications that are needed and I expect the person to ask about (What if there is no most frequent character like in an empty string (return an empty list), multiple characters equally frequent (each equally frequent character should be in the returned list once), etc) and has some basic logic requirements.
I am continuously astounded when people can't do this. I encourage them to use the language they're most comfortable with. I give them time to think about the problem, encourage them to ask questions etc. If they seem stuck I'll ask them to walk me through what they're thinking about, etc.
I've had multiple people with master's degrees in CS or with years of professional programming experience get stumped by this problem and it just makes zero sense to me. I took undergraduate CS courses and this would've been an easy problem in those classes. I was a TA in an undergraduate intro to programming course and I would've expected all of my students to be able to solve this problem in similar circumstances.
It seems to me like someone with a Master's degree in math has come in and I'm asking them to "solve for x" in the kind of equation you might give to an eighth grader and they're unable to do it.
Unfortunately people tire me very quickly, I have a limited budget of people time per-day and interviews frequently blow that budget early in the day yet continue into the afternoon (talking particulary onsites for bay area companies here).
I would say 9/10 I manage fine but there have been occasions especially when I have been in between jobs and taking lots of interviews when I have just bombed because a particular company stacked all the code heavy interviews late into the panel.
Separately from that speaking as a hiring manager for several roles I don't think coding on demand is a good metric. I don't expect people to do it once hired so it's not useful as a measurement of their fitness for the role either.
Design heavy interview questions on the other hand I find very useful. They generally are less specific and speak more to experience, knowledge and taste. Things that are very relevant in an engineers real-world productivity. They are very punishing vs code on-demand interviews for new grads though so if your pipeline expects to take a lot of new grads you may need a different approach.
However in my experience questions like "How would you architect a replacement of Amazons S3 service with similar durability" or "Given a smart grid reporting metrics from every home in the US how would you store this data for analysing both long-term trends and intra-day anomalies" yield by far the best signal to noise ratio.
This isn't obvious at all - I've interviewed many people that have faked their GitHubs or were just completely incompetent at software development despite their amazing GH, OSS profiles or CVs.
People lie, a lot. I'm sorry, but having "a profile" isn't a useful signal for an interviewer :(
> However in my experience questions like "How would you architect a replacement of Amazons S3 service with similar durability" or "Given a smart grid reporting metrics from every home in the US how would you store this data for analysing both long-term trends and intra-day anomalies" yield by far the best signal to noise ratio.
Agreed, in my experience these questions give by far the best signal for a good coworker in general. Sadly they're not appropriate to interview junior developers.
So yeah, it might not be useful for recruiters/sources that don't understand the technical side deep enough to evaluate but it 100% sure be good signal for an interviewer if they know the fields and projects in question you are hiring for (they probably should...).
(I was also a TA in a course that did these sorts of trivial challenges, and going through that experience really made me take a hard look at how I perceived those 1st year students)
But the thing about interviews is that you always want to heavily bias towards false negatives. A single bad hire is just astronomically costly. I think it’s fair to say that many smaller companies have literally been bankrupted by one incompetent person. When it comes to engineering positions, this is especially true.
I’d rather nuke 20 good candidates having a bad day, then risk letting one bad engineer through the door. That’s why I’m skeptical that the interview process can even be “fixed”.
Fundamentally it’s a mis-alignment of incentives. Employers have very strong to rule out candidates at even the slightest hint of incompetence. But for the interviewee, being judged so quickly, harshly and unfairly on a such a personal level makes them feel vulnerable and hurt.
There's only one (not counting ties), so you want a single-item list? Or if 's' is the most frequent, because there are 5 of them, do you want [s, s, s, s, s]? Or since the list just needs to contain the most frequent character, seems like [r, s, t] would be correct as well. Now that I think about it, just returning the original string would have to contain the most frequent character. Also, do capital and lowercase of the same letter count as different characters?
Just seems like a rather unclear directive.
I think of it as the FizzBuzz version of getting requirements for the thing you're supposed to be doing.
I would expect a skilled programmer to be be able to make intelligent inferences about the intended behavior in the face of ambiguous problem statements. FWIW, I would probably explain my reasoning as I went -- would you interrupt a candidate to say you'd rather see a different behavior (ex: my prototype would return a list with an empty string, but maybe you'd rather an empty list), or just silently judge them? Also, it look me about 2 minutes to produce a reasonable prototype -- maybe if the challenge were harder I'd expect someone to ask more questions up front, but in this case it's trivial to iterate.
If the only thing you are testing for is the instinct to clarify unclear requirements, you'll surprise people who are used to the leetcode-style, thereby getting false negatives. It's just as much a trick question then, just in a different dimension.
Expect professionalism from both parts.
First, I would feel like if I have to ask questions if you then I’m missing something obvious you want me to be able to deduce. And if I take too long I’d fail your test.
So I’d jump in merrily and work up a quick and dirty version before even realizing that the output has ambiguities. Hopefully at that point I’d ask how you want them resolved, but maybe I decide you want to see me to make my own decisions and I’d come up with my own way to handle them.
I had just received offers from two FAANGs, and I was flown into the Bay Area to interview at a third. One of the interviews was mostly behavioral, with 20 mins at the end for an algorithms problem that I had done before on LeetCode. I did so badly that after 20 minutes the only thing on the whiteboard was the problem statement. I still remember the interviewer taking a picture of it, and me kind of cringing because of how bad it looked. I didn't get an offer, unsurprisingly.
Because they disqualified themselves from hiring me. What I remember from that phone screen was getting mentally _TIRED_, sighing, and then lying about not being able to do it.
Seriously, don't fucking ask me to describe code or algorithms over the phone. The number of times I've had someone do some variation of this over my lifetime is such that I don't want to have you as a co-worker or a manager.
Do you know why I have this attitude? Because the number of times I've seen people actively be wrong in what they're looking for, as in technically wrong and therefore I failed, is enough that I've decided if you think this is the right way to go about hiring, you're doing it wrong and I don't want to be in that hellhole.
I imagine if I was hiring someone to play in a band or sing in a choir it would probably be a good idea to ask them to play or sing a bit before extending an offer. If I was hiring someone to work at plumbing company, I'd probably ask them a "How would you fix..." type question or two.
Saying that these questions are forbidden just because you are sometimes tired of doing them strikes me as unreasonable. While, of course, you get to decide what signals would make a company a bad fit for you - e.g. if I asked you my question you might think my company would be a bad fit for you, the interviewer decides which answers are a bad fit for the company. I would definitely take an answer like this as indicating a bad fit for the company.
This line from your comment is interesting to me though: "I've seen people actively be wrong in what they're looking for, as in technically wrong and therefore I failed". I frequently ask questions where I expect the candidate to want to clarifications or challenge what I'm saying. That's part of my job, and I'm trying to test for it the best way I know how. If you're happy with your job interviewing performance, then more power to you, but if you aren't - you may consider changing how you think about instances where the interviewer is wrong. Perhaps they are presenting opportunities for you to engage with them in a productive way, and even if they aren't doing so intentionally, maybe they are doing so unintentionally.
You know what's unreasonable? Interviewing a bricklayer and asking them to describe how they lay bricks over the phone. And this is why your comparison to singing is hugely flawed. The phone is literally designed for carrying and hearing voices, not for more complex things.
You act as if it's literally impossible to figure out if someone can code unless you're trying to do it over the phone.
> This line from your comment is interesting to me though: "I've seen people actively be wrong in what they're looking for, as in technically wrong and therefore I failed". I frequently ask questions where I expect the candidate to want to clarifications or challenge what I'm saying.
I once had someone ask me what _type_ was returned from a controller in asp.net MVC, and when I gave the technically correct response this person told me I was wrong, a view is returned, then immediately ended the call. The number of times I've seen stuff like this happen is way too high.
But more than that, I want to address the attitude in that last point. This is a _HUGE_ problem with interviewers. They often come to conclusions they ought not come to based upon the circumstances. And the defenseless interviewee's are left trying to suss out hidden requirements like "expects them to ask clarifying questions". Interviewee's come in and have to try and navigate these byzantine, hidden requirements. And the question is, why do interviewers have hidden requirements like this? Because they know if they're _HONEST_ and communicate what they're looking for, the candidate will give it to them. That by itself should be a wake up call that interview's are hostile and un-real.
I'm just in a position in my career that I have the leverage to refuse to play those games. If you do something like that, you fundamentally have no clue how to actually hire people, and I refuse to bring any value to you or your company.
But asking a candidate for 30-50 lines of code is completely normal.
The fact that the internet exists and the interviewer could have used an online utility doesn't change my responsibility in that equation, that's just nuts.
For senior engineers I would say it's all about design/architectural problems and talking generally about concepts and taste to gauge fit.
Mid-level engineers I think the take-home is king. A scoped problem, 1-2 hours at most that can be followed up with an interview talking about an extending their solution is pretty bulletproof. You get to understand what tools that are comfortable with, practices around source-control, testing etc that you would expect from an experienced but not overly senior engineer.
Finally for entry level candidates I don't really know. Personally I hire entry level people on gut feel. Algorithms and datastructures just tell me if they a) went to university and b) paid attention. Seeing as I did neither of those things and still write IMO decent software that seems like a hypocritical measure. So I stick with how I feel about them, their level of enthusiasm, how well they researched the position, what stuff they have played with etc.
At the end of it the biggest burning question they had was "where are the unit tests". I was like ... you see this circle here with this label inside it? This was critical to the entire system and therefore had unit tests surrounding it.
didn't get the job, they seemed kind of annoyed that I didn't get into great detail about things like unit tests and instead talked mostly about the architectural problems I solved.
I wish companies actually valued that level of knowledge.
The company itself wasn't actually solving complex problems and therefore the developers involved couldn't actually understand the level of complexity I was able to deal with.
I get the feeling a lot of developers believe dealing with state in a react app is a complex problem, but I've been doing this stuff for 25+ years and left that behind as complex long ago. I know that's probably going to come across as arrogant, but it's how I feel about the entire thing. I see people all the time squabbling online over very specific source code related things as if they're important.
It was a VPS automated deployment system that could deploy both windows and linux to various datacenters around the world while also installing some 50+ pieces of software (as configured).
Hire them, give them a week to catch up, and their off; or wait months to find a perfect match? Any developer with a few years experience knows how to pick up the next piece of technology, the same way everybody else does: read docs, search blogs and forums, ask StackOverflow, etc.
Now if all you find is Perl guy, that is another problem.
Some people can. Some people can't.
The secret to a good hire is probably to exclude people from doing the interviewing who are essentially poor judges of character.
In other words let's stop trying to auto filter the candidates and instead let's filter the interviewers. Pick good interviewers.
Let those firms that play sunk cost Frat games get the sort of people who will work all hours for next to no pay. Those people always burn out quickly anyway. Instead treat the interviewees' time with great respect and you will get better long term people anyway.
I suspect saying that interviews are an hour's chat and then a decision will get you a set of outstanding interviewees who are just too busy to be bothered with anything else. And they will stand out from the crowd even on paper.
When hiring what works for me is looking at past experience. If someone says they are a cpp performance guy, that creates an expectation of what they should be able to chat about. If you're making it up, you will run out of stuff to say, and I'm patient enough to let people keep talking. Hopefully I'll learn something too, it happens often with top pros. But basically if you're experienced in a field it should be hard to come up with anything where you draw a total blank. If that's the case you're pretty good in my book, the main differentiator after that is whether you've done exactly what the firm needs or just something similar.
With entry level people it's hard. That's why they typically make less money, it's an Akerlof lemon cut due to the being a lot of people who just aren't going to be good at it. This isn't even necessarily a coding thing, but general life maturity. We had a guy who couldn't wake up on time to get to work, and he couldn't phone in when he was late either. I reckon there's no way to know about this kind of attitude problem until you hire them. This is also why you can expect a relatively high jump in pay after your first few years, it's proven that you aren't one of the lemons. The distribution is a large mass of hard to differentiate professionals, plus some duds of near zero value.
So for entry level, your best bet is simply finding someone you think is smart. You'll end up falling back to the same thing as everyone else who is hiring: the prestige of the uni they went to, and some pot luck questions to make sure they picked up something. However that's really not enough to know what you want to know, which is whether they are able and willing to learn what you need.
Of course, it doesn't mean the opposite holds. They could be interested in something but not end up being interested in the job.
If there's one thing I don't want to do, it's work by assignment. I'm not a school kid, I'm a professional software developer. I'll tell you what needs to be done. You can inform me of the goals you have for the company, and I'll make sure that happens.
What I'm saying is that despite all my successes, you would consider me a 'dud'.
45 minutes - do code review of this extremely buggy code like you'd review a coworker's code
45 minutes - presentation on a technical topic to one or more team members
45 minutes - behavioral. Let's talk about failures, conflicts in your career. Who are your role models? What do they do well? Leadership abilities, career progression
45 minutes - troubleshooting. Here's a docker container. Make the service inside it work
Also number 1 (writing simple API matching a spec) - this doesn't require prep if it's using language that I am using currently / have been using in my daily job for a long time.
But I have found often been required to write code in languages that are on my CV but I haven't used them in 5+ years.
Also talking about past failures, career history. I can barely remember in detail 2-3 years so I would need to spend a little time to just be prepared about a random very specific question about a project I worked on 5 years ago that might come up during interview because it would silly if I didn't remember (which I wouldn't if you ask me randomly about something I did 5 years ago).
Small things like these add up, obviously the more you want to get a certain job the more time you would spend preparing, it's only natural. If you are only looking around you might just wing it since you don't care much.
I am not criticising your system, it seems pretty sensible actually btw.
Am interested to learn more. Are there open positions?
If you were a team lead tomorrow, how would you go about hiring your coworkers?
Honestly, that should tell you everything you need to know about:
a) What kind of technical skills they value (i.e. will work to improve in themselves if necessary)
b) Their interaction style with coworkers.
Sure, this interview can probably be gamed, but IMO a lot less than your typical "reverse a linked list" style interview.
That works best for programmers with entry to mid-level. In Senior level you need to focus more on problem solving.
Lots of people say this however I've never seen anyone come up with evidence that their team actually is excellent.
You want someone that can easily talk about the pros & cons of using different data structures or algorithms to solve problems. Who can move up & down layers of abstraction without getting confused & lost. Who can hide complexity to make clean, readable, maintainable code.
Many people, especially junior/less experienced engineers, have only memorized solutions to problems. They don't have a strong toolbox of solutions and the experience to know how to apply them (and evaluate the options).
A good problem solver is methodical - they don't just guess at random data structures, they identify what the requirements are of the problem and then choose the appropriate data structure (or compose one) which meets those requirements. They'll have a process, which may be different than your process, that they use to break down problems into smaller, solvable pieces.
It sounds to me like we are forgetting all the other important things.
Like how easy it will be to replace that data structure if maybe the requirements changes? Do we really need to optimize this thing fully now? Or maybe for the next 5 years to simple solution will be more then enough? And so many other questions.
A lot people need to think longer then 1 minute to come up with a good solution or maybe you need to try or maybe you need to learn something?
Isn't that what we actually do? How often did we think that framework X is the best but after a while it was not that great? Don't we need to play around with an API to understand it better? Or let me read about that and tomorrow we can talk about because last time I was doing something with binary trees was in university because in the real world I don't need it that often because yeah we have other important things to do.
There's like 5 basic data structures. Choosing the right one based on desired functionality and explaining why you picked it, that's just table stakes.
I'm not paying you six figures to write software professionally if you can't tell me when to choose a linked list over an array, and apply that knowledge to solve a problem.
The right data structure does not only depend on performance. I can write you the most performant code in the world if you want to but it will take 5 years. Oh we only have 1 month? then sorry for now an Array will do.
In most cases you pay someone for the skill to learn new things fast and stay motivated. Why? Because in good companies you want to hold on to the people for a long time and this people need to educate themselves. Why? Because even the basics depend on the context they are in and languages, frameworks and runtime and other stuff will change over time. 20 Years ago memory space was a problem? Do we even think about memory now? Not really unless your working on some embedded systems.
1. Those are the building blocks of software, and
2. They don't stop being useful when you move from domain A to domain B
What I said was that "movement" was a building block of life and your response was "no, cause if you do polka-STYLE movement while trying to lay bricks, you'll FAIL".
Right, but what they both have in common is movement. The fact that you felt the need to add style is exactly why I'm correct.
uh, not really.
In my world majority of performance issues comes from N+1 queries, I don't need people who can do crazy things with trees. I need people who write testable code, actually write tests, care about security and take responsibility for their code/refactor.
Of course you can argue that N+1 query fits algorithms part, but very often algos mean leetcode
But, since "you guys" always talk about data structures then I'd want to ask - how often do you use something fancier than literal basics?
I've had plenty of candidates who don't know what data structure to use in that example, and they just start randomly listing things...tree, hash, linked list, etc until they stumble on the correct answer.
Most articles complain about the status quo but don't offer any practical advice. When they do, it is some approach that may not be an option on most cases - e.g. 40 minute interview followed by a 90 day probation period.
Live coding interview can be pretty tough for candidates, but we try to make a lit bit less difficult by:
- Giving them time and resources to prepare.
- Letting them use their preferred language and IDE/editor.
But since you have no control over format or time, maybe try to:
- Use reasonable problems - to assess skills people will actually need on the job. In our case it usually boils down to using comodity structures like a dictionary or map to do simple tasks efficiently.
- Use problems that start simple but can be extended. This is good because when candidates finish the first part, they get a boost in confidence. It is also nice because if they freeze, you can help them finish the current "level" and still have material to assess other skills.
- Set them up for success during interview:
- Reserve the first 5~10 minutes for introductions, ice-breakers or questions about the interview.
- Reserve the last 5~10 minutes for the candidate to ask any question about the company or the hiring process.
- Make clear that is OK to ask any question.
- Help them on small blockers.
- Be friendly.
Some links/references:
[1]: https://lawler.io/scrivings/erics-guide-to-hiring-software-d...
[2]: https://medium.com/@alexallain/what-ive-learned-interviewing...
[3]: https://www.holloway.com/g/technical-recruiting-hiring/about
Rather than saying how you are constrained, what freedom do you have?
Of course, this requires knowing what skills & knowledge are needed for the role - you may need to review the job description or push on the hiring manager to quantify them.
When asking problem solving or coding questions, work with the candidate as if you were two teammates solving the problem together - don't be adversarial, don't play "gotcha!", don't try to show off or prove yourself smarter.
If you have an awesome candidate, they will be smarter/more skilled/more experienced/more knowledgeable than you in some technical areas. That's a good thing!
- algorithms
- experience with a specific technology
- getting things done
- communication in "not being a dick" sense
- generally "being smart"
I want a person to score "B" in all of them and "A" in one.
It's important to realize that you don't need all A's: it's good if you manage to find an all-round A-student, but it's rarely possible and takes huge amount of time.
One "C" may be allowed, but then I'd probably want more than one "A".
So I keep my questions simple. For algorithms, for example, I ask to write a bubble sort. If a person can do it - it's a firm "B".
1. Hire based on the resume, gut check on the interview. Is it plausible that they played the role that they said they did? If you were a major contributor to an RPC framework in C, but you can't walk a linked list... Maybe your resume is too inflated. This means that the questions are basic because their purpose is not to rank candidates but to ensure the validity of the resume.
2. Focus on reading over writing. I would rather a candidate who can read code someone else wrote, articulate what it does, and maybe plan for how to extend it than someone who can pretend to invent something new.
#2 I have mixed feelings about. it may make sense if you’re after an expert on that language but may evaluate wrongly candidates that would be great reading code once they get just a bit more familiar with and get the overall architecture. In many companies, you’d be forbidden from sharing actual code with non employees, so the excerpt shown in the interview would have to be purposely crafted and loose value. I propose a more interesting exercise could be to examine a random piece of opensource that Both the interviewer and interviewee are unfamiliar with and trying to get a good discussion from it, but it’s a risky move leaving the interviewer exposed.
I generally take a look at their resume and then do some research about the tools they've used in advance. Then, during the interview, I ask about what they like/don't like/find interesting about those tools. The goal is absolutely not to gotcha them, but instead to find out what they're interested in in that space. If I ask a question that it becomes clear they've lied/fabricated about on their resume, I say something to the effect of "No worries" and change the subject.
Depending on the role, you need more info than just what languages/frameworks they've used. For more senior roles, or roles that involve architecture/cloud functionality, I'd ask about how they've built systems in the past. If they call out AWS, I ask about what resources they've used, how, and why. If you've written down DynamoDB but cannot speak intelligently about access patterns or secondary indexing, it's kinda clear that you just used a system someone else defined. Whether or not that's a problem depends on what role they're applying for. If they can speak intelligently about how they got to a specific DynamoDB structure, they probably are being honest enough about their experience. Note, it needs to be clear that the candidate is not speaking in the abstract, but about things they've actually done. Googling stuff is easy, finding the weird parts of tech in practice is hard.
Ultimately I want them to feel comfortable enough to get chatty about development. Usually I find out enough about their skills while they're chatting - I think most would be surprised to find how clearly you can understand a person's abilities without directly asking about them. You just kinda have to spend some time up front learning pros/cons/common pitfalls of the tech on their resume.
There's always a ton of posts, but I have never had a pleasant or unbiased interview experience in my life.
It shows to the interviewer how a given person works in a programming job, how they researched solutions to the problems they were faced with in the past etc.; as opposed to the usual exercises that have nothing to do with the job and won't really show you how well they will perform.
It's also less confrontational which should put the interviewee at more ease.
So what I started was a local library program (pre-covid) that I would attend before / after work to be around people who are doing career switch to coding.
Then when students learn enough and have strong foundations that I look for, I recruit them to work on open source side projects that benefit new students who are learning.
For the students who have shown the maturity to build and maintain production features, I refer them to the company I work at and they interview and (hopefully get in).
In the past 2 years, I've referred 5 students successfully and they're all doing pretty really well (I secretly stalk their internal code contributions).
I hope that one day, I could have enough reputation at my company to hire engineers based on their open source contributions. I also hope that the stock market doesn't crash next year because I plan to use my RSUs to start a fund to hire junior engineers students so they don't worry about medical benefits while they contribute to open source projects and have a hard time finding a job.
If you mentor a student as they are learning, you can instill good engineering practices: like taking the time to communicate and write good documentation, write code and comments to make life easy for the next engineer to take over, how to handle tradeoffs between time and quality, etc. These are critical engineering skills that are very hard to test in 1 hours but are very effective if mentored early on.
I find it astonishing how many developers absolutely cannot think through a tree model. I deal with tree models absolutely every day in programming, management, and personal concerns. For example: the DOM, file system, company/employee organization, outlines, security models, business/customer needs, and so forth. All these things are tree graphs and I encounter many of these EVERYDAY.
Many developers simply cannot explain walking from one point on the tree to another point. I am not even talking about code. Even in casual conversation I am astonished at how many times I have encountered people like lawyers, QA, bosses, and others who can engage this sort of logic while some of my developer peers are just lost on the subject.
That is absolutely something I tested candidates for as a filter and will continue to do so. If a person even talking through such a real world common form of abstract logic how can I expect them to write code? It’s like they expect some magic wand to simply provide the logic for them and just sprinkle on a little bit of syntax.
1. Tell me briefly about your professional development, including challenges you have faced, mistakes you have made, how you overcame them and what you have learned. Have there been any critical mentors or resources? What motivated you in each position and how did that lead to your current application.
Opens a floodgate of possibilities for discussion, generally gives a sense of career pattern or path, potentially personality.
2. What are you learning right now and why?
Screens for curiosity and/or goal orientation.
3. Choose any recent technical or business idea you have had and explain why you considered it to be interesting.
Possible insight in to how prolific their idea generation may be, and whether it is circumscribed by professional domain or not.
With a bit of two-way questioning on these, you should have a basic sense of their capacity as a communicator as well as personality.
Important: Predominantly co-workers and direct managers should interview, not random HR departments, external contractors or disconnected tiers of management.
We work on fixing that, so we implement a process: founders pick some personality traits they find essential for a great hire, we prepare an interview playbook, and they can be certain that the new guy is actually what they are hoping for.
Interviewing for personality (cultural fit) is hard because we (humans) are hardwired to survive so we give answers that we think are in our favour - and not necessarily truthful. Key to our approach is that we use falsifiable questions, meaning that you can't guess what I believe is a right answer.
If this helps any fellow founders here, am happy to lend a hand.
Now pick some arbitrary rating system (5 stars, school grades, 1-10 points, whatever). It does not matter that much which one you pick. The hard part is to stay consistent about it over time.
Now, here is the way how to make it more pleasant for your candidates. After 3/4 of the time, write down your rating and discuss it with the candidate. E.g. "I gave you four stars for cultural fit. I get good vibes with you but you would be the only PhD in our startup, so you are probably more academic than everybody else." Give them a chance to correct your rating. "Oh, I forgot to mention I'm involved in my brothers construction startup. I feel fine in that non-academic world."
Giving such honest and clear feedback to candidates is a better experience than most companies.
A lot of interviewers get lost with these obscure algorithms, and forget what they're actually trying to test: the candidate's thought process.
A counterpoint on why I wouldn't ask anyone to implement something (such as a web server) using a framework: They're ever-changing. Furthermore, if I were to try and see if candidates understand building an API well, I'd rather ask them a mix of networking and open-ended, system design style questions.
Can you elaborate as to why this is a contradiction? Yes, they’re ever changing but doesn’t mean it’s not an interesting problem to solve.
How would you design/write xyz
What's wrong with this code [code] (bug, security issue, bad code)
What do you think about
Use multiple (2-4) recruiters paid on contingency. Receive about 5-20 resumes a week from them (total, not per recruiter!)
Filter these resumes to about 3 per week. Interview these three yourself for 30 mins each. Just ask them for more details about technical experience on their resumes to detect liars. If your recruiters are any good they will have filtered out people based on the soft skills / career questions.
Reject the liars and accept just one of the honest people (if there are any). Push the one forward to your team. Let your team loose on the candidate with any bizarre (but legal) questions they like. Give them the final say on the hire. Repeat until position is filled.
I never focus on algorithms or particular technologies. When I interview someone, I'm looking for two things:
- First and most important: we both are in the same wave. I want to make sure the applicant understands what's the goal of the position. For example, once a guy with over 10+ of experience applied for a junior position but demanding responsibilities and salary of a senior.
- Second, expose a general problem and work with the applicant to get an idea about what can we do. I don't expect a solution in 5 minutes, I'm looking for ideas of how we can tackle this problem and how we would start working on it.
2 what would be a moonshot feature for P? (If you had a perfect AI lib or unlimited processing power or devices spread all over the world)
3 if you had to find someone to take over your job (for you to manage the new one), what would you look for?
-Don't ask questions designed specifically to gauge how the candidate 'thinks'. If you do pose such a question have another interview with the candidate some days later so that the candidate can think though various solutions.
- Don't ask questions which are cliched like 1) Tell me about yourself 2) Tell me your strengths and weaknesses
-Don't continue interviewing a candidate if you have decided not to hire him/her.
What you can do:
- Ask them to look up an answers using google/search
- Paid assignments
You discuss different architecture options, feature trade offs...
It's easier for both parts to connect, it will give you a glimpse of how the candidate faces requirements from other teams and you give him a taste of the inner workings.
It was an easy assignment that covered a lot of concepts, and it was followed by a 2 hour techinical discussion on it and all things related and unrelated with my future colleagues.
Outside of that the interview stuff was just around personality and culture fit.
The algo interviews are partially to test can you do the stuff. If you have done similar and can describe it in sufficient detail, then you know it.
First couple of rules:
- quick turnaround. We want to give an answer to a candidate within a week they engage us. If we make an offer, we give them plenty time to sign the offer, even shop around, before they make the decision
- no step can be repeated or returned to. If a candidate progressed from step 2 to 3, our hiring managers are not allowed to go back to the previous step
- we disclose as much information as possible early so there are no surprises
Here is the process:
1) Initial 30 min call with one of our recruiters. We cover mostly the company, the compensation package for the role (usually range based on the level they fit in) and our expectations. We also cover the culture and share as much as we can so the candidate can make informed decision if this is a right fit. We ask some high level questions, but usually focus on the company itself.
2) Take home challenge. We have one for every type of position, from engineering to marketing. They are usually simple and fairly arbitrary problems that illustrate candidate's thinking about problem, how they analyze it, what tools they choose to solve it, etc. When we design the challenges, we are aiming for it to take around 30 minutes. We are not trying to get the person to any significant work for us, rather just figure out how would they solve problem if it was given to them at their position.
3) The candidate meets 4 people:
- their manager where they cover the role, the expectations, the growth at the company and anything related to their job
- random colleague. Essentially someone they will not work directly with. We use this person to help us to focus on the candidate themselves rather than their skills
- our COO, who covers with them more high level topics like what they did/didn't like about previous employments, what they care about in working environment and such
- our CEO (me). I usually focus on their individualities, especially passions and hobbies. I want to see that they care about something, anything. I also answer any unanswered questions from vision, company's cash position, customers, how we generally work, etc
As I mentioned, we try to get an offer to candidate's hands within a week. With this process we get around 95% offer acceptance rate.
As an interviewer, I don't really care if a candidate happened to know the answer to an algorithmic puzzle or has a deep knowledge of academic minutia. I want to know three things: can this person reliably and meaningfully solve technical problems, can they communicate their findings to the team, and finally – do they strive to learn the details or do they just skim/superficially solve problems? Obviously expertise in a job-relevant domain is great too - but I'd consider it a bonus; the difference between a senior and a junior/mid-level hire.
The first two are part of a quiz we give candidates, it has a couple basic questions on it that simply require them to logically think through a problem they likely haven't seen before and provide feedback. I don't even care if the answers are correct, but they should be able to explain their thought process and reasoning clearly in a couple followup questions. Bonus points to candidates that ask for feedback on their answer.
For the last point, during the interview I get the candidate to describe a project at a recent workplace, once they have given the description, I'll ask questions to see if they evaluate and learn from their previous work.
I am constantly shocked at how many candidates I interview that fail one or more parts of this spectacularly (especially senior level candidates), to the point where I question whether evaluating candidates these ways is unfair, but also knowing that if someone doesn't do this they likely are going to fail when given a problem without an obvious answer.
There are many awesome people you can hire that you wouldn't hang out after work, and that's OK - you don't have to be friends with all your coworkers, you just have to be able to work together.
Instead, identify qualities & behaviors that your company values or are exhibited by successful employees and ask for examples of the candidate demonstrating them.
Instead, I think we should attempt to really dig down into elements of "culture" and identify the ones that are relevant to the job vs those that are not. For example, some relevant elements might be:
* Preferred development process.
* Pace of delivery and willingness to ship bugs.
* How disputes are settled.
* Using libraries vs build it in house.
* Process for adopting new technology.
* Preferred communication methods. Email vs chat vs video calls vs in-person (back in the pre-plague times).
And there are plenty of things that are _not_ important:
* Hobbies and how people spend their time outside of work.
* Favorite films, music, books, food, drinks, etc.
* Whether you work on side and/or FOSS projects (unless there's a legal issue with side projects).
I'd also note that for some of these things I expect the candidate to ask _me_ about them if that's something they care about. In fact, those sorts of questions are a very positive signal to me. It shows that the candidate has some insight into their own work and what makes them happy and productive. I'm very happy to find out that this isn't a good fit _before_ they've started working with me!
> * How disputes are settled.
Suppose you pose to the candidate a situation in which they are in the right, and ask them how they would proceed. They respond by giving the impression that they wouldn't easily compromise on their correct point of view. Is this positive or negative?
For many interviewers the answer will unconsciously depend on the candidate's gender. If the candidate is a man it's positive because they are "assertive" and "stick to their guns", whereas if they're a woman it's negative because they're "pushy" and "unwilling to listen to other points of view".
This should, ideally, depend on your team and your organization, right? There's no one right answer here.
I agree that there is an opportunity for bias here, but that goes for pretty much any question you could ask. You could ask them about programming languages they've used and find out they've used a lot of different languages. And the bias response would be to think that for men this means they're inquisitive and curious, but for women this means they're flighty and unable to stick with anything.
One thing that I think can help is to talk with the hiring team in advance about what you hope to get out of each question. For example, in asking about how you'd handle a technical dispute, you could focus on whether they would go directly to their teammates or manager before going to the next level up in the management chain. And on the flip side, you may want to make sure that their answer isn't "if I'm right I will never let it go because it's important to be right." By focusing on a few key points you're looking for, it may (I hope) limit the potential for bias.
But it's legitimate to ask about non-technical soft skills in an interview. Those are a _huge_ part of doing any job that involves interacting with other people. And conversely, there are people who are hugely destructive to a team or organization because of the lack of such skills. I don't think the answer to bias is to simply avoid any topic where bias could come into play.
Corporate culture is not about being similar to your coworkers, it is exactly what you said - company values and ways of working together. Don't get tripped up by the word "culture" - it has multiple meanings.
Yes people in teams should like each other because otherwise you will have a bunch of unmotivated pixel pushers as developers that only think when the time runs out.
In all of the companies were soft skills did not matter and I worked for people were coming and going.
No you don't have to be friends outside of work but usually you are still spending 8 hours 5 days a week with these people and maybe for some people it is okay to not like the other people in the company but from my experience this leads to no good in the long run.
1a. Optional coding test at home (no time limit)
1b. Optional coding test in front of me (1hr time limit)
2. Very simple fizzbuzz-level problem. Essentially can you break down a problem with an if statement and while loop. This has the least bearing on the process.
3. System architecture/design "whiteboard". We ask a question that's essentially an entirely different problem space from out company but who's end system design, and lifecycle, looks almost exactly as the design our company took. Essentially if you `s/A/B/g` you'll have a 1:1 idea of what our company's infrastructure looks like by the end of the problem. There are absolutely no wrong answers, just answer how you would if you were given this ticket ("design X") and I'll ask from time to time "how many QPS can this handle", "any bottlenecks you can identify", "how much bandwidth/cpu/memory will this take", etc to see if they understand the implications of their decisions.
Surprising facts:
1. Most people can do the take home coding test "enough" but do horribly on the on the phone coding test. This has been shocking to me because I'm not someone who struggles with this aspect of the interview process (I'm horrible at explaining myself).
2. When we get into the system arch/design session 99% of people have no idea how to proceed. They've never sketched out the design for something in front of someone else and have often never thought about latency, memory, CPU, disk, etc. We had one engineer take it who couldn't even come up with a way to estimate how many QPS a service might receive in the worst case. Another engineer settled on a design that was going to cost >$1mill/month in AWS DB costs and could not imagine any way to reduce this bill and thought this was reasonable (just needed to put a cache node at an edge location).
3. People use buzz words. They use them a lot. If I had a working feature for every time I've heard someone just say "autoscale it" whenever they hit a bottleneck during the arch/design section I wouldn't need to hire an engineers. It shows a core lack of understanding of where the bottlenecks in our interview question are coming from.
4. We see some (small) percentage give up on doing the coding test. I'd say it's less than 10% of inbound leads. Obviously a significant portion of people don't want to code in front of someone and don't want to code at home. Unfortunately I can't advocate for hiring someone until I see code that they've written. We've had horrible experiences hiring "senior" engineers who actually don't know how to program.
5. I've had people freak out, and start cursing at me, when we get to an interview question they get stuck on. It's something I'd never have imagined happening in a professional setting but, at least, they've been filtered and won't be working with me in the future!
6. Resumes and pedigree have the worst correlation to quality of the engineer I've ever seen. I've seen undergrads working part time in the summer output 10x value to full time engineers with 5+ years of experience. "10 years of experience or 1 year 10 times" is the most accurate pattern to describe this.
Yea, it's a very different mentality to get yourself into. I find the system design question to be the most fun. Anyone can learn to code, anyone can learn your style guide/convention, etc. Very few people can be:
1. Given a problem ("{A client,we} want...")
2. Come up with a solution ("Ok, we'll be able to...")
3. Implement that solution with no oversight or assistance while only asking for pointers on where existing things are defined ("I'm trying to X, Y does X, where does that code/config live?")
I mostly work at small startups that need to improve their engineering quality so most of the roles I am attempting to fill are engineers who can be a massive value add and lead projects on their own without much hand holding.
If I was instead working in a company where 99% of the work was maintenance, phone calls, and busy work I'd probably forgo this screen-er. I'd likely not work for that kind of company but it's just something to keep in mind.
It allows your company to learn, to get a feel for the person's ability to communicate, and to stay away from stupid things like tests.
It's also easily the best interviewing process I've ever experienced.
---
edit: Since it appears that multiple people have taken this to represent an hour+ long presentation, let me clarify here.
A better way to think about this would be a long-form discussion about a topic. The initial "talk" would be 5-10 minutes, with a Q&A afterwards to ask for clarifying questions.
I've been working in this field for over 25 years. I spent roughly 30 minutes of prep for the interview and it was easily the best interview process I've ever experienced.
If you can’t boil your work down into accessible presentations and adapt them for technical or non-technical audiences, you won’t get anywhere.
Nobody will agree to help you, allow your team to grow, give you budget for tools you need, make compromises with you around integration requirements or shared design patterns, unless you can frequently convert your software work into highly professional presentations.
I’d argue that competent presenting and writing skills are of equal, or even greater, importance than software skills or expert knowledge, whether for a research software job, software contractor job, SWE within tech / ecommerce / banking etc industries - across the board there are pretty much zero types of software engineering jobs in which software skills are actually the most important thing. Presenting and writing are frequently much more important.
This isn't true in my experience, there are plenty of well paid jobs where you don't have to talk to non-technical people. For example, lets say you are developing infrastructure at Google, how often do you think you talk to non-engineers? Never, everyone in your management chain will be engineers, your product managers are ex-engineers, your users are engineers etc. This is true for directors managing lots of people as well as well.
And even when you develop services much of it is just getting the technical parts right so you can work there as well at Google with very little time spent talking to non-technical people, so senior positions there are mostly about technical stuff.
You could argue that Google and other big companies are a special case, however if we exclude low paid jobs where you just earn half of what you'd do at Google or Facebook then pure technical work like this is a very significant fraction of the market. I'm not sure why I'd work on my presentations skills so I could get hired by a company paying me less.
I can speak from experience since I help run the site reliability incident response team in my (large ecommerce) company.
How often do rank and file SRE ICs need to speak with non-technical staff about projects? All the time.
They give tech talks and council presentations, they contribute work to quarterly roadmap project pitches, and they give business scorecard presentations to walk higher level managers through the breakdown of costs and benefits, why we should pay money for a new cloud tool, what time & effort some org-wide planned migration or upgrade is going to require.
Let alone basic stuff like writing effective tickets / issues / RFCs / PR descriptions, postmortems, etc., and walking teammates through work artifacts daily.
For every two hours of software work, I’d expect 1 hour of “wider audience communication” work - and that is for deeply technical ICs in roles like SRE. The closer you get to product engineering, the more the ratio shifts towards communication.
I can tell you this translates specifically to Google & Facebook - because most of the SRE colleagues in my company came from Google & Facebook, and attest to that being the case there too.
To be clear, explaining your work is a part of every developers job. But top notch presentation skills don't need to be a part of it. Developers can talk to other developers in sub-optimal ways that are still effective. But when your hiring process has an hour long presentation as a part of it, you are optimizing for the thing that is very likely not the most important feature of the role. The point is to structure your hiring process such that it optimizes for the right things for the role.
But also, I really really dislike when others talk about "optimizing" people. It's not possible to optimize your hiring process for anything useful. Due to this, there should be a lot more empathy in the hiring process than their actually is because everyone wants to make it robot-like by "optimizing" something.
I agree with the other commenter, it baffles me that we think that asking someone to write code, when the majority of their job will be writing code, is somehow an unfair interview process so the solution is to have them give only hours of powerpoint presentations.
I've come across many a smart person who would abhor the idea of listening to a presentation, let alone give one. Give me a document they'll say, and they will go through it at depth and collaborate amazingly over text. A mix of both is required. You don't want everyone to be giving presentations all day, just as you don't want no one to be attending any presentations. Diversity is essential, and it's more effective to create structures that allow different types of people to work with one another than to mould people into personalities.
It's akin to saying that because some people have anxiety issues, we should completely do away with interviews and just let people randomly walk in and do work.
Because that's the only way to potentially avoid the criticism you gave here, only that's also not true because locale comes into play. "You didn't create a building every 500 feet that people can walk into, so now you're disenfranchising people without cars!".
My point here is that this is not a useful criticism. Come up with a useful grievance, one that's actually actionable, and then maybe we can talk.
Not every personality difference can be reduced to anxiety. That is precisely the kind of closed world-view I was talking about that can form within homogeneous groups.
You act as if the people listening aren't aware of this.
What is it you believe exactly, that if person A sounded nervous they wouldn't be hired?
The entire point is to get a conversation started, and the following Q&A allows them to do something everyone needs to do in a job, which is to answer questions.
No interview process is perfect. You can literally give me a description of any interview process and I can start finding problems with it.
Just as you can literally give me a description of any software solution and I can start finding problems with it.
That you can find problems with it doesn't make it useless or harmful. Please come up with better criticisms.
I can do presentations, however you are asking me to hold a free 30 minute lecture teaching your employees about something. I'd rather not, basically every company gets excited to hire me anyway and I'd rather spend a few hours doing some comfortable coding over spending a few hours preparing a presentation.
Technically speaking, if we're in an interview to "optimize for the right thing", then we would just have them come in and type.
But we all know that's also optimizing for "the wrong thing" despite it being what we do physically day in and day out.
This is because you _CAN'T_ "optimize for the right thing" in an interview. It's not possible and this needs to be acknowledged.
Therefore you go for something else. Anyone who is a software developer can listen to a 5-10 minute presentation from another developer and start forming general, broad opinions. They can then start asking questions afterwards to help clarify those opinions.
But the Q&A also has another purpose. Way too often interviewers will come to conclusions that don't follow (non-sequitur). I once had someone tell me they didn't feel I would be a team player because I told them as long as I had headphones, open office would be ok. The main difference I've found between business people and technical people is that business people will ask clarifying questions. The Q&A sets the expectation for the _INTERVIEWER_ to ask clarifying questions.
Some developers are more comfortable with communicating via 1:1 conversations, design documents, proof of concept PRs, etc.
I've done a lot of public speaking and presenting, but am aware I am in the minority of developers. I've also managed a lot of people and many of them have struggled with self confidence and presenting to large groups. This has not been a reflection on their performance as an engineer.
To explain, college curriculum's require atleast 1 public speaking class. These classes are infamously difficult for a lot of people who find it scary to talk in front of a class of 30+ students. But society as a whole considered it to be the young persons responsibility to overcome this fear.
Nowadays, it's far more likely that people are going to make arguments such as "the class shouldn't be required" or some other thing to try and help the young person avoid the scary public speaking.
My point here is that yes, it's true there are going to be some who struggle with presenting to 5-6 people. That's on them. If they can't talk to 5-6 people they're going to find something as simple as a daily standup to be scary as well, should we just get rid of it? (Actually, I hate standups and think they're a waste of time, but you get the point).
It's also not clear to me why you characterized this as a "large group", but I'll give you the benefit of the doubt and say maybe you're used to interviews where 20-30 people get involved. That's certainly not what I'm suggesting.
And finally,
I can't imagine a young developer not being able to have a 5-10 minute talk in front of others about something like, say "What is memory safety in rust and why should it be used for new projects" when we're hiring for someone to work in rust.
A more senior person would obviously get more generic options. The one I personally spoke about was roughly "What can we as a company be doing to empower software developers to create the best software possible".
* in a formal setting,
* to people you don't know,
* with a high stakes outcome (performance determines whether you get the job)
Is not neither a skill that developers need to have, nor is creating a "better experience for applicants". This is in no way analogous to giving an update at standup.
I've had applicants physically shake during interviews when I'm doing my best to create a comfortable environment. Some people just get really nervous. And when hired those same people have become comfortable quickly and performed well in their day jobs.
I've done a lot of both traditional interviews (presentation on research or previous work followed by 1:1) and "tech" interviews (coding challenges and whiteboarding) from both sides. Tech-style interviews evaluate a much narrower swath of skills are are much more easily gamed.
I feel that a presentation evaluates broader things: how well do you communicate and how comfortable are you discussing and answering questions about a topic you know well.
It's a better indicator of job performance, i.m.o.
Our group hires people in the software+hardware+mechatronics space, and we look for people that bring diverse skills we don’t yet cover well. The kind of person we want can teach us something new in 30 minutes, and a good project for that could be backyard hackery or university research. Either is good. I totally love the format. (Then again, I am a “Talk is cheap. Show me a working (even if janky) robot.” kind of person.)
Related, a former BioMed engineer coworker said Medtronic used a similar process - start the interview day with a presentation.