Do you disagree with this premise? If so, why?
How do you gauge team skills during your interview process? What do you look for? How has it worked for you?
Most people opted to do the exercises while thinking out loud. If they paused for a few minutes or seemed lost, we'd offer hints or ask questions, etc. Let them take the discussion where ever they wanted.
All in all worked very well. Candidates seemed more comfortable than just doing whiteboard exercises and we got a really good feel for how they worked with others. We didn't really care much about the code; we wanted to see how they approached problem solving, how well they responded to feedback or questions about potential optimizations or offers of help. Or what happened if they were shown that their approach was a dead-end, but they still doggedly kept pursuing it.
It revealed personality issues very quickly. Nervousness was normal. Some people were hostile when bugs were pointed out, some simply couldn't get started even with extensive help, and others went heads down for half an hour and then came back up with the answer.
It wasn't perfect, but of all the interview techniques we used over the years, it was the one tool that we never got rid of because it proved to be so useful.
How did people who took this approach fare in your scheme? Thumbs up for solving the problem or penalised for “not a team player?”
We're hiring them to be developers first and foremost, so if that's how they work best, so be it. Then we move on to discussing their solution and if they thought it could be improved in any way. The rationale for this phase of interviewing was mainly to accomplish the following:
1) Can you take a problem statement and show progress in writing code in the language of your choice to solve it? You don't have to complete the entire exercise, but you do need to show us that given enough time, you'd probably get the right answer and that you can explain your logic.
2) Attempt to weed out the personalities who can't take negative feedback or refuse to accept guidance when they're wrong.
Once in an interview for a startup company(~10 people), when I got into the office they had an uncompleted coat hunger from IKEA and there was a person trying to assemble it. They intentionally let me wait for a bit next to him and he started to look confused with the manual so naturally, I offered to help him and got on my knees and started to read the manual.
They told me much later that that was a test and I had successfully passed it.
I am not sure about the validity of such a test and maybe someone else can verify it.
> maybe someone else can verify it
I'm really curious to see if anyone has implemented such a test over and over again.. The biggest challenge right now with changing up the interview process is validating the change is tough. You need to use the same challenge in a lot of interviews, and track how well the candidates did over time at their jobs, if hired, in order to be able to verify. Tough to do for a startup.
http://worknearyou.net/wp-content/uploads/2013/08/ikea_job_i...
1. Personality and language skills - are they sufficiently outgoing and conversant to be able to interface well with others? Introversion is common in our profession but in order to work on a team you've got to be able to talk to people in person. Panel interviews are good for this; anyone who can keep up decently, seems personable, not stressed by the situation, and manages to leave an imprint is likely a good candidate by this measure.
2. Written communication skills - while arranging the interview, you're going to be emailing back and forth a few times. Ask some questions and see how they respond.
3. Asking about previous teams and their views on team composition, dynamics, and work distribution; handling large workloads, dealing with absence due to vacation/sickness/etc. This is a good kind of question because unlike asking about what their previous teams _did_, which someone can embellish, asking about the actual process of dealing with different scenarios shows their own understanding. Present a few scenarios and ask the candidate what they think a team lead / manager should do in that situation.
It's going to be hard to get a quantitative metric for this, you have to rely on your intuition.
I'm not sure high stress environment test is a good measure. I've worked with some amazing teammates before that might have been nervous if in such a situation.
>Present a few scenarios and ask the candidate what they think a team lead / manager should do in that situation.
I like this approach.
Regarding the question examples you gave, do you look mostly for how they reason through those situations, and their ability to put themselves in their teammates' / managers' shoes, or do you also have a spectrum of [right --> wrong] answers for those questions.
Talking with a few of your future coworkers in a room where nothing is on fire shouldn't be "high stress" - interviews are always a bit of a stress inducing thing, but ultimately if you're hiring someone who may have to deal with tough situations from time to time on a team, this sort of situation should be easy enough for them to handle.
If they're not in so much of an ops role and won't be dealing with things that are on fire, you can always reduce this stress by starting 1 on 1, and then adding people as the interview goes along in order to give them a more comfortable base.
> Regarding the question examples you gave, do you look mostly for how they reason through those situations, and their ability to put themselves in their teammates' / managers' shoes, or do you also have a spectrum of [right --> wrong] answers for those questions.
For me, it's more of a matter of determining whether they've just worked on a team as a member, or if they've actually thought about the dynamics of teams and have their own opinions and ideas about what makes for a high performance team vs. a dysfunctional one. Getting a feeling for what sorts of business processes they like, how they like to communicate, what level of tracking artifacts they require/feel comfortable with, how they think work should be broken down, etc. is a good way to understand if they're going to be someone who works well with the way your team is already composed, and if they're going to be someone who is going to help evolve the team in a better direction.
If they say something that makes you bristle, pay attention to your intuition. If you think, "gee, person X on my team isn't going to like that", consider it, because you want person X to be happy and productive and introducing a new team member who is going to potentially have conflict with them may not be worth it. Every aspect of your evaluation forms a heuristic in your mind that you can use to decide if they're the right blend of worker for your particular team.
I really like trusting a lot of this sort of thing to my hindbrain. There's a lot of magic happening back there that combines things in a big-data sort of way that we don't consciously understand, but when it comes to interactions with other humans, we evolved to be good at this kind of judgement - since it's life or death, in the state of nature - and we should trust our instincts, provided we've tuned them to not be irrationally prejudiced.
Instead, just work with them. Have them pair with 1 or 2 people for a few hours. Maybe even engage them in a small team meeting if one comes up naturally. There's no assessment as good as actually doing the things they'd be doing and working with the people they'd actually be working with.
It's a better, more accurate assessment in both directions. For them to want to join your team and for you to want to have them on it. Its also a ton easier to train your team how to execute this method effectively. The standard alternatives are spurious generally, and what little value can be derived from them requires well-prepared, expertly-trained assessors.
Why contrive a weird edifice?
Disclaimer: am currently a person responsible for reforming and building a high-performing team in a very demanding field across hardware and software (both internal and external) disciplines.
My concern is that oftentimes, there's a lot of context to absorb before being able to contribute, and a few hours wouldn't be enough. Don't want to extend interviews to multiple days --> biases against candidates that have packed interview schedules.
I'm thinking we could structure a problem to take a few hours, and pair program with them, with the interviewers swapping out every hour and half (which would also help test how well the candidate brings someone who doesn't have context, up to speed).
Or perhaps, have multiple candidates work together to solve a problem, and observe how they work together.
I also assume you know well ahead of time what kind of work you're hiring for and what kind of work (in general shape) your team will be doing, so that you can schedule the interview or the team's work to appropriately coincide?
I have to hire people who either have an esoteric skill-set or people who will have to rapidly learn one (formal methods, vehicle hacking, control theory, microkernel architecture, etc.), and I've yet to find a decent substitute for just planning ahead and setting up the environment to get the existing team and the candidate directly engaged on something immediate and real. I've seen a ton of bad hires made trying to contrive scenarios and meaning from more traditional methods.
:shrug:
And that exactly is the problem with interviews where we can't measure what we should really measuring, so despite these marathons and what not, ending up with misfits. I can't explain it better than below: source [0]:
Your scenarios are different, the normal scenario inside a company is: a) Your boss tells the new problem to the team at Monday morning. b) You have your first approach in your mind to the problem. c) The team discuss the problem for 50 minutes. You still don't have a clear idea about the problem and for sure you don't have any idea about how to solve it. But who cares? Is pancake day ! d) Lunch, you think about the problem by your own, for first time in a "serious" way. e) One day later you write an email to clear some doubts about the problem. f) You talk with another member of the team. He notices some potential problems with your approach but also you realize that, after all, your approach is better than his. You walk in the supermarket alley with your wife thinking about how to explain why your approach is better using a metaphor and trying to foresee objections from other team members. g) Thursday you finish those TDD pending upgrades, do the pull request and then you think an hour about the problem meanwhile you are still in the "zone". You have those beautiful "I got it! " moments. h) Friday you "play" with some library that you find googling and you read some code from github from a guy who did something similar. You then notice that there is a part or a step that you didn't consider before, that could change your design to solve the problem in a deep or shallow way. i) After redditing for an hour, yo do a minimal code that solve the functionality. At this time the problem has been processed in your mind for a week from different angles and using different tools. j) Monday at lunch time you take the marker and you write in the whiteboard trying to explain to the team your solution. The team decides to follow your approach. The other scenario is: a) The interviewer tells you an isolated and descontextualized problem. b) Ten seconds later you have a marker in your hand in front of a empty whiteboard. My problem is: how the real cognitive process can be replicated in an interview? Notice that in the real scenario is not the most smart developer but the best "architect" who gets the best solution after many trails and error experiments and dead-ends paths. Of course companies must have a selection process, but the whiteboard kind is just producing a lot of false negatives.
[0]: https://www.reddit.com/r/programming/comments/4h15a1/why_is_...
I believe it's called Behavior Based Interviews. But basically the interviewing team looks at a list of qualities and picks a few (~3 per person) that they think are valuable for someone joining the team. For example, I chose to look for: Priority Setting, Compassion, and Self Development. Once we have some qualities chosen, we develop questions that are specifically meant to test for those qualities.
Overall I really enjoy this kind of interview process. I feel like these qualities are more important than technical ability, so it's nice to put more emphasis on them during the interview process (We still do assess technical ability of course). And for us, it results in new team members that I'm excited to work with!
> I chose to look for: Priority Setting, Compassion, and Self Development. Once we have some qualities chosen, we develop questions that are specifically meant to test for those qualities.
What are some questions that you used?
Priority Setting - Have you ever felt overwhelmed by the work you've been given? How do you manage the work so that it gets done on time?
Self Development - How do go about seeking feedback from those around you to facilitate your personal growth? What is something that you've learned about yourself recently that you found helpful?
Compassion - Tell me about a time where you worked with someone who was struggling, for whatever reason. What were some of the challenges? How did you go about helping them deal with their struggle?
1. High functioning teams are in the order of 5–10 people 2. Everyone in the team is respected for what they bring to the group regardless of experience.
If you’ve 20 years experience you’re valued for your knowledge of pitfalls etc. If you’ve 2 years you might be valued for your enthusiasm. Or if your a fresh graduate you could be valued on your ability to question conventions that others do not.
My question is: given this, how would an interview ever hope to sift out a potential candidate. Especially when the qualities of the final aren’t (could never?) be properly specified relative to the team they were due to go into.
I agree with this.
I think there are ways to notice this during interviews. I'm a fan of a junior engineer interviewing a candidate for a senior position - as a way to see how well they interact. Perhaps the question could be more open ended (e.g. explain x,y,z concepts - maybe oven ones that the interviewer doesn't know but wants to learn). If the junior feels like they had a good interaction, were able to learn, and felt respected through the process, great. If the candidate seemed offended by the lack of experience of the interviewer, then not great.
A highly subjective signal though.
What about the person who can resolve conflict. The person who always takes on a task to the bitter end regardless how bad it smells. The person who thinks ahead. The developer who loves process and documentation. The developer who can code you out of a tricky situation. The person who’s always trying new things.
Personally I think it’s a lottery. Like putting together a football team but we don’t even know the positions yet. We just know they can play football
I'm trying to think through how to assess team skills well that would also fit into the constraints of an interview.