Just curious what folks thoughts are on why the industry is skewed this way?
On the other hand most people's GitHub profile is a mishmash of forks and projects that are either trivial or blank and mostly useless for hiring. Or, maybe the candidate has a serious GitHub presence in which case it might take hours of questioning to even understand what they've done, let alone if they're any good at it. Their activity might be mostly in a natural language you don't understand, and how could you evaluate that fairly?
Also GitHub activity is skewed toward people who have time and energy to create a portfolio for free. Their GitHub portfolio or lack of it might say less about their skill and more about the rest of their life which is not relevant for hiring.
That said if a candidate provides their GitHub or other portfolio then I think you should consider it just like any other part of their resume, not to replace the coding challenges but as part of the chat about past experience.
That is a great point! Similar to admissions tests to a college I suppose.
> Their activity might be mostly in a natural language you don't understand, and how could you evaluate that fairly?
Good point as well.
> That said if a candidate provides their GitHub or other portfolio then I think you should consider it just like any other part of their resume, not to replace the coding challenges but as part of the chat about past experience.
I think I understand. Would it be fair to say that, you consider the the "score" of the coding exercises as weighted higher than potentially significant contributions and maintenance of open source projects _because_ they are normalized?
I have no explanation for this phenomenon.
Why do you keep using these agencies that send you unqualified candidates?
Also, I think the better way to structure coding challenges is as a series of increasingly difficult problems that build on each other. If the candidate gets stuck on the easier problems, you can just quietly ignore the later ones. It feels very risky to start an interview with something like "Okay, hah, here's a real simple one to get us started," because if they fall apart there because of nerves or something, they will not recover in time. What do you think?
Corporate politics.
> as a series of increasingly difficult problems that build on each other.
...I mean, that's exactly how I structure these. When a candidate is stuck on the very first one of the series, though, and spends the entire interview on it... I think that's a fail. What do you think?
Assuming your process isn't broken, there's a psychological risk to asking a qualified person a question that is excruciatingly basic. If they know it's a question only a fool or liar would get wrong, then a momentary blank can lead to an anxiety spiral. I think even the first question should be plausibly challenging.
I also don't understand why you're letting someone spin out for an entire interview on a simple question. Are we talking like half an hour or more? Maybe just let them try for a couple minutes and then pull back, give them a breather if they look stressed, and gently confirm that they've done actual hands-on programming in the past. Maybe their bootcamp misled them, or their previous job was a project manager sort of thing. Bad fit, sorry we didn't catch this earlier, best of luck to you.
there, you see what I did? none of what you wrote above makes any sense.
I rather spend my time developing IP instead of repetitive leetcode practicing. either will take you several months, but the former will result in an asset which you own the rights to and can monetize.
Since devrob asked the question and they link to their github in their HN profile, I took five minutes to check it out. Don't want to spend time making a full evaluation, but it is obvious that devrob's strength is at least above average.
It takes a strong hiring team to do it this way, knowing what they want and making decisions based on rules they've set for themselves. Weak teams will go by gut feelings, ego, random quiz questions, stuff like that. Put it this way, they say you don't really understand something unless you can teach it to someone else. You said you can evaluate programming strength within an hour. Could you teach someone else, or put down in writing, how to do that consistently? That's what I'm saying hiring teams should do.
If the interviewing is inconclusive it's sometimes directly consulted by the decision makers.
More pragmatically. How do you know they wrote the code on their github? Do you expect people who are already working for a company to write even more code when they're not working? How will candidates get a feel for the company themselves? How do you ensure some level of consistency? How would you generate documentation that could be used in any kind of discrimination case to protect yourself? What kind of systemic biases do you think GitHub/portfolio review style interviewing would create. How do you understand a person's disposition without adversity?
I can't say how many candidates I should have passed but didn't, but pretty much every candidate I said yes to performed the job itself pretty proportional to how they performed in the interview.
My only counter consideration might be that one's resume which states:
"I built XYZ at $COMPANY and resulted in $Xm in revenue growth" and one could simply show the person the product or area under which professional worked.
Per your point regarding candidate pool to evaluate, would you consider it fair to say that you see the purpose of the live problems and technical projects as a means of normalizing the ground of analysis (i.e. comparing apples to apples versus apples to oranges)?
What did you, specifically, do? Did you lead a team of problem-solvers, did you glue together someone else's black-box solutions, did you personally invent something radically new? I've seen people across that entire spectrum make statements such as this, and they are best suited for very very different positions.
This is absolutely a great thing to chat about during the background and past experience part of the live interview, but if I'm hiring for a hands-on position, I'd still like to see how you solve problems.
A. System design and architecture
B. Problem understanding and solving
C. Team work and communication including navigating and working with stakeholders
D. Building requirements and negotiating tradeoffs
E. Ability to deliver consistently and reliability
F. Inventive and resourceful thinking
(Obviously this is non-exhaustive, but some top of mind considerations)
But at the end the deliverable is a combination of documentation, code and software is it not?
Supposing $COMPANY had an interview structure as follows:
1. Basic technical screen: Confirm candidate can program
2. System design: Evaluate candidate's ability to design systems
3. Algorithmic problem solving: Evaluate abstract problem solving
4. Behaviourial interview: Determine if candidate is good fit
Where would you map the above A-F or any other critical elements which I may have missed to how they manifest in 1-4?
Also note that in the real world, many of the best developers have nothing to show on GitHub. All of their code is proprietary closed source.
No. At the end the deliverable is built based on communication between individuals on a team. If that doesn't work, it doesn't matter how good the code produced by your individuals is; it's going to be a mess.
It's common for people's hobby projects to use libraries and tools they wouldn't normally use, to not bother with tests or documentation they would create for professional work, and so on. They're just sharing something they thought others might find useful or interesting.
A live scenario will force time constraints.
I have almost 20 years of IP development behind me, none of it is on github, because I paid for it out of my own pocket. I have products in the market. companies i've interviewed with don't want to take any of this into account BUT they would sure love to get their hands on my work, and try to extract tidbits of IP out of me during interviews (even though they do not want to let me talk about my work as an alternative to their ridiculous and irrelevant leetcode questions).
- can they communicate? Can we talk about tech stuff in a way that we actually understand each other?
- what kind of domain expertise they have? E-commerce? Banking? Cloud? Compilers? ...
- how good they are at building abstractions
- ... a lot of stuff like: not being an asshole, being a team-player, being positive (or realistic), honest...
- how good they code
Nevertheless.
If it's a toy problem, I'd rather it was my pick of toy problem than your pick of toy problem. If it's a hard problem or a significant open source contribution, it will take much more time to fairly evaluate than a toy problem, and so there needs to be some form of screening of candidates first because there is only so much time and there are more candidates than that.
In order to evaluate your open source contribution on github, I need to invest considerable time understanding the codebase you are working on and the problems your changes are solving before I can even begin to properly think about your changesets, then prepare myself as I would for a live code review in order to give you a fair hearing; otherwise it's just Elon Musk's "send me your best line of code" all over again. I could enter the interview without the preparation and just let you talk me through things from cold, but then what I am really evaluating is your communication skills, not your problem-solving skills; which may be appropriate for some positions, but does not help me much if what I need is someone to problem-solve.
When there are more candidates than available positions, it then takes even more investment after the interviews to compare candidates to each other, and the results are relatively subjective.
Such a process may be appropriate for some senior or specialised positions with relatively few applicants, but in general does not scale well during We Just Got New Budget So Let's Recruit All The People month.
The live interview with simple questions and toy problems, for all its faults, has the (admittedly dubious) advantage that it is self-contained and that multiple candidates get similar or identical questions, making it easier not only to agree on a shortlist but also to spot potential problems with the interview process itself. One might then go look at github repositories as well as scheduling longer meet-and-greets for the most interesting people to make the final selection.
This isn't great, but choosing whose codebases to properly look at based just on the candidates' CVs is even worse, and that (or even worse methods! - they exist!) is what we are left with if we exclude the phone screen / live interview as a tool from our hiring pipeline.
(Note that writers, when submitting their novels to publishers, also don't throw the entire novel at the publisher from cold (or if they do, they aren't likely to get very far) - they generally have a cover letter with a brief summary, and supply a representative sample for the reviewer to read; different publishers have different rules for these, but generally no more than a chapter. Moreover, many publishers don't accept direct applications, or only accept them during specific, brief, time periods - by requiring the author to get an agent to agree to represent them, they are effectively outsourcing that initial screening step)
Is this to do with your comfort level in understanding their approach?
> need to invest considerable time understanding the codebase you are working on and the problems your changes are solving before I can even begin to properly think about your changesets
Completely fair.
> during We Just Got New Budget So Let's Recruit All The People month.
Lol
> One might then go look at github repositories as well as scheduling longer meet-and-greets for the most interesting people to make the final selection.
I see, so using the activity and portfolio _after_ the fact as a means to gauge additional signal once the baseline has been met
> Moreover, many publishers don't accept direct applications, or only accept them during specific, brief, time periods - by requiring the author to get an agent to agree to represent them, they are effectively outsourcing that initial screening step)
Interesting, thanks for sharing!
No, this is to do with knowing that they can, when asked to solve a problem, actually solve the problem. Just like Spolsky's fizzbuzz test. A surprising proportion cannot.
As part of a live interview, it also tests a bunch of other things that help me work out whether we would be happy collaborating on things: what's this person like to explain things to? What are they like at explaining what they're thinking or why they're doing what they're doing? Are they the sort of person that actually listens, or the sort that pattern-matches and ends up answering a different question to the one I asked? And so on. But a quick show-me-you-can-problem-solve-at-all, laugh and move on is the main goal.