Don’t “Hire by Keyword”
michaelrwolfe.com
michaelrwolfe.com
The broken part of tech interviews isn't that they ask technical questions, it's that no one on the hiring side looks at their process and asks "what is it that this part of our process is measuring?" and "does that factor matter in making a hiring decision?" Asking brain teasers and CS "fundamental" questions is fine as long as there is a good answer to those two questions, but instead everyone does cargo-cult interviews because Medium says that's how Big Tech does hiring.
Yes, that's exactly what we should all be doing more of.
Hiring is always a risk, and the "accepted" risk-mitigation strategies (hiring by keyword, brainteasers, etc.) have been shown to be stubbornly non-predictive. At best they apply a misleading veneer of certainty on an inherently uncertain process, and if I'm going to take a risk I'd at least prefer that I not fool myself about it.
So yes, accept more risk in hiring. You're already taking risk, but by accepting the feeling of more risk you'll at least be more deliberate about it.
IMO, this comes from the mistaken assumption that you want to most technically competent person possible. Instead basic competence is a minimum barrier and everything else combined is more important.
For me, if you pass the test of basic competence, and you can demonstrate evidence of accomplishment in any language, and you seem to pick things up quickly, then you're probably a risk worth taking.
(Actually, ignore everything I'm saying. I don't want the rest of the market to catch on. :)
If you hire an electrical engineer, do you test them on understanding Ohm's law?
If you hire a window washer, do you test them on cable strength integrity questions?
Yes, an interview in English is an English skills test.
Programming has the odd problem where you want someone that speaks code so you need to ask them to speak it in some form or other. Yet, if you ask someone to demonstrate even the most basic form of competence some people lock up.
IMO, the best option is to give programmers the same kind of fuzzy test your giving HR managers. You don't need someone that can decode for (int i = -6; i++ < ++j; k++) ... but they need to be able to understand how to use a loop to get something done.
If interviewers dealt with candidates from any other professional field in the same way that software professionals are commonly interviewed, there'd be a ton more people walking out of interviews.
Maybe this isn't universal, but I'm used to most applicants being hopelessly unqualified.
2. Damn right (more specifically I'd ask some kind of basic problem about circuits)
3. Of course not - don't be ridiculous. I'd test them on their knowledge of chemistry, fluid dynamics and celestial mechanics (duh).
1) No - you screen those out with spelling mistakes in their resume
2) Maybe? I'm not sure - I'm not an electrical engineer. Then again, credentialing an EE is probably easier than a programmer.
3) This one's not even a good analogy - of course not.
What makes programming a little different is that it's turtles all the way down. While you might not need to find the average of an array, you will need the ability to reason about such things. You'll be reducing a collection of items, or mapping them or summing them.
You basically must deal with abstraction as a computer programmer. (And I'd argue that yes, assembly deals with abstraction)
3) It's a very apt analogy. Tons of especially lower level candidates are doing weird tricky CS problems at interview even if they're building yet more basic reporting systems.
Regarding abstraction, why don't you ask them to describe projects they did, instead of grilling them on concocted problems you know are beneath the level of person you wish to hire?
It is well known that this companies test for this knowledge. They even tell you about it during their process so you will be ready for it. At the very least, it tests that you can learn this things from study. If you fail at them, you either did not prepare in the way they told you to, which is a failure to follow instructions, or you tried to study and it wasn't enough, which is a failure of ability to learn. There are probelm with this; the conext on which you are evaluated is vastly different than the environemnt wher eyou will ahve to apply it. But that doesn't mean the assesment is completely random. It tests for some qualities that will be important on the job.
Maybe it's just me, but when someone uses the term expert, I tend to be doubly suspicious. Especially if they use that for more than one or two skillsets.
Firing people frequently is also expensive and it's demoralizing for the team. You also don't earn loyalty by showing that you have none yourself.
That should be an impetus to improve your hiring process, assuming it isn't a rare occurence.
> They probably had a job before they accepted your offer. Now they're unemployed and it's your fault.
It is not entirely your fault, unless you didn't effectively articulate what the job was. Otherwise, they took a job they knew was over their head.
> Firing people frequently is also expensive and it's demoralizing for the team.
See point #1. Firing people frequently is a sign your interview process is broken.
It's mostly your fault. You know what the job is and what your expectations will be far better than the candidate. You should be hiring only when you are confident that they will be able to do the job.
> See point #1. Firing people frequently is a sign your interview process is broken.
Yes, that's exactly my point. "Fire fast" as a strategy should not replace "hire well". Your hiring process is entirely messed up if your strategy is "tasks risks in hiring but fire fast".
Sure, if you made a bad hiring decision, you may need to fire someone. But your hiring process should in general be filtering out poor candidates.
Businesses either bias their individual hiring process towards false negatives or false positives, but no one has apparently figured out how to find the balance.
Firing people frequently after a short term of employment is a sign that the hiring process (interviewing is part of that, but perhaps not the problem part) is broken.
Firing people frequently but long after they were hired may be a sign of hiring process problems, but is more likely a sign of post-hiring working environment problems.
(Well, actually, in both cases those assume that the firing decisions are, themselves, justified in the individual cases; obviously, either of the above could be a sign that the firing decision process is broken, instead of some other process.)
I think it largely depends on how do you fire people.
If you just decide 'fuck it, he's out' than yeah, that's awful. But if you give a chance to correct the issues, and actually give the chance rather than imposing some impossible requirements with your mind already set, that's different. Sometimes you just need a little push.
>>> Firing people frequently is also expensive and it's demoralizing for the team
Not sure about the expensive bit (it's better to fire sooner than later if you're going to fire anyway), but also concerned about the morale. Unless that's a very rare occurrence for quite obvious reasons, it will no doubt demoralize the team.
There are few things more demoralizing for a team than keeping jerks around.
Unfortunately, jerks are an overrepresented demographic in programming.
Yes, sometimes you will need to fire people. But the strategy should be to hire the right people in the first place.
I agree that the latter version is wrong, but there is room to adjust how much you work on predicting future performance. How much of a jerk people are is one of the harder things to predict in interviews.
I get the feeling we're both speaking with previous unfortunate experiences in mind :)
In general, I tend to prefer people who are able/motivated to learn, as opposed to X years of experience in keyword Y.
That said, you also have to have an understanding of your interview process: no brain teasers, no language lawyer questions, etc. A peer reviewed collection of interview questions and a rigid methodology to make sure time pressure/stress/etc. are not part of the interview.
This might be why a lot of companies use these tests. Because in some places, you need to filter out 'bad' hires before it becomes difficult to dismiss them.
Within the first few months of employment (or less/longer, as set out in a contract), the employee can be easily fired. They want similar reasons to a normal firing (underperforming, inappropriate behaviour, etc) but the information I can find says that the bar is set much lower and they're more just worried about you "acting in good faith".
I've rarely (if ever) seen it used. Usually people want to take the time to try and develop the employee, but they end up shooting themselves in the foot. If you hire someone that's not working out you can let them go pretty quickly. Once their probation is up you're stuck with them. So fire fast, don't dick around.
I've seen this from the other side, when I accepted a job that turned out to require skills I didn't have. It was an unbelievably crappy experience. Prompt action -- within the first six months, maybe -- would have saved a lot of grief.
EDIT: Side note, we also generally build the interview questions, then write the ad. Since we already know the answers we want, the ad comes fairly naturally from that.
> I think in any field it's important to decide on the desired answer (from what traits and skills you need), then build the question.
My previous job liked to hire by committee that was created by randomly selecting employees to participate with hiring. (already a mistake) Even though these included the department head actually hiring, we'd often find that most of the time in meetings was spent asking or telling each other what was "really" necessary for the job, since even if we had a solid job description and list of wants, the committee members would squable since the jobs were allowed to be too flexible. When it came time to design questions, we would run into the same problem each time: We were more focused on "good questions" than deciding what we really wanted the answer to be and designing questions around it. This often resulted in really poor interviews where we let applicants just talk their way through any of our concerns because there was no concrete idea of what we were looking for.
I think in general this tends to be an issue with employers who have difficulty with hiring - they are not good at articulating their wants beyond "we need a web developer" or "database admin" - undoubtedly it's managers who were told "don't get too specific with your wishlist", the idea being to prevent job postings with absurd requirements, but it's swung too far in the opposite direction in many cases with employers relying exclusively on keywords as indicators of success. ("They know X language and have worked with Y hardware - they're perfect")
The hard truth is that even though hiring sucks and can take forever if you're not careful, it's worth taking the extra prep time to ensure you have a clear idea of the type of hire you want. Anything less is a disservice to your company, your employees, your customers, and your new hire. The HR hatchets are generally not useful either, as they're too broad sweeping and the same tools that weed out the absolutely unqualified also end up weeding out those who haven't jammed the right keywords into their resume. Neither ends up getting you what you want; while it's shitty to have to read potentially dozens or hundreds of resumes, if you're struggling to fill positions with good people, it's a necessity.
Generally, good people are always willing to learn, and challenge themselves, and listen to others, and doubt themselves sometimes, but not too much, and act according to their values, and have a vision and enjoy life.
Find as many of these traits in someone, ane you would have found a great partner.
Admittedly, judging ability is very subjective and hard to do, but it's worth pointing out that judging someone's ability is fundamentally different from keyword matching.
With keywords, the only possible answer is a yes/no. Do you have Java experience? Yes/No. Do you have VP-Sales experience? Yes/No. Do you have 10 years of work experience? Yes/No.
Keywords provide no judgement on how good a performer someone is. They only tell you what that person has done in the past. Anyone can spend 5 years programming Java. But does that mean he's going to be a good Java programmer during his spell at your company? The keyword provides no way of answering that question.
Judging someone's ability on the other hand, even if it's imperfect, even if it's flawed, at least makes an effort to understand the person's potential capabilities. And that alone, makes it superior to keyword-based recruiting.
(or at least that's the argument the article seems to be making)
Asking brain teasers is fine as long as that's what your technical people actually do all day -- sit around and ask each other wacky logic problems they half-remember reading about on the internet somewhere, continually trying to one-up another (and if you don't answer with 90 seconds or so, you get fired, on the spot).
But for most software engineering roles, not so much.
What's really cool about it as well is that he expects both the candidate and interviewer to read the same book. Pretty powerful.
Other than a brief flash of insight that you can do something like: 37^2 = (30 + 7)(30 + 7) = 900 + 210 + 210 + 49, I'm not sure what this was supposed to test. It was a fairly small company, but I'm sure they could afford a calculator or two....
> Industry experience is often a negative, not a positive
I can see why someone would say this. If you looking to innovate, you don't want employees always saying, "This is just the way we've always done it." At the same time, people with industry experience can avoid novice pitfalls.
Also, I cannot stand all the emphasized keywords. I get why he did it, but it's so jarring to read.
The only comparable jobs are high impact sports. An older boxer or football player is assumed to have accumulated injuries that make them a worse value proposition. Are devs the same? Am I slowly accumulating the equivalent of career ending sports injuries the longer I am stuck in a job living like a Dilbert comic?
Probably not. It is really just ageism, and my work history doesn't matter to these hiring managers as long as I'm over 30.
> We ran circles around competitors who had teams of B and C players with security expertise but little else going for them.
He feels experience is negative (I'm guessing) because the experienced candidates only have experience. It is implied that they have no other team skills - just their knowledge of the area. It doesn't matter if you hire the world's foremost expert; if they don't mesh with your team, everyone is going to have a bad time.
But all other things being equal are you still suggesting that experience is a negative?
Problem is, now that I am older I really don't want to be stereotyped as one of those dinosaurs.
I've met people with over a decade of experience in programming and had to convince them that hey, maybe we should write unit tests because customers keep complaining about bugs? Had hard time in doing it because hey you just refresh a browser and see.
Well, you get the idea. :)
The question isn't whether experience helps. It's whether experience helps more than the worst part of the applicant filter that you're using. If it does, you should use more experience-based filtering. If it doesn't, you shouldn't.
That said, a hard pass fail filter is a bad idea. Much better to score and rank.
I can believe that experience of doing (say) chemistry 200 years ago might be so different from current practice that it would be of negative value. Tech changes a lot faster than that. Certainly in practice I've found working with experienced people worse than working with fresh grads, because fresh grads tend to be empiricists, whereas experienced people tend to have fixed technical views that are often outdated. (Of course this is just my subjective experience).
> The only comparable jobs are high impact sports. An older boxer or football player is assumed to have accumulated injuries that make them a worse value proposition. Are devs the same? Am I slowly accumulating the equivalent of career ending sports injuries the longer I am stuck in a job living like a Dilbert comic?
Older mathematicians are famously less effective. I honestly feel like I'm less good at my job than when I started. Certainly experience with outdated tools/paradigms seems like a net negative - it will mislead you when working with modern tools/paradigms.
They don't like this idea because they want to believe their idea is new and groundbreaking. It may very well be, but the pieces aren't. That's where experience comes in.
I guess you're right, it's ageism, but it's also just being 25 and being so sure of everything that you couldn't possibly learn anything from "industry experience".
45 is pretty much right in the middle of the adult years. (Hence "middle-aged")
This is such a horrible misconception that I think you've never worked with senior people in your life.
> Industry experience is often a negative, not a positive, for a new company. Could Hollywood execs have started Netflix? Would a book publisher had started Amazon? Jack Dorsey, not Visa execs, started Square. He admitted, when asked how he learned about the world of payments, “We Googled ‘accepting credit cards.'”
He clearly talks about founders and high-level employees, and provides examples. An obvious way of countering him would be to provide counter examples -- which truly innovative companies have been started by industry insiders?
> There are some industries where getting up to speed just with the basic domain knowledge so that you don't have to google every other word/tla in the user story takes a good amount of time.
My point is that people with experience can help spur along development. I'm not calling his statement false, because I know of some company that did it better my way. I disagree (perhaps erroneously), because I know there are not-so-obvious mistakes people can make in this industry that eat up resources.
Getting a bunch of greenhorns together for sake of out with old is probably not ideal. You want people with experience to help avoid traps that prolong your process. Security comes to mind. Security is notoriously hard. And while you may not want to hire only security gurus for your business, it absolutely helps to have someone with security experience in your stable. From the article:
> We ran circles around competitors who had teams of B and C players with security expertise but little else going for them.
That last part is the issue. The employees had little else going for them. Your people with experience should have other qualities going for them. In addition to knowing security, they should be team players, open to new methodologies, effective communicators, and all those other soft skills that make teams function well. A team with no security experience can learn what needs to be learn without issue. I do not deny that. However, expertise doesn't automatically imply that everything else about the candidate is garbage, and having employees to give guidance helps avoid growing pains.
Yeah, there are some people in various industries that either couldn't see approaching issues or just flat out ignored that they could be issues.
But there are also people who saw say, the rise of services like Netflix as a positive or likely thing. People in the publishing industry who saw the possibility of selling books online or replacing them with eReaders like the Kindle.
They just might have not done so for other reasons. Perhaps they saw what was coming, but thought they didn't have the entrepreneurial mindset to start their own company. Maybe they did start a company like this up, but they didn't find the right connections or investors or anything else to catch on.
Always been bothered with the assumption that those in 'disrupted' industries are too stubborn to know what could replace their business.
If you already know Java and Javascript, you'll get Scala / Kotlin in no time. If you're experienced in Python and Django, RoR is a few weeks away. If you're experienced in C++ and Python, it won't take long until you're good to Go.
I get that companies want someone who's productive from Day 1, but they're missing out on a lot of great candidates that might only be productive from Day 15, but they'll be a lot more productive over time.
I agree that a competent, experienced developer can do this. What's worth noting, however, is that it takes two weeks to cross the bar of "can contribute code in this language that produces deliverable results". But two weeks cannot get you across the bar of "can contribute robust, maintainable code in this language".
Even though you are not a beginner to programming, you are a beginner to the language and its ecosystem and so will still make some significant newbie mistakes. Ruby on Rails is a good example of this: half of being good at RoR is being a good programmer, the other half is memorized knowledge of the idioms and patterns of the framework and community.
Unfortunately, this is so true and yet so difficult for people to realize and understand.
I've seen in real life wherein somebody hired at a supreme decision-making authority comes in and says "We'll rewrite everything in language X". Existing folks in the team quickly start ramping up on the language and simultaneously writing production code. Much of the code thus written tends to be pathological.
So do 10 years of experience. "robust, maintainable code" is just coding convention that a group of developers comes up with. The convention can vary from one team to another.
That sounds about right for many programming languages. My personal average from "having heard of the language" to "writing non-trivial programs" (let's say, something that is able to talk to a REST API and do some processing) is a non-stop weekend programming session. [1]
The time-consuming part is learning the libraries and how to write idiomatic code. That can take months or years (C++, I'm looking at you).
Still, knowledge of a specific programming language is usually the wrong thing to optimize for. Perhaps a more useful metric was the number of programming languages one has been exposed to? That is more useful, as once you have learned a bunch, you've seen most of the important concepts, and can focus on whatever subset is implemented by the new language.
[1] Except Haskell. That's a nut I still haven't cracked.
Filtering these clowns out should be rather simple (make sure their resume mentions at least a few different languages and ask the right questions during a phone screen), but most companies aren't willing to expend the effort.
One should also be vary of the other extreme: There are people that switch their favorite language every three months, and insist that they use the latest one for their day job. If you allow it, it may lead to lot of wasted time, small wars within your team, and so on.
I prefer Alan Perlis' advice: "A language that doesn't affect the way you think about programming, is not worth knowing."
Edit: I categorize myself as one of the programmers who can learn syntax, but it takes me much longer to understand the language.
The game these employers have set up is unwinnable. Don't play it.
- General mental ability (Are they generally smart)
Use WAIS or if there are artifacts of GMA(Complex work they've done themselves) available use them as proxies.
- Work sample test. NOT HAZING! As close as possible to the actual work they'd be doing. Try to make it apples-to-apples comparison across candidates.
- Integrity. The first two won't matter if the candidate is a sociopath.
[1] http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%...
I bet I could be a perfectly fine RoR developer, given a little bit of time. I don't know Ruby, but I pick up languages quickly. Or, I mean, you could just keep the position open for a few months and just screen for keywords. It's really your choice.
They are literally searching harder and waiting longer, to pay more for a narrower skill set.
Good luck finding that "javascript applications developer" or "ruby on rails engineer" or "SPARK/RTEMS engineer", guys. If you needed someone to be productive today, all you had to do would have been to hire them last month and give them a $200 book budget.
Where I work the first two weeks are explicitly training. You will not contribute to the bottom line at all, even if you are an expert already. And sometimes this feels like not enough.
Every PHP developer who I've seen pick up a nodejs project has end up fighting against asynchronous code and promises. You're fortunate that you were able to grok that aspect of node easily.
As others have said, the article identifies a big problem in hiring. However, it doesn't give you a lot of handles to solve anything.
The idea being expressed here is that if you bring in a new employee, even one that has no previous experience with the technologies being used, they will perform best if they are naturally in tune with the company's ideals and methodology, and can quickly develop camaraderie with the existing crew.
On the other hand, if you hire someone that knows all the necessary tech, but is constantly at odds with how things are done, alienates their coworkers, or manages to clog up the process in other ways--in those scenarios, everyone loses.
It has nothing to do with a cultural hive mind; it's just a matter of keeping a team synchronized and efficient.
See some related HN submissions and threads: Inside the Mirrortocracy https://news.ycombinator.com/item?id=7930430 ; Guess Who Doesn’t Fit in at Work https://news.ycombinator.com/item?id=9789928
*Edit: Okay, I read both of the articles, and they were fairly interesting, so thanks. I do have a pet peeve / complaint though: Both of the articles purported to link to research, but neither actually did. the first article linked to a list of slide decks and the second article linked to a summary of research. I've read enough bad summaries now that I am working on becoming a primary sources junky. Do you know of any research on this issue that's been done?
How do you measure that? By seeing if they listen to the same types of music or drink the same types of beer?
> they will perform best if they are naturally in tune with the company's ideals and methodology,
Oh good Lord, how are you ever going to measure this?
> and can quickly develop camaraderie with the existing crew.
Aw, shit.
Basically, if you're the kind of people who will deceive customers, throw your coworkers under the bus, etc., we don't want you here. It seems to work out well for us; we consistently win awards for both customer satisfaction and employee satisfaction, our net promoter score is industry leading, and it's incredibly common for new hires to come from employee referrals.
For example: http://goo.gl/forms/YICrUuz7XNjiTcId2
Here's one of our software engineering forms. It's worked extremely well at filtering for technically qualified candidates. But we worry that we're still missing people because https://imgur.com/RIamYIZ
Eh, that's a pretty reasonable questionnaire. If it was pages and pages, it'd be too much, but there's only 12 questions here (including the citizenship and email address questions) and I'm assuming you don't expect an essay for each of these. Someone who's indignant at the idea of filling this out is probably not worth working with.
I have to say, I find the SSH question at the bottom odd. Who is this screening for? Are there people who hit all your other requirements but don't know how to use SSH? Would you actually eliminate someone for this, as opposed to spending 5 minutes introducing them to SSH (or 30 seconds sending them a link)?
Would the resume be used for the actual screening/selection process, or is it instead just to get all of the other details that might be necessary later in the hiring process as a catch-all?
In Toronto, I keep seeing all these job posting of companies that need people with 10 years in social media experience, or 10 years in mobile app development and it's quite laughable how they are able to hire those people.
I'm curious how those people can actually perform.
But as far as I can tell, the candidate crams for the tech screen, learns the basics between offer letter and start date, and picks up the rest as they work. The dirty little secret is that nobody actually needs any specific experience to do the job. You just need a decent brain and an Internet connection, and just enough people skills to fake it until you make it.
But it probably works better in larger organizations, where none of the people who would be able to detect the deception are involved in the interview process. You don't need to fool your peers if you can fool your boss.
It never would have worked in any of the smaller companies that I have interviewed with, as they have all (perhaps coincidentally) been ultra-paranoid about resume frauds, for some reason.
Very true, and this is why these keyword searches are so absurd. It wouldn't be difficult to setup the entire hiring process (applications as well as interviews) around that dirty little secret.
I've worked with a fair number. They refused to deal with people except those that told unreasonable lies.
The results were never good. But they were satisfied with the results because "it's not my fault. That person lied to me."
Makes me skeptical to store money with chase.
javascript:document.styleSheets[0].insertRule("strong%20{%20font-weight:%20normal%20}")
No one hires by keywords, they filter resumes by keywords. If you're smart enough, you will load up your resume with keywords if that's how your target company is filtering resume by. You just want to get to the next base which is getting an interview.
How do you get hired after an interview? Some companies will give you tests, puzzle, white board coding, all conversation/culture fit, etc. You likewise have to be a fool not to adjust accordingly. If you want to join Acme corp so bad and they require you wear a suit, wear a freaking suit. If they are laid back and cargo pants and t-shirt will do, you have to be an idiot to go in with a suit. If they are big on whiteboard coding, then practice so and do load's of exercises prior to your interview.
game theory 101, people will always look for ways to maximize their outcome.
I cringe when a recruiter wants to know which language I program in and he/she does not get it immediatelly when I reply "any that you like" or "what ever the customer wants".
Keyword based job match is however the only automatic solution, all other are to work intensive so that it makes economically sense.
When the VPs and execs panicked, they started at the hiring level... They made a plan to "stop hiring junior level" employees, and our team became full of Senior level people. Then they said we need more PhDs... because a PhD is prestigious and we need prestige. There was even a brief mandate to high an industry celebrity type, basically just for show. It all started with a "hiring by keyword" mentality and pretty much led to a massive decay of passion across the entire team.
* outsiders can bring some fresh thoughts and challenge ideas that everyone inside keep as status quo and unchallenged;
* people who define themselves as e.g. 'Ruby on Rails dev' (or JavaScript dev or whatever) might have hard time adapting if the landscape changes, or they need to do something outside their comfort zone;
* specific technical skills (programming language, etc) are usually easier to develop than soft skills such as leadership, teamwork, communication.
Of course, it's not unconditional. Sometimes you really need someone who has deep knowledge of some niche, or is really a master at some technology. It's just probably not as common as one might believe.
Even if I can quickly grok the equivalent, I'd get frustrated after a months/years of solving problems using my less preferred stack. The stack can also tell you a lot about the philosophy of the shop: flexibility vs. convention/readability, stability vs. cutting-edge.
Some recruiters, not all, are basically no better than a robot. I don't have a B.S. but I have an A.S. plus 10 years of experience now. Give a recruiter a req of "B.S. or equivalent experience" and they will turn me away without a phone call because I don't have the degree. When 10 minutes on the phone is all I would need to convince them I have the experience they need. I've basically given up on interviews for awhile. It drains so much energy just to get canned responses back.
It's like aiming to hire someone to build a high-rise out of wood and going "We need someone who can use a hammer and nail". Not really wrong, but it's kind of an odd emphasis.