Has there been any research into the predictive power of such assessments? Is there any evidence that a particular type of question correlates well with job success?
> The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years
> This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed.
https://www.semanticscholar.org/paper/The-Validity-and-Utili...
Even jobs that don't seem cognitively demanding, like janitors or infantry. Higher IQ candidates almost universally do better, even if they start with much less experience.
The takeaway as it applies to modern software hiring, is that skillset is way overemphasized. It's quite common to see job ads that heavily focus on the company's specific tech stack. ("Must have experience with Rails, React, Travis, and AWS")
It's much better to cast a much wider net, and try to find the smartest people anywhere. High IQ people can easily re-tool their specific skillset. What's interesting is this is much closer to how the most successful tech firms, like Google, tend to hire.
And obviously applies in lots of areas. If I'm looking for someone to head up a digital marketing or marketing research initiative, someone whose only work experience is software development is probably not the best choice no matter how smart and motivated they are.
OTOH, as you get to the level where people are more managers than practitioners, their specific skill sets presumably start to make less of a difference.
Added: And, as peer noted, people do shift careers significantly all the time. I never really directly used my undergrad (or grad) engineering degrees all that much.
Having switched professions twice, I now use practically nothing of what I learned in university from about second year forward. Math and to some extent physics are still relevant, but that's pretty much it.
It might be painful to admit, but the practical value of that rather specialized knowledge that took me several years of hard work to obtain is pretty much zero now.
It's sad that your professors didn't tell you beforehand that you were going to learn how to learn during that time, not just study a particular technology stack.
That said, you need tools of some sort if you're going to actually build things as opposed to just learning, say, algorithms in pseudo-code. And it probably makes sense to use some fairly standard language to do so. There's not much point in making things deliberately obscure by making students use some language that the professor designed for his PhD thesis.
The waste was close to total, but this is only obvious in hindsight. Given the information I had at the time my choices were rational.
I haven't read the research myself but if they are only looking at interviews, this could be explained by the fact people with absolutely no prior knowledge would be filtered before reaching the interview.
While Alice may have fewer years of experience than Bob, she might have effectively more experience if she absorbed understanding at a faster rate. An unexperienced, high-intelligence person usually starts at an initial disadvantage, but manages to "get up to speed" quickly.
This also underscores the particular importance of intelligence in software. The field is constantly awash in new technologies, where nobody has had time to accumulate extensive chronological experience. So, it's really important to find people that can absorb new concepts quickly.
[1] https://www.sciencedirect.com/science/article/pii/S019130851...
In other words, structured interviews are better because they're less noisy measures of intelligence. The takeaway very much is IQ uber alles.
[1] https://digitalcommons.unomaha.edu/cgi/viewcontent.cgi?refer...
From the 1998 meta-analysis
> The most well-known conclusion from this research is that for hiring employees without previous experience in the job the most valid predictor of future performance and learning is general mental ability ([G M A ], i.e., intelligence or general cognitive ability; Hunter & Hunter, 1984; Ree & Earles, 1992).
> Work sample measures are slightly more valid but are much more costly and can be used only with applicants who already know the job or have been trained for the occupation or job.
http://www.flume.com.br/pdf/Scmidt_The_validity_and_utility....
From the 2016 meta-analysis by the same author.
> Overall, the two combinations with the highest multivariate validity and utility for predicting job performance were GMA plus an integrity test (mean validity of .78) and GMA plus a structured interview (mean validity of .76)
Work sample tests do not work as well as the old research suggests
Table with effects linked below
More recent paper
https://www.google.com/url?sa=i&source=web&cd=&ved=2ahUKEwi1...
I do know that cognitive abilities and long term memory abilities aren't correlated. Cognitive abilities and short term / working memory are positively correlated though.
A little bit of preparation helps a lot, but beyond a certain point preparation doesn't help anymore. All interview questions require solving a new problem in the interview itself.
Depending on the interviewer and question, the difference between a hire and no-hire recommendation can be quite marginal. There is a huge luck factor involved.
A SCOTUS ruling[1] has found that IQ tests are assumed to disfavor minority employees, and therefore using them as a major factor in hiring decisions may run afoul of the Civil Rights Act. There are ways around this, but generally speaking you have to prove the specific test you are administering either does not disfavor minorities, or else show that it is directly related to the specific position you're hiring for.
[1] https://en.wikipedia.org/wiki/Griggs_v._Duke_Power_Co.
I am not a lawyer, do not take legal advice from randos on the interwebs. You should consult with a Real Actual Professional on how to deal with this risk. There have been both laws and court cases since the above that impact the ruling, which I am not qualified to analyze.
But that is so hard: we have pride, we have "company culture", we are stubborn, we are talking about "A players", etc. etc.
So yes - try to make interviews structured and organized so that there is no random noise.
That reply button's in italics?!
After meeting my PhD professor. I stopped trusting academia for truth.
Now I skip the abstract and go straight to Data.
If I am wrong here, someone please correct me.
Bias creeps in in other ways, too. I have done both on interviewing.io as a candidate. You probably remember how Instacart has interviews on your platform under the terms that it was just an informational chat, and anyone who met the criteria to interview would then get a real technical interview. I did the informational chat and was then not interviewed after deanonymizing. And, BTW, after I emailed support I never got my technical interview, nor was I given any sort of resolution or information about what happened.
It’s a good experiment, but it falls short of what I’d call “anonymous” and doesn’t really remove very much bias in the overall process.
I have a longer reply to a different HN thread with more detail about my experiences here: https://news.ycombinator.com/item?id=11679844
Testing in an environment completely different from 'production' isn't going to give a super strong signal on collaborative problem solving imo.
How can you tell? Is this always based on how you experienced the interview, or do you ask candidates afterwards how they felt?
(no insult intended, but) you made an empirical statement, and I'm curious where your evidence is from.
I failed 100% of them with various, strangely random feedback by HR/recruiter.
I've come to conclude it's the hiring mechanism of social incompetents / unempathetic sciency types. And it was always a dev from another group than the one I was applying for.
No thanks.
I recently started pointing that out, in a roundabout way, when I run interviews.
Why? If a candidate starts arguing about xml versus JSON, the candidate misses the point when I bring up xml. (Hint, I only discuss xml because it allows discussing widely known concepts that are very general and have little to do with xml versus JSON.)
Or, if a candidate says to me, "why can't we use async-await" in a particular question, the candidate also misses the point of the question. (Hint, I'm trying to test knowledge of concepts that's easy to test when doing something without async=await.)
Interview questions are always contrived and end up as an "if you want to work here, you are going to do what I say" exercise. If a candidate can't work within the constraints of a short interview exercise, then what will happen when the constraints of the job come into play? We never get to use our favorite patterns and APIs all the time.
</sarcasm>
It seems gaming trivial metrics works in this case.
2. No one has been able to come up with a better alternative.
I wasn't implying its a prefect metric or even a reasonable one. If you really need a quantifiable metric then thats all we got.
Most of the time, what people want isn't even what they want.
I figure that eventually it'll be similar to the Voight-Kampff test and kept on file forever while being shared with all potential employers.
"Because they have to kill their own kind, they constantly need to be assessed as to whether their work is having some kind of moral impact on them'"
Also, Google's former head of people (Lazlo Block) founded a company that is focusing on that area: https://humu.com/.
While I wasn't always a fan of Google People/HR procedures while I was there, I did always appreciate the data-driven/research approach Lazlo and the People team took.
In general, there's a lot of research, though with some weird gaps, and a lot of results (though not all) are inconclusive.
A noob could be successful hire and not ultra productive compared to say a more experienced hire.
And that doesn't account for all the places hiring / think they need / hired "rockstar" developers .... to maintain their crud app.
Of course software engineering research in general has always been a little unimpressive because what you really need is several teams completing projects, some using method A and others using method B. This could cost millions, and would only settle the bet between methods A and B.
Lacking this, we're left with the publically available research that relies on unconvincing proxies for success and very small samples. It's better than nothing but far short of what we ought to have given how important software is.
Even worse: it would only settle the bet between methods A and B in a particular context.
To be able to truly predict whether or not someone will be successful means that the interviewer would have to able to foresee how that person will interact with others in their workgroup , rise to challenges that don't yet exist, and be motivated to remain for a "long-enough" engagement with the company (whatever that means). Predicting the future is just damn hard, and it's even harder when nebulous desired outcomes such as "job success" are used.
To make matters even more difficult, this problem also has "another half" to it which people tend to ignore: the employer doing the interviewing.
These types of questions seem to take the point of view that an employer is presented with a bowl of fruit (candidates) and all they have to do is be able to select the best fruits.
But it doesn't work that way. You may not get the fruit which you want, that fruit may want to go elsewhere or others might have already grabbed the fruit you would have wanted. The fruit which looks good now may end up rotten a short time later. Some fruit that looks undesirable today, may be awesome later. You may grow tired of apples and desire pears, but you've already filled your pantry with apples.
What would happen if some organization was able to "figure this out" and truly maximally optimize their candidate selection using interview techniques? I am not so sure it would be easily distinguishable from what other similar competitive employers are doing. Predicting the future can only go so far.
In my 12+ years career, I've worked with so-so engineers, but never truly bad ones that would ruin a project. And even the ones that weren't great, what damages did they really cause? I've seen many more companies failing because of a bad product, bad sales strategies, bad market fit, bad business model than engineering teams using the wrong programming language, or not building microservices the right way.
I find this constant obsession around identifying "rockstars", avoiding "bad" engineers at all cost to be truly unhealthy.
There's no way to guarantee a successful engineering hire, but organizations try to because their culture is so blameful or fearful of conflict that all the incentives are misaligned.
I believe most organizations copy the Google, Facebook, etc. hiring processes with the goal of adopting an "industry standard process" because they don't know how to conduct an interview process that's tailored to their own organization, and they don't want to think too deeply about it.
Having a coherent interview process means dealing with your organizational baggage: understanding it and either resolving it or crafting the interview process to select for people who can integrate well with it. Usually it's the latter, since most of the issues start at the top.
Google, Facebook, etc. have spent a lot of time and money crafting a good interview process for their organizations: specifically what they value and don't value (even if the individual interviewers aren't always fully aware of what the organization is selecting for). They can also support a lot of false negatives.
But the copycats don't have those same needs or candidate pool, so it always comes across as a bit disconnected to me.
I would say bad engineers tend to know they are bad and either tend to go into managing, coordination roles or grab a subset of tasks they get good at, like being the only one on the company working with eg. a specific third party system and doing support and integration for that.
Bad engineers that have worked with the same system for three years are way better with it than super-engineers that have never touched it for at-least a couple of months.
As you said, I have never really had problems with bad engineers messing things up.
EDIT: Reluctance to hire engineers and saving pennies for a dollar on the other hand have messed up some projects.
It's when that bad engineer starts protecting "their" turf at the expense of the rest of the org in order to provide themselves with job security that things start to really go south.
or you can just waste all the time firefighting with consequences of bad software.
it's not the only factor, but it can matter. the majority of companies i worked suffer from this. what really mitigates the problem is that competitors have the same problem on average.
Also, there's no one size fits all theory. There are companies where every ms of time you save by writing a better algorithm makes a big difference and most where it doesn't matter. Take home assignments, pair programming have all been tried and is still used by some companies, but majority still rely on white boards and it seems to work fine.
Probably not "a lot", but personally I have _only_ worked in companies where they are our bread and butter (Biotech and finance.)
If someone is fresh out of a 4 year program that was all about data structures, algorithms, operating systems, etc, then demonstrating mastery of those topics tells an employer that a person is smart and can learn large amounts of complex technical material.
How do we know that? Interviewing is near a total crapshoot from what I can tell based on my 13 years of experience (~8 of which being actively involved in the process, 3 as a decision maker.) I have yet to find a method that weeds out people who can whiteboard, but can't deliver, or those who seem to have a great personality and work ethic, but stop showing up to work and/or throw tantrums when they don't get their way.
Obviously some people emit red flags like the Sun emits energy, but I'm I don't believe you can assume it's "working fine" without some baseline definition of "fine" and a corresponding study.
Because businesses are still able to solve problems, service customers, etc. Of course, there is room for improvement.
As the sibling comment mentions, companies are able to build large scale systems by engineers who got in through these kind of interviews. I am not saying this is the right way or wrong way, but it definitely works.
Of course when you do this, you miss out on some amazing candidates, but the big companies which started this can afford to do that because of the insane num of applications they get.
A professor I know from undergrad did research which could essentially evaluate, with some adaptation, whether or not people would work well in software engineering situations. More centered around whether team members would work well together, but relevant nonetheless: Malte Jung
This has consistently been my experience.
Nevertheless, the ability to code is necessary but not sufficient. You're going to want to dig into character at least a little.
- How are they in a team context?
- Do they treat others with respect?
- Do they act with integrity?
These kinds of attributes are just as important as whether or not they can code, and I've been badly bitten when they've been lacking.
I’ve seen some programs do away with references, resumes and even interviews.
The programs that I’ve seen remove interviews usually did replace them with standardized stations with different scenarios.
E.g.: “Explain to someone how to wash their hands if they never have before”. If you can do that well, you can probably explain some health procedure you’ll later learn about and have to explain.
A good student will probably continue to be a good student.
But how to select which students will become good practitioners may be a different story.
And you're right. It's not really surprising that getting good grades/test scores at one level of school correlates reasonably well with the same thing at another level. Which I imagine is one reason universities/grad programs generally don't optimize purely on that metric because their objective isn't solely about cranking out students who study well in school.
ADDED: This, by the way, is more or less just a variant of general intelligence measures being correlated to a lot of outcomes. The SAT and so forth aren't intelligence tests, but I'm sure they're very correlated.
I imagine it'd be really hard for an academic research institute to do so. Firstly, they'd have to focus on technical interviewing in specific. Secondly, they'd have to get some big tech companies to share information about trajectory and whatnot for employees once they've been hired -- I imagine the extra work of building those pipelines/anonymizing the data/etc probably wouldn't be worth it for a lot of companies.
[1] https://erikbern.com/2018/05/02/interviewing-is-a-noisy-pred...
What makes a great software engineer? https://dl.acm.org/citation.cfm?id=2818839
HN discussion: https://news.ycombinator.com/item?id=15892898
There is likely to be internal research at some companies.
My suspicion, however, is that these kinds of problems have limited predictive success and yield a lot of false negatives.
https://medium.com/@cspa_exam/a-guide-to-building-your-own-t...