Making software engineering interviews predictive of job performance | Hacker News Reader

Making software engineering interviews predictive of job performance

qualified.io

131 pointsgeoffroberts109 comments

In the end, it's a crap shoot... Even their highest marker of validity is only about 50%.

In the end, if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel. Market pay will also vary wildly and dramatically, and perception is equally varied.

Where I work now, we have a small pre-interview code challenge... it's not particularly difficult and in general should be resolvable in under an hour. I've seen questions on the other side, that I've literally spent 3+ hours on something that said an hour. I suggest those coming up with these things actually implement their challenge on a clean computer (only tools, no reference code) and time themselves. I now cut myself off at an hour and submit what I have stating as much.

As to structured interviews, it will depend. It bugs me both ways when I see more generic, low level pattern questions in interviews only because the practical implications are usually very limited. On the interviewer side, it does help to ask some of the more quiz-like questions as it tends to give you a decent baseline of what the person being interviewed understands.

In the end, personal questions along the lines of what a person works on (developers) outside of educational or work settings gives a ton of information on how they work as well as what they know.

Culture, for as much of a B.S. thing that it is, is actually very important as well. A functional team or set of teams needs a few different types of personalities and worker types in order to do things well as a team. Being able to get along together is also imperitive.

In the end.. it's a crap-shoot and understanding the needs, fill and fit are all varied a bit... trying to normalize doesn't help much. Also, business needs and talent pools will vary greatly.

> if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel

And where you set the bar will depend on, roughly speaking, your prestige. Google can afford to be extremely selective when hiring technologists. Similar thing goes for Harvard and the Air Force.

> what a person works on (developers) outside of educational or work settings gives a ton of information on how they work as well as what they know

The usual counterpoint: nuclear engineers and surgeons tend to have hobbies with no bearing on nuclear engineering or surgery. Nothing wrong with a competent technology professional keeping it to work hours.

All teams should be selective. But if you don't have a Google-sized brand and budget, you need to develop a clearer understanding of your teams strengths/weaknesses and needs.

This understanding will enable you to figure out which hiring criteria do you need to maximize vs. which do you need to satisfy and where the bar is.

Also, by interviewing more effectively, you punch above your weight by spending more face-to-face time selling your company/team/role.

I love this point—the more time you can spend selling the candidate the better. And I think the more effective your interview process, candidates take notice and probably need to be "sold" that much less.

Great points - there are things we're trying to address over at Interview GPS. We're building a Wikipedia/Stack Overflow-style knowledge base of the best interviewer guides/rubrics: come join the cause at https://www.interviewgps.com/community/

We tag questions in 3 areas:

Personal Values: instead of "culture" we help you assess shared values. For example: do you value fun vs. achievement?

Competencies: does your software developer need attention to detail and critical thinking? Or creativity and collaboration?

Skills: what skills are necessary vs. what can be learned on the job? E.g., should you assess for React skills? Or assess for programming/debugging skills in general?

Also, P.S. you have to be careful/thoughtful when asking about what developers work on outside of work. Worst case, you end up breaking the law by discriminating against protected groups. I'd suggest only looking to that if their work history is relatively lacking and you're trying to develop a positive case for the candidate

Love your site—thanks for sharing! And +1 with regards to the potential legal issues.

By what they do outside work/school... I specifically mean if they work on software projects, using skills related to the skills they use at work. I've seen both sides of this and tend not to discriminate when someone has more experience vs. less... it's an offset, more important for interviewing someone out of a bootcamp, or with no formal education.

It really takes all types, so to speak. At this point, I've interviewed and met a pretty broad spectrum. The same goes for on the interviewee side as well.

Again... it's a balance... work history, personal projects (github activity), the pre-interview code challenge, as well as the technical interview are a matter of finding what's needed... someone can be less in one area and make up for it in another. To me at least, independent drive makes up for experience in a lot of ways, especially for more junior-mid roles.

> In the end, if you make it too hard in an employee market, you will have good to great candidates drop off... if you make it too easy, you will wind up at the bottom of the barrel. Market pay will also vary wildly and dramatically, and perception is equally varied.

I mean, you're talking about a different thing than "job performance" here.

Job performance is a satisficing measure: you only need to get someone who's good enough to perform the duties you need of them. (Yes, even in software engineering.)

You're talking about ranking measures. Does it matter if someone is "the best" if they're still not good enough to do your job? Does it matter if someone is "the bottom of the barrel" if they are good enough to do your job? I would say no, in both cases.

IMHO, the hiring bar is way too high in most bigcorps; they care about vanity metrics about having "the best", when 100% of the work assigned to these hires could be easily done by people who are not, in fact, "the best."

The point of a job interview, in the abstract, is to filter out the candidates who aren't qualified to perform the job duties even after a few weeks of on-the-job training. This leaves you with a pool of people who are all qualified. At that point, you should either just hire one of them at random, or try to optimize for a second criterion, such as "is willing to take the lowest compensation to do the job" (which usually translates to "has the fewest formal qualifications to do the job, while still actually being qualified in practice.")

> Job performance is a satisficing measure: you only need to get someone who's good enough to perform the duties you need of them. (Yes, even in software engineering.)

Is that really the case for software? I mean, I'm very skeptical of the "only the best and brightest" nonsense, as I think it selects for the wrong people. But I'm also skeptical of the "eh, good enough" heuristic.

Right now I'm helping a friend deal with a code base that was built by an outside dev shop. It undeniably works; the business has been running on it for a number of years. And honestly, like most software, it's not a complicated app; mostly CRUD stuff. But overall it's a spaghetti snarl. Any given concept might be in the code, or in a config file, or driven by SQL data, or even driven by JSON blobs stuffed into the database, mutated by the main code base, and passed along to the JS front end.

Now that I can see the code, I understand why the dev shop needed so many people working on it, and why all features were hard to do. It's because when you use a lot of mediocre programmers, they all create work for themselves and one another. Not intentionally, but just because they aren't able to see how the differing approaches and sloppy work slowly diminish productivity.

So although I agree with you that a lot of companies hire on vanity metrics, I'm really skeptical that "whoever, as long as they barely qualify" is a better approach.

> Now that I can see the code, I understand why the dev shop needed so many people working on it, and why all features were hard to do. It's because when you use a lot of mediocre programmers, they all create work for themselves and one another. Not intentionally, but just because they aren't able to see how the differing approaches and sloppy work slowly diminish productivity.

To me, that sounds like a leadership problem at the outside dev shop. Properly supervised, properly supported, and with reasonable deadlines, mediocre developers can provide extraordinary results per dollar spent on probably 80-90% of dev tasks.

I'm familiar with the theory, but I've never seen that actually happen. And really, that just shifts the problem. If you need really amazing managers and really amazing senior engineers to make it work, you now just have a different hiring problem. As well a robustness problem, in that choosing a senior person badly means the whole team can go off the cliff.

Personally, I'd rather start with the amazing senior people, hire bright juniors, and mentor them into being amazing developers as well. As well as investing in tooling for non-developers. If a chunk of work is boring enough that a mediocre programmer can do it without risk (e.g., creating reports), it's perhaps boring enough that you can just enable the users (with, e.g., report builders).

> Does it matter if someone is "the best" if they're still not good enough to do your job?

Except we are hiring for a mix of performance and potential - weighted towards potential.

I don’t want to hire a junior engineer that can never make it to senior.

When I am only hiring for performance I hire contractors - and the bar is a lot lower.

Part of the idea of a universal standard in bigcorp is so that you can shuffle engineers between projects as they want to and as your strategy/needs change over time. And for cohesion.

Also I think there is a huge benefit for having generally capable employees, even if they are not directly utilizing many of their talents, and not counting the fungibility: they will make generally better decisions when reasoning about things outside their specific work domain and do better in ambiguous, not perfectly-specced situations. And in software at least, having general/complete knowledge and problem solving ability is absolutely vital if you want to be self-sufficient in diagnosing and solving problems, IMO.

I can't help but object to statements that begin with "The point of a job interview is..."

It seems like we tend to lose the basic reality of gatekeeping and social ritual in these discussions.

It's never going to be empirical. It's how you strike the balance between the empirical part and the social ritual part, and no one ever wants to discuss that.

But I've certainly sat across the table from hiring managers who I knew weren't going to choose me based on a cursory glance, or just seemed like deeply frightened people.

Agreed that interviews are both a gatekeeping and social ritual, but from the perspective of the employer I do think "the point of a job interview" should very clearly be to identify candidates with the highest probability of being successful in the role.

Sometimes when I delegate it costs me more time explaining and supervising than it would to do the work myself.

Sometimes I get back a perfectly decent solution, but in 5x the time I would have considered reasonable.

Most often I get back something that’s not amazing but looks fine, and works at first but turns out to have had some sloppiness or impedance mismatch down the road.

Sometimes I get back not only a solid and thorough solution, but a coworker who went way deeper into the topic than was necessary for the particular task, and is excitedly teaching me new things about it.

From certain people I get back an explanation of why what I’m asking will turn out to be a mistake 8 moves from now in the 10-dimensional chess game, and thing I actually needed instead.

The third group is technically “good enough.” They pull their weight, and we turn a profit on having them around. But if I can manage it, I’ll always try to get the fourth group for junior positions and the fifth group for senior (between those two it’s the same set of people at different stages of development). Those people are maybe 20% of my division, but they do 80% of the work, for what I expect is closer to 30% of the cost.

I completely agree that if you're designing an sort of assessment process, you should go through it yourself or ask other engineers to go through it so it's reasonably scoped in terms of expected time commitment. You can also use assessment tools that specifically limit how much time an engineer can invest in working on the challenge, which is also useful in showing how much progress each candidate made in a set amount of time. The problem with this approach can often be that the candidate feels like they're working against the clock though, which isn't "real world."

That's fair... a few days ago, got a solution (the problem is generally a cli program to transform a csv) that included a full API, data access layer and React ui. It was frankly impressive, and total overkill... turns out the request was unclear from management on our end.

I'm still surprised that almost nobody has used an actual library for csv parsing (naive solutions reading a line and splitting mostly), which to me results in so many headaches in practice.

> In the end, it's a crap shoot... Even their highest marker of validity is only about 50%.

Here's the thing though: all these different tests aren't completely independent. You can combine a couple of them and get a criterion validity value of up to .65.

See https://books.google.com/books?id=IvKYhrXAfRUC&pg=PA198&lpg=...

“General Mental Ability (GMA) tests (like the IQ test) are very predictive of future job performance, largely because people with high intelligence can learn the skills needed to be successful on the job more rapidly. However, due to legal concerns they’re not recommended for companies hiring in the US.”

This is what a classic FAANG interview is: an IQ test which is disguised as a relevant skills test for legal reasons. And they work quite well.

Do they?

They’re the worst form of interview, except for all the other ones.

Why not just give an actual IQ test?

Not allowed by law.

> IQ tests (for interviewing) are typically spatial comparisons, some math, logic puzzles, reading comprehension, and pattern recognition

I definitely got these in tech interviews. Thoughtworks gave me one last year.

It might have been a de facto IQ test, but they should have disguised it carefully.

That opens you up to liability for race-based discrimination. The accepted wisdom is that IQ tests are inherently racist and favor white evaluees over black and brown ones.

Wouldn't coding interviews show the same bias?

The difference is, the Supreme Court has already made a ruling on IQ tests.

Isn't this ruling inherently racist?

That's formulated the wrong way. An IQ test per se cannot be racist because it is not a human being with intentional behaviour. This means blame must fall on the test designer or test conductor to introduce a racist biasing. In places where accurate results matter, the best available methods of psychometrics are used: http://enwp.org/Progressive_Matrices These are free of bias: culture, language, reading/writing ability does not matter. If these are used and assuming the test conductor did not screw up, and if white evaluees score better than black and brown ones, then it is an objective measurement of reality, not racism. Thus I think the "accepted wisdom" is wrong. If you know of any examples of racism through IQ tests, indicating intention, I'd be glad to hear of them.

I agree, but it's not obvious that courts would agree, and lawsuits are expensive even if you win. The legal liability worries are still there, alas.

What about other countries?

>The accepted wisdom is that IQ tests are inherently racist and favor white evaluees over black and brown ones.

Huh? Where did you hear that?

The reality is that Asians score the highest on IQ tests, whites score somewhat below, blacks score lowest.

The accepted wisdom is that Asians have the highest IQs, followed by whites, followed by blacks.

Well, that's not entirely true. Ashkenazi Jews have the highest IQs, but their IQ-advantage is in the verbal area, which is why when you see lists of the highest-paid lawyers a disproportionate number are Tribe members.

Keep in mind that these are averages. There are plenty of dumb Asians and there are plenty of smart blacks! You can't apply group statistics to individuals.

That’s what they think, there is a bias when all you do is the same interviews. Is like eating Your fav cake from the same bakery and thinking that is the best in the world

My first job out of university was answering tech support calls and the interview process was primarily based on the Wonderlic test[0] (famously given to NFL players).

[0] https://en.wikipedia.org/wiki/Wonderlic_test

by "classic" do you mean the "if you were stuck in a blender?" questions? you haven't interviewed in a while if you think they're asking those. it's all standard DS&A whiteboarding, etc. there's nothing even close to resembling a general aptitude test

FAANG coding interviews are definitely not IQ tests.

IQ tests (for interviewing) are typically spatial comparisons, some math, logic puzzles, reading comprehension, and pattern recognition and are done so under a time budget of around 30s to 1 minute per question

FAANG coding requires CS basics - hashmaps, arrays, sorting strategies, trees, recusion, basic ops for all of the data structures, and a few other concepts. There are typically only a handful of questions with 60 - 90 minutes per question

That's why the parent said that they're "disguised" IQ tests.

If they used a test that strongly correlated with the outcome of an IQ test, then it would also, legally, be an IQ test, and they'd have to stop doing it.

So instead, they use a test that weakly correlates with the outcome of an IQ test, and then try to extract the a measure of IQ from the "noise" that is all the other variables the test depends on.

IQ has nothing to do with whether you know CS basics or not, that's the problem. It doesn't matter that they said it's a disguised IQ test because it's not an IQ test in any fashion.

> IQ has nothing to do with whether you know CS basics or not, that's the problem.

In practice the Google/Facebook interview has nothing to do with knowing the CS basics either.

Edit: In practice understanding CS basics is the baseline that gets you to the interview - kind of like English is the baseline for an IQ test delivered in English.

> In practice the Google/Facebook interview has nothing to do with knowing the CS basics either.

What the heck is this statement? You do realize many algorithms that they test for are taught in an algorithms class for a Computer Science course?

I doubt you took an IQ test before or even properly accessed what it is. there are IQ tests composed of entirely of pattern matching shapes and they do not have any knowledge/language prerequisites. Get outta my internet.

> What the heck is this statement?

If you think knowledge of CS fundamentals is what is separating the hired from the rejected...

My experience from interviewing hundreds of candidates over the years at a top tech company is that the majority of candidates we reject have strong CS fundamentals.

> there are IQ tests composed of entirely of pattern matching shapes

And there are IQ tests/proxies that are entirely “verbal” like the GSS Wordsum - what is your point?

> I doubt you took an IQ test before or even properly accessed what it is.

I’m certified in my state to administer IQ tests....

> If you think knowledge of CS fundamentals is what is separating the hired from the rejected...

I've never stated that. I am aware of the other behavioral tests if that's what you're referring to. But the fact remains that they give you algorithms questions that we dont use in most of our daily work and if you can't meet the rather high basics you are rejected.

> And there are IQ tests/proxies that are entirely “verbal” like the GSS Wordsum - what is your point?

That you can have IQ tests without knowledge prerequisites unlike what you stated before the edit?

> I’m certified in my state to administer IQ tests....

Well either you're BS-ing or you've been confined to some kind of corporate environment where they've somehow convinced management people (who dont code) to adopt a test so detached from reality.

> I am aware of the other behavioral tests if that's what you're referring to.

No I’m not. I’m saying FAANG are hiring based on fast pace general problem solving skills that are difficult to game and have very little to do with computer science.

That is why “our” interview processes are being compared to IQ tests.

“We” have rejected people who teach computer science fundamentals at prestigious schools and hired teenagers who only started learning about algorithms a few months before the interview.

> That you can have IQ tests without knowledge prerequisites unlike what you stated before the edit?

How is that relevant at all?

> to adopt a test so detached from reality.

How is it detached from reality?

> No I’m not. I’m saying FAANG are hiring based on fast pace general problem solving skills that are difficult to game and have very little to do with computer science.

These are very vague words. What are these tests then? Can you give a sample? You're telling me that you get engineers to interview these candidates and they don't give questions that involve computer science? What are they supposed to ask then?

Or are you telling me you have tests that weakly correlate to an IQ test thanks to a number you computed? And therefore that makes it an IQ test?

> How is it detached from reality?

IQ tests are inherently just a number for one's potential, and they don't tell how someone will perform on the job - there are so many other factors involved.

you can pass every facebook iterview by practicing facebook tagged questions on leetcode. People do that every single day, here is an example from today

https://leetcode.com/discuss/interview-question/551434/faceb...

Its NOT related to IQ.

You can search leetcode forum for tons of examples of actual FB questions, all of them are tagged on leetcode.

You can get better at IQ tests by practicing them.

What a loaded statement. 1) You're already assuming these FAANG tests are an IQ test when people are discussing whether they are. 2) You're taking an assumed characteristic that tests that can be practiced and that makes them IQ tests.

Practicing for these tests makes you a rote learner and and rote learners by definition have too much crystallized knowledge, and they fail when met with new situations, which is not what intelligence tries to define. FAANG tests hence are not IQ tests if you can practice them.

Essentially these assumptions that FAANG tests are disguised IQ tests are just a reflection of the ego of those who are perpetuating them.

Anyone who feels like replying to this out of emotion they because they're in FAANG or whatever should take a psychology/psychometrics class first (yes, something outside of the software sector) before replying.

Your post makes absolutely no sense given that you can practice for IQ tests and improve your score on them. This isn't up for debate. You can improve things measured by IQ tests (i.e. spatial reasoning and pattern recognition) without rote. You can also improve them with rote. You can improve them with or without another person helping you as a mentor/tutor. It doesn't matter which one you do, the end result is the same if you invest enough time.

And you've lost the point. If these FAANG tests are meant to be IQ tests and you can rote learn them they are NOT a good measure of intelligence unlike some people claim them to be.

Edit: Also, good IQ testing involves a 2-3year break in between IQ tests. The thing is, you can leetcode continuously to game these tests.

You're arguing that real IQ tests don't exist, then. Any test is gameable.

They exist if you dont game them too much. And there isnt that much incentive to do so given the wide variety of tests.

so any test in the world is a disguised IQ test? lol

I'd rather have a hard worker with an average IQ than someone who's lazy with a high IQ.

Besides, IQ tests have biases that don't translate to good job performance. For example, if English isn't your first language you may do worse on an English-based IQ test.

Agreed. Ambition is an intangible that should also be assessed during the hiring process.

Have you by chance read anything by Kurt von Hammerstein-Equord? He’d strongly disagree with you:

> I distinguish four types. There are clever, hardworking, stupid, and lazy officers. Usually two characteristics are combined. Some are clever and hardworking; their place is the General Staff. The next ones are stupid and lazy; they make up 90 percent of every army and are suited to routine duties. Anyone who is both clever and lazy is qualified for the highest leadership duties, because he possesses the mental clarity and strength of nerve necessary for difficult decisions. One must beware of anyone who is both stupid and hardworking; he must not be entrusted with any responsibility because he will always only cause damage.

This is a quote from one man's experience and you're going to treat it as though it's the answer to his concerns? really?

Some of the points here are good, but I think some of them are too simplistic.

This article makes a huge assumption: that performance is the only factor you're hiring for, and therefore unstructured interviews are worthless. That would make sense if you're interviewing for a factory of emotionless robots. But humans are social beings, and human performance as a team is more complicated than a sum of individual productivity. You need to take into account the affect that the hire would have on the entire team. Just about everyone has an anecdote about a coworker sometime during their career that was super productive, but completely demoralizing to the team. This is a very valid reason to give unstructured interviews, and if you ask smart people who give them, this is what they're looking for.

I totally agree with the utility of work-sample tests. There is nothing better to see someone's performance than simply giving them a small but relevant chunk of work to do. It is very important to keep them short and flexible though.

If you make them too lengthy, you'll skew the results against people who have better things to do with their time. What kind of person works for free anyway? Certainly not the caliber of people I want to work with.

And if you make the test too specific to certain technologies, you skew the test against smart engineers who are less familiar with that tech. If you want a problem solver, give them a problem in plain words, and let their solution be open-ended.

I don't agree with making tests particularly hard, though. Make it nuanced, yes, but of normal difficulty for a given assignment. You need the appropriate people to do the job. If you only hire top-tier algorithmic experts for your basic CRUD app, then you're going to have a very expensive CRUD app with a bored and demoralized team. If someone does sloppy work, you can spot this on a normal exercise as much as you can on an unreasonably difficult one.

>> What kind of person works for free anyway?

I would expect everyone at some point has done something (worked) for free, either on speculation, as a favor, or for bragging rights. Open source wouldn't be a thing otherwise.

Most work being done on significant open source projects is by people who are being paid for that work.

Great points—I completely agree that the human element of hiring is of critical importance when it comes to building effective and high functioning teams. This article is intended to focus on the more technical/job focused component of hiring. That said, I agree with your comments here across the board.

With respect to your point about unstructured interviews - keep in mind that any statistical measure of performance isn't based on an objective measure of output - it's almost certainly based on performance ratings, which take into account "impact on the entire team." If anything, my experience is that charming, likable people that do well in unstructured interviews are also better at getting better performance ratings than they deserve.

Also, I think you're implicitly assuming that structured interviews cannot measure soft skills as they relate to workplace behavior. I would argue the opposite - the only way to ensure high standards for these types of skills is to have a structured interview. Otherwise, you're not systematically trying to hire people whose workplace behavior enhances group productivity, you're just hoping that whatever random biases that interviewers bring to the table end up being a net positive. Being a productive team player isn't about being able to impress a random person in an unstructured conversation - it's about knowing how to behave productively (combined with the willingness to do so, which unfortunately is contextual and virtually impossible to directly observe) in a wide variety of challenging situations.

I don't think you can. Job performance is something that gets figured out on the fly.

Unless you're working in a crazy standardized operation thats a moving target.

e.g. We've got a ton of manager & they all sorta do their own thing...they gravitate towards their strong suit essentially. You can't really test for that in advance.

Some people excel at leading 30 man teams on stable jobs. Others excel at being parachuted into technical shitshows and sorting it out. Very different personal attributes.

It's a crap shoot as tracker1 says and will continue to be one

If you know what kind of company/situation you're in, you can certainly select for candidates that are more likely to thrive in that environment.

As you mentioned, people gravitate toward their strong suits. So by asking behavioral questions, you can learn how they approached situations in the past and whether their strengths/weaknesses are the right mix for your needs.

If your company knows what situation it is in, and if that situation persists for more than a year, your company is already better than many. The company I just left had an endless parade of new missions, new "values", new career ladder descriptions, and the median tenure of managers in engineering was about 6 months. The idea that you can either rate the performance of your staff or predictively hire engineers who will succeed under these conditions is a joke.

Yep, if your company is having leadership/strategy problems you're not going to go far.

But as a hiring manager in that situation, you can select for team members who excel at dealing with ambiguity, internal locus of control, vision, etc. Additionally, you would look for people who value learning vs. psychological safety, etc.

You're in a known situation, and that situation is constant change and churn

That's true.... that might not be a company you want to work for. If there's not clear strategic direction and things constantly shift, it's very hard to hire anybody effectively into the org.

>If you know what kind of company/situation you're in

That's my point. You don't unless you're in a very simplistic business.

I really don't think you can define/bottle this easily - even with high quality interviewing.

Nor do I think you should. To use a very cliched phrase the hottest hires are the ones that think outside the box AND are killing it. Without some flexibility in terms of filtering and criteria you might miss out on that

This whole article is predicated on the idea that you can quantitatively measure the performance of a software engineer. That seems completely unfounded.

I don't think that true—what the article is saying is that different hiring activities are more or less useful in predicting the future success of a hire. And the more measures you employee intelligently, the more predictive the process becomes.

Why isn't it? Any job where enough information can be known to know that someone should be fired or promoted is a job that can be quantitatively measured to some degree.

So you need to design you assessment process to reflect the real world scenarios the hire will encounter. It's surprising to me to hear so many developers push back on this notion that you can't design a predictive hiring process. If you were hiring a dentist, wouldn't a great way to assess them be having them fill a cavity? Or asking a chef to make you your restaurant's signature recipe? It seems very logical and the key here is that the work sample is reflective of what they'll encounter on the job.

The problem with the examples you mentioned that their skills are fully transferable. 30-50% of my knowledge is non-transferable because of a)Company domain knowledge b)In a technology the next company won't use.

It would be a better comparison to a bridge engineer to build a bridge in a different country. The laws of physics are the same but regulations are not. Building any non-trivial software is more comparable to this.

If you hiring for Wordpress (or any other) sweatshops that's a different thing. But then you hiring factory workers.

If you hire all the exact same kind of people, what happens when you were wrong about the sort you need? A little diversity is good. We don’t need everyone to be an expert on everything, we just need a few experts for everything.

The cited data is from a paper The Validity and Utility of Selection Methods in Personnel Psychology https://www.researchgate.net/publication/232564809_The_Valid... in 1998 as a meta-analysis of other findings, but they are not specific to software development (one account is sourced from 32000 employees in 515 diverse civilian jobs in the 1980s). Maybe software is like any other job (sales, clerical, accounting) but maybe not.

Why would it not be?

I have some guesses, but even if you take that it is, the authors were not very transparent that they made this assumption, in fact they did not point out that the research was done on all jobs while they spoke only of software. That is a bit misleading.

Oh wow I spent all that time reading with the assumption that this was a novel and relevant investigation over that year of being focused on the specific problem of software inteviews. Interview articles are the hot-button clickbait of Hacker News. Not necessarily the author’s fault, but this topic has a tendency to trend above the contribution of its content.

Data Science interviews almost always have a take home interview as the technical challenge. The in person interviews tend to be cultural fit interviews.

This is because DS work is larger than a bug fix, implementing a new feature, and even larger than writing a program from scratch. Even a small DS problem is a multi day problem. Most interviews allow one to take weeks to do a "quick" 2-3 day problem.

This process may be out of necessity, but I feel it reduces noise. You really do have a better idea of knowing what you're getting.

Software Engineers may groan at the idea of a take home project, but just throwing this out here: What if a company let the interviewer choose between a take home problem or an in person 30 min style white board interview? What kind of results would come from that? Would the people who take the take home interview work out better at the company, or would the whiteboard interviewers end up working out better?

There are a few ways to do a 2-3 day data science problem: 1. Random plug and chug / datarobot 2. Copy someone else 3. Expert feature engineering 4. Overly-documented EDA

I don’t think any of this is relevant, but maybe it gives some indication that they can do anything at all. However, most good DS will pass because they‘ve been burned too many times by companies just looking for free help on a problem.

If you’re a DS, don’t ever work on a company-relevant problem for free.

2 comments on this:

1. The authors of this paper are selling something. This is an advertisement. Not saying it's wrong or untruthful because of that, but it is an advertisement.

2. As another commenter said, the cited data is from a paper that doesn't look at software engineers specifically. Given that software engineering has one of the highest amounts of variability in productivity of any profession, that should be taken into close consideration.

The purpose of this paper is to present research that's completely independent of any product, and show how the concepts apply to software engineering specifically. What's your source in saying "software engineering has one of the highest amounts of variability in productivity of any profession?" That seems hugely unfounded.

> The purpose of this paper is to present research that's completely independent of any product, and show how the concepts apply to software engineering specifically.

Lol, the article actually ends with a link to the product you're selling:

> Ready to build a more predictive hiring process for software engineers? Schedule a 1-on-1 review of your hiring process or request a free trial of Qualified here.

The best interview process is not going to account for bad corporate policy, terrible bosses, or other confounding factors.

No argument there!

So true!

It's sad it's so hard to tell if someone is going to be a terrible boss or if there are systemic issues.

Some of the best interviews I've had, have had the worst bosses, and vice versa. Though, totally anecdotal, especially given I can only go off of the jobs I've accepted which is a sample set of the best interviews I've had.

Just to share my experience.

1. We sucks at hiring, so we need to train juniors on how to hire early. 2. Programs live in the head of their creators. The code is just a draft left for computer to execute. Build a human culture where the aspiration is discussed. 3. We don’t replace people, we hand over the responsibility. Be ready to work differently. 4. I conduct interview by asking the candidates to explain me I piece of work she is is proud of. If I understand the what, how and why, she usually pass. 5. Remember: most issues are not technical but organizational. We can train people, but we can’t change them. 6. Set high quality standards for your team. Make them proud of their work. On boarding will be easier.

So, it is less about how to hire the shinning star and more about how to welcome and how to create a culture around your values.

My two cents.

The best interview I've ever had was I was asked how I might design a real user feature for their company and review a sloppy PR with errors. For the first I came up with two options and weighed the best reason for each and which I might go with depending on the circumstances of the project. For the second I wrote PR comments I would leave any developer with suggestions for improvements, concerns, and praise for good work. That was representative of my actual job as a software engineer.

The normal is taking some sort of tricky pop quiz styled questionnaire, which in my opinion was not representative of anything about me beyond I know how to use a for loop efficiently. Yet this is what I find these companies offer. They tout them as a quick fix cause its easy to standardize, analyze, and hard to do therefore good, and last but not least waste the candidates time but not the companies. Their the metrics are moot. Not only have they been proven to be ineffective by the big companies that spawned them, but I also find them demoralizing and a turn off.

It seems like the industry is heading towards a self imposed SAT style quiz where the lucky winner of this startup war holds all the power and provides little benefit to either side of this process.

I think the important thing is to realize that you need to rely on realistic factors and many different signals to get useful data.

I've been to plenty of interviews for "senior software engineers," where they used the standard gauntlet of algorithms/problem solving/data structures questions in rapid succession for eight hours.

I tend to deliver most of my projects on time and of a high quality. I value things such as correctness and maintainability, and am conscious of other factors to consider such as viability and availability of time and capital. I spend a good amount of my time writing and seeking feedback from colleagues and think it's important to share our ideas and talk about our work.

And when I practice my work I don't think I've ever been in a time-constrained scenario where I had to come up with an implementation of an algorithm while being judged and evaluated for how I think. I know enough about my work to understand when I need to use BFS or DFS and when to use a tree, a map, or a heap; but I also know that if I can't think of it off the top of my head I can walk over to my library and pull out a book or look it up online.

I'm also a proponent of formal methods. I've written specifications to help solve race conditions and validate implementations of locking algorithms. I can write basic proofs.

And I'm a great writer as a result. I spend a good amount of my time writing and communicating my ideas, seeking feedback from colleagues, and in general improving my understanding of problems.

How does finding a cycle in a sorted array or reversing a linked list signal to you whether I would be a good fit for your company as a senior engineer? Does your company also value clear writing and good communication skills? Do you value engineers that can manage themselves and get work done autonomously?

Modern interview practices at most places I've been through don't know how to get those signals. It's useful knowing that the people I work with are also familiar with when to use a heap versus an array but I also value other things at the senior level that most companies aren't really testing for.

I always prefer data over gut instinct but I feel like the process of hiring people is not amenable to clean signals... it's a very messy process and I'm not hopeful that there will ever be a silver bullet.

What does your interview process look like?

It's not perfect either but we're pretty open about it: https://weeverapps.github.io/interviews/

Internally we use a rubric for the various job roles we hire for and we aim to remove or counter biases at every step of the process. Our teams value diversity and are pro-active about equality and inclusion.

A big part of the process is code review, something our engineers actually do spend a lot of time doing, so the process is mostly focused around that.

These are the foundational aspects of our hiring process that we openly advocate for. https://www.qualified.io/blog/posts/the-qualified-manifesto-...

What we advocate for is making the technical assessment component relatively quick and painless, but using the work sample as a basis for assessing the other sorts of skills and intangibles that you mentioned. You can use your in-person time with candidates not to discuss their technical work, but to add requirements, talk through how they might build an additional feature, and generally assesses how well they communicate and solve problems beyond writing code.

To boil this interview thing, you are trying to make a very far prediction of someone based on very little data. The industry standard way to gather that data will probably get you to see if that person has done it before but not how well. Unless that person is sorta famous and even then he could be an asshol to the team and a disease

Furthermore, the unstated unknown that many interviewers have on their mind is whether the candidate will be successful in this specific company.

“In economic terms, the gains from increasing the validity of hiring methods can amount over time to literally millions of dollars,” writes Schmidt regarding the importance of valid assessment methods. “However, this can be viewed from the opposite point of view: By using selection methods with low validity, an organization can lose millions of dollars in reduced production. In a competitive world, these organizations are unnecessarily creating a competitive disadvantage for themselves. By adopting more valid hiring procedures, they could turn this competitive disadvantage into a competitive advantage.”

The problem is that this misunderstands the purpose of the hiring process. The purpose of the hiring process is to get people who are qualified, while signalling that your company hires good people, and staying out of legal trouble along the way.

I'm not sure I agree that an organization can gain "millions of dollars" by fine-tuning its hiring processes to the nth degree. While I absolutely agree that there are skill differences between programmers, I contend that the economic differences attributable to skill between programmers are dwarfed by the economic differences between any automation and no automation. Simply put, having a terrible programmer automate your business processes today is far better than sifting candidates to find the perfect programmer to automate your business process two years from now. Hiring is about satisficing, not optimizing.

I understand your point—it's not about optimizing your process above all else (at the expense of speed, for example). But you can and should optimize your hiring process just like you would your sales process, onboarding process, etc and the outcome you're looking for is hires that are successful once hired. It costs companies far more if they sacrifice in the name of speed and make poor hires.

"Hard Interviews, Happy Workers". This note near the end assumes correlation shows causation, which it does not. You could also argue that more workers want to work in certain jobs, creating more competition and leading the company to make interviews harder. The workers in those jobs may be happier, but not because the interviews were harder.

The elephant in the room is that we can’t accurately measure job performance by any objective metric either.

Many things would be simpler if we could: performance reviews, promotions, incentives, salary equality, recruitment etc.

Interviews aren't just about talent and ability. Some fairly abstract concepts are important too.

How the individual fits within the team is vital. You might have one of the smartest people on the planet but if they can't effectively communicate or understand and work within the cultural norms of the team you're going to have a problem.

These norms and practices vary and shift between teams, countries, and even within the same team over time.

For example hiring a Linus Torvalds for an intermediate, client facing, political position, is probably going to have poor results.

These are difficult things to test for and are as much a decider of job performance as raw knowledge and technical experience.

so you're gonna pay us to do these "work samples" at market rate, right?

Sarcasm aside, this is exactly why we make ours as short as possible in duration. You don't want to filter out people with valuable time.

Actually, yes! Erik Bernhardsson, the CTO at Better.com who is cited in the article a few times, does exactly that for finalist candidates that they ask to provide a significant work sample. I think that's 100% appropriate. But that said, a work sample doesn't necessarily have to be something that takes hours on end either.

ayy, that's awesome if they pay candidates for significant work samples. i think there's some discussion to be had about what's significant. anything more than 1.5-2hours in my opinion. an hour or two is how long an interview might take in any profession. asking me to work unpaid for 4 or 8 hours (one project: 3 days!) is excessive. maybe call it the "half-day" test. is it a half-day of work? if so pay. if not, then you can do that once during an entire interview process.

Aptible compensates for the amount of time we want you to spend on a work sample.

tl;dr: offer your applicants test exercises

Yes, but not generic tests—in order to design a hiring process that's actually predictive of future job performance, the "tests" have to be designed to be directly relevant to the real world work that the engineer will be tasked with.

I wondered, have any others been expected, to reverse the interview direction? As in ask the company manangment, sales and tech ceos, to answer your test scenarios?

"You have developed a new product with broad market appeal, and sales already found you a huge first customer, which would secure you for years to come. In return they expect you to tailor the product to there needs. How do you react?"

It sounds mad, but it seems somehow to be expected by some. Like they expect you to test your company for warning flags as thoroughly as they test you..

Reply on news.ycombinator.com