Technical phone screen superforecasters
blog.interviewing.io
blog.interviewing.io
The next part is the entire tech screening process. I remember back in the day when we used to just ask people if they could do FizzBuzz now we're looking at if they used an interface to wrap their ORM usage. That's something that can get picked up at code review and taught. Not everyone is going to code by default at the style of a company but they can learn to do the things the company wants.
Then there is the obivous, lets test people for things they aren't going to do. Google and co made this the in thing and the fact we now have books upon books just designed so people who are good at tech can pass a tech interview is a sign in itself that there is something rotten. Hell, there was one guy that took almost a year off to study for the Google exam, err I mean interview. In a world where the shortage of tech employee means they're on the eligble for a visa list in pretty much every country in the world.
I've literally had someone be failed on a test before they did far more than expected therefore didn't have time to finish the test. No, that's the person you 100% want. They exceeded expecations.
I've made pretty much every mistake in interviewing others, I've had these mistakes while I was interviewing. I've come to realise what is important is, can they do the basic level programming required? If so get them in and if they really suck let them go during the trial period. Our worst developer at the company I work for interviewed amazingly, he falls asleep during meetings.
A bunch of companies keep trying to improve the hiring in tech, the problem is, they are the problem.
The problem was "graph algorithms" used to be like "C pointer indirections"--a task that indicated that you were one notch above the average.
Of course, once Google made this a metric, it lost all meaning.
If you put a condition between someone and $500K, they WILL figure out how to convince you that they passed the condition.
no way. i just cannot believe that's true. i'm based in the U.S and i'm currently interviewing ... and spending my evenings studying to be better prepared for upcoming interviews. they all do the same thing: crazy programming puzzles.
i believe what's happening is that companies have become extremely choosy because there is a never-ending stream of candidates. they subject to insane technical interviews and reject good candidate after good candidate with the hope that maybe the next will be just a tiny bit better. And if not, well, there's more after that.
i'm convinced that in 2020 the software engineering profession is absolutely, positively supersaturated.
i'd say more, but i need to stop procrastinating and get back to the video about dynamic programming that i was taking notes on.
I was a new dev around 2008 and while there were software jobs around, fewer companies were willing to hire newbies. That could be playing out again.
It's a matter of perspective as well though, so both sides are correct. "The job" here can talk about very different engineering roles and levels, which will have a different hiring pool by market and by saturation.
From my own experience running interviews, even in the same "community's" hiring pool in the same location you can expect large differences depending on what position and skill set you're hiring for at that moment.
I find it more worrying that over years I've seen that some engineers are attempting to optimise the interviewing process on their end. Obviously instead interviews should optimise to appeal to candidates and make the process equitable, quick, and accurate, without nonsense like coding challenges that don't relate to the job.
You seem to be saying there is an area of softwre engineering that is not saturated candidate-wise.
Well, don't hold us in suspense -- go ahead and tell us specifically what sort of engineering role you were referring to here!
As we go up the engineering career ladder and land on more specialised roles with the necessary experience, both from a technical and managerial perspective, the funnel definitely shrinks. I won't deny that that looks different from place to place, but it's hard to deny that it will take years still for any saturation we see to take place on the higher part of the career ladder as people specialise more as they get there.
Specifically once we started looking for engineering management and principal level roles that still bring the technical expertise we'd expect the hiring pool was definitely not saturated.
This is not at all true for the entry level market. This might be case for mid-level or even seniors but the entry level market is so saturated that these tests are bound to be necessary to cull the pool in a sense.
Side note - I do NOT use whiteboard technique in my interviews. I ask the candidates questions about the stuff they mention in their resumes and go deep in the technical discussion based on their responses (or I told them to walk me through one of their recent projects be it a work project or their side project). Using that 'simple' technique, you can easily filter out so many pretenders (as I'd like to call them, who have splendid resumes but little to back things up when grilled the details about their resumes).
But asking about work projects specifically can be tricky, since you have to avoid any proprietary bits, to avoid any possible appearance of IP taint.
If they don't have open source projects to dig into, hypothetical plausible system problems (not leetcode exercises, nor puzzles) are an option.
You still have to watch out for people who know a lot of the right engineering things to say, and/or have done prep memorizing and drilling, rather than speaking from experience. (Example from one of the links in this thread, which had an example of a repetition flash card app, saying that UDP is connectionless. I don't know what good purpose would be served by having that info on a flash card for interview prep, other than to help someone who knows absolutely nothing about it to bluff their way through an interview that asks about it.)
From what I can tell, there's a sea of people who can barely do anything with some hand-holding, then each level of knowledge, competence and experience you want to go higher, you have to slice the pool by 100. By the time you get to level where people can competently drive non-trivial projects, there's a severe shortage.
Jobs that have actually had talent shortages do not operate this way. They hire candidates based on degree alone. Interviews are a formality. If they are really desperate they offer to pay the training of unqualified but interested people. The idea that they would reject multiple candidates with degrees and years of experience based on interview performance is absurd. No one is rejecting math teachers with years of experience because they failed to score 100% on a pop quiz about math, covering everything from set theory, to calculus and geometry. Sitting on the hiring side of the table at a big company, I saw us reject so many qualified candidates. The most absurd rejection was when someone with 8 years of experience at a FAANG was rejected for a Senior Frontend Role because they couldn't answer one of the interviewers pet questions about differences between MYSQL and Postgres. A situation like that would not happen if there was actually a talent shortage.
Also, the idea that there are all of these unfilled jobs strikes me as fairly suspect as well. I know someone who runs a startup and noticed that they had the same job posting on their site for a year. I asked them if they were having trouble finding someone and if I could help them by reaching out to people who might be interested. He admitted that they had no intention of hiring anyone, it was simply advertising. It makes them looks successful, like they were "growing". Having been at bigger companies where we rejected qualified candidate after qualified candidate, I suspect this practice is widespread. Better to project growth than to admit you are actually in a hiring freeze.
The last point I will make is that if there was a talent shortage, the bar would stay the same or decrease. From multiple discussions with people at FAANG, they have said the bar has increased markedly in the past 3 years. Again this is not something that would happen if there was actually a talent shortage, but would happen if there was a talent surplus.
There isn't a talent shortage because the candidates you interview don't talk about their projects up to your arbitrary bar.
>Then there is the obivous, lets test people for things they aren't going to do. Google and co made this the in thing and the fact we now have books upon books just designed so people who are good at tech can pass a tech interview is a sign in itself that there is something rotten.
So which is it? Should interviews be related to the work at the job or not? The Google type interviews evolved precisely from the philosophy in your first paragraph I quoted: that a lot of domain knowledge can be learned easily and that it was most important to test for more fundamental knowledge and abilities as well as general problem solving skills.
Ultimately I think it's a lot easier to criticize the interview process than to come up with a good alternative, as well as establishing some basic criteria about what should be selected for and against. Not that I think Google style interviews are perfect or anything, mind.
If a company wants to test someones basic logical knowledge and fundmental programming knowledge, seems fair enough.
There is a massive difference between writing FuzzBuzz and knowing the differences and uses from all the different sort algorithms and being to write optmisied computer science code. One tests programming and logical skills to a childs level (FuzzBuzz is a kids game), the other tests computer science skills to a high education level.
If someone wants me to write a basic sorting function, then I would have no issues. But if someone expects me write an optimised quicksort function, nah.
The list of the things these Google style questions ask is big enough thath the books for these interviews is quite big. (Lots of fun to read tho, highly recommend them for people wanting to nerd out)
>Ultimately I think it's a lot easier to criticize the interview process than to come up with a good alternative, as well as establishing some basic criteria about what should be selected for and against. Not that I think Google style interviews are perfect or anything, mind.
Well, I think I did come up with a good alternative, basic interview with work trials. You want to know how good they work, not how good they interview and the two actually are different skills.
Let me be a devils attorney here. Understanding efficient sorting and how quicksort works is fundamental CS knowledge. I would say the idea is if you know how quicksort works, you should be able to write the code, no? If one is okay with writing a basic sort function but not quicksort - it could mean (1) that person does not know quicksort, (2) knows quicksort but not a good enough engineer to write code for it or (3) idealistic.
That said, I personally don't believe in testing whether a candidate can write the code for quicksort. I have however found that 'general problem solving skills' typically correlate with good engineers at places where one isn't just building yet another MEAN stack app.
> Well, I think I did come up with a good alternative, basic interview with work trials.
Not practical enough. Do you expect, in the current scenario where there is a demand surplus, for candidates to happily sign up for a trial stint - that is for the "interview" to last one or three months? I personally won't be okay with it because it messes up with a persons stability, their ability to interview at multiple places.
Even if you expect this to be AFTER the onsite interviews, like a probation period, both the company and candidate will want to maximize the # of candidates staying on, which means it doesn't really solve the problem you intended to solve.
Companies are hiring software engineer not computer scientists. If someone wrote a sort algorithm in a normal job I would expect that not to be used and a library to be used. This is also knowledge that people who studied and aced their tests in Uni forget, because no one in their right mind writes a quicksort.
> Not practical enough. Do you expect, in the current scenario where there is a demand surplus, for candidates to happily sign up for a trial stint - that is for the "interview" to last one or three months? I personally won't be okay with it because it messes up with a persons stability, their ability to interview at multiple places.
People actually sign contracts like that on a daily basis. That is literally what you sign up for. Here is the actual process: CV screening -> Tech Test -> Phone Screening -> Interview -> Trial or "probation".
And the rest of it about people looking for jobs, well that's why companies should be selling/recruiting. The whole "but people won't do that" is clearly wrong since they already do and often jump at their first offer.
Typically these questions are asked to entry-level software engineers and not ones with some experience.
Again, I don't believe asking people to write quicksort provides any signal but I was arguing that there isn't any difference between asking a candidate to write a "basic sort" vs "quick sort" in an interview.
I also mentioned this is not a good question to ask if all you are hiring for is some MEAN stack app - but is an appropriate question if you do any development where some optimization is required.
> People actually sign contracts like that on a daily basis. That is literally what you sign up for. Here is the actual process: CV screening -> Tech Test -> Phone Screening -> Interview -> Trial or "probation".
Yes, typically for roles where it is difficult to vet a candidate's skills appropriately during the interview. It is also not an improvement for Software Engineering, imo. If you ask Software Engineers if they might prefer this process over the current process, I think, most would not.
I think you would be surprised, I am sure there are lots of people who wouldn't be interested in such a methodogly. But you need to remember what the purposal is. Do you want to be interviewed and tested on your skills in an interview and then sign a job contract or would you rather do a possible 2 day tech test. You also need to remember the job market, if I have 5 companies in my job pipeline (I stop adding companies at 5) why would I want to do 5 1 day tech tests?
And 1-2 tech test is extremely common.
Although, I do agree that sometimes engineers ask questions that they don't expect to work on at the company. When I interview I ask questions (leetcode style because of the nature of systems involved) based on what problems we typically solve. I never go into advanced graph etc. But good understanding of basic data structures, understanding time complexity, testing is utmost important.
Quick, without looking it up, in 45 minutes, write an implementation of CC-Radix, cache-conscience radix sort. It is a sort, it deals with performance, therefore it’s fundamental, right? After all, in my last Google and Facebook interviews, the questions were all about performance.
While we’re talking about performance, let’s talk about cache. It’s a fundamental part of computing after all. I tell you that you can pick any architecture you like, except one that is four way set-associative. What percentage of candidates can answer that? 50? 25? What percentage who did not go to certain schools can answer that? Of those who could answer it, how many saw this issue ever come up?
I love hardware, but these days most things are out of my control. Either I’m running on my Mac or on some VM instance I didn’t spin up.
Our field is VAST. It is a natural tendency to be biased towards what you know. Many things can be argued to be fundamental, even writing AND, OR, NOR, and NOT gates from NAND. That’s about as fundamental as it gets. It provides zero insight into the candidate for most groups.
Personally, I thought this was shift in focus was pretty reasonable. I didn't prepare much at all for system design and relied on my real-world experience, which feels like how an interview for a senior candidate should go.
Its a given that if you are trying to get into faang you are going to need to spend a considerable amount of time trying to game the interview process by doing as many leetcode problems and mock interviews as you can before the interview. And then still there is no guarantee, even if you ace all rounds and pass all the coding challenges.
It probably doesn't matter to them as long as they get their positions filled, but its prolonging their hiring.
We would thus expect a candidate being interviewed by a charismatic interviewer to score more highly than they would in the presence of a social dolt. This state of affairs brings into question the utility of phone interviews as measures of job performance.
* a personal interview (i.e. the one for sociability - mostly with HR people)
* a more technical interview (generally with tech people who don't answer questions about non-technical stuff)
* an interview with stakeholders (i.e. a potential future boss, or a department head who can answer questions about long-term career development, perks, etc.. This also tests for sociability)
(For full disclosure, I should mention that I received an "You passed 4 out of 5 of these interviews, and got a 'maybe' on the fifth" and thus received no offer).
When you meet these people, it can really be its own kind of drug.
That’s all you need to know. They quantified interviewer quality in terms of offer rate. This article is about self-celebrating a hiring binge without regard to the impact on the team or company.
The title uses the word 'superforecasters', which makes me think the author has read some of Philip Tetlock's work, and is trying to apply it to their situation, which I find admirable.
But I agree what's presented here is noise.
Nor did they test their system by doing blind interviews where already hired and well performing employees would go through the pipeline.
I don't know if they actually wanted to take a good hard look at their own system.
That’s not infallible, but they’re committed to it being an ok metric—-it’s already what determines who they’re hiring. If you think that their decisions in on-sites are sufficiently bad, you probably should give up on the idea of optimizing how phone screens are conducted--it would be wasted effort, and you'd have no idea how to make sure it worked.
You can easily imagine that these results are just noise, but it’s more difficult (though hardly impossible) to spin a story in which it’s making their process worse. I think it would require that these interviewers have a knack for selecting weaker candidates who can still pass the on-site.
I've been going through some technical phone screens lately, and am pretty surprised by how bad the questions are that some people choose. Questions that I wouldn't think would provide any signal whatsoever for hiring
When I was interviewing candidates at a company that didn't have a structured interviewing process, I used to ask people to write "the world's shittiest JSON encoder." What that is is a very limited JSON encoder that could handle strings, numbers, None, dicts, and arrays, without regard to performance, on their laptop or a loaner, with full access to the internet, and a pinky promise that the candidate wouldn't just look up "how to write JSON encoder," or anything similar.
I would give the problem in stages, starting with primitive types and working up to arrays and dicts. If we finished that, we could talk about other stuff, like how to write the world's shittiest JSON encoder, adding additional types, testing, and maybe a little optimization.
I thought it was a good problem, and we got a good candidate or 2 out of it through interview loops I participated in.
There were lots of twists and turns to the problem, but it wasn't some commonly asked question like LRU cache, and it didn't demand a lot of code.
I assume you wouldn't have classified me as a "hazer," right?
The idea that anyone should be able to solve a problem like this in 45 minutes as a demonstration of their skill as a software engineer is just ludicrous. If you can solve this problem it means you have studied this or similar problems. All it means is that you have the luxury of study. This has almost nothing to do with being a software engineer.