The problem isn't standardized tests, it's that leetcode questions are about having the time to have learnt the answer beforehand, rather than raw ability for problem solving.
LC tests typically copy problems from the university the interviewer graduated from. College programs differ, so this is really a case of what you were introduced to.
There's a fairly popular online LC test company in my corner of the world which was formed by graduates and lecturers from a certain university and they started out by just giving the problems from the curriculum. Result was heavy bias in favour of students and alumni of that university.
If that were the case then a normal, well accomplished software engineer shouldn't need to "grind" leetcode to pass an interview.
its just a cargo cult. Getting someone to do a code review is a much much better test of skill:
Do they ask questions?
are they kind in their assertions?
At what point do they go "I don't know"?
Do they concentrate on style or substance?
do they raise an issue with a lack of comments?
do they ask why the description of the PR is so vague?
When they get a push back, are they aggressive?
All of those are much better tests than "rewrite a thing that a library would do for you already"
No, what this shows is that the skill range for accomplished professional software devs is absolutely massive. What these companies want is to find the tail end of this very wide distribution. Leetcode interviews do a decent job at this. If you have been coding for a decade and can't do leetcode mediums with almost no prep, and hards with a moderate refresher on data structures, then you're simply not the in the right tail of the skill distribution and they don't want you. This is what so many in our industry can't accept: you're just not talented enough to earn a FAANG job.
Background:
I'm in a FAANG type company now, a YC company, 3000+ engineers. I'm a Staff SWE with 20+ years of experience (ECE degree) and make $600k+ per year. I've went through the promo cycle here (it sucks).
I can't do any leet hards and can do leet mediums after studying. Some easy's take me a couple of tries. I usually do very poorly in interview coding exercises.
** throwaway , main account is 10+ years old.
If you are lucky and join/lead/get a new project off the ground, then you'll also grow.
If you're just trying to move the dial on a normal project, then its very difficult to make any kind of headway. You are surrounded by overly enthusiastic hall monitor types who will put in more hours than you, or post more than you.
> [describes self, story completely contradicts parent post]
I'd say you're a perfect example of why "Leetcode problems are a good filter for getting the top 0.X% of talent" is false.
I mean it doesn't, because I'm at a FAANG, At a FAANG you are infantalised from the very start, sure you passed a very difficult interview where you have to balance a binary tree efficiently as possible. But you're going to use none of those skills here.
what you actually end up doing is copy/pasting some random code you found using internal code search, because the sensible way of doing it can't happen as that would involve porting a thirdparty library, and doing all the procedural work that follows.
so you hack some shit together, ship out out and hope that it doesn't break. You then decommission the nasty hack you shipped last year and claim credit for innovating. Is your product not hitting the right metrics? loosing users? doesn't matter, so long as the super boss is happy that you've hammered in the REST API for the stupid AI interface, you're not going to get fired.
In a startup/small company, if you fuck up, the whole place is going under. Need metrics? you'll need to find a small, cheap and effective system.
Here, we just record then entire world and then throw hundreds of thousands of machines at it to make a vaguly SQL interface. Don't worry about normalising your data, or packing it efficiently just make a 72 column table, and fill it full of random JSON shit. Need to alter a metrics? just add a new column.
In short, don't praise or assume that FAANGs are any good at anything other than making money. They are effectively a high budget marvel movie, Sure they have a big set, but most of it is held together with tape and labour. Look round the side and you'll see its all wood, glue and gaffer tape.
FAANGs want the top .1% of developers, they don't necessarily need them for most roles. But the point is to hire developers that you could put into any role in the company within reason and have them be successful. 99% of development work at a FAANG is pretty unexceptional and doesn't require exceptional developers. They hire for that exceptional 1%.
FAANGS want a load of loyal, naive people who are willing to work loads of overtime and not ask too many questions. Who better than posh kids from great universities who haven't quite figured out that life isn't a meritocracy yet!?
Sure they also want the top 0.1%, but they have a different interview track. Do you think all those OpenAI engineers that were going to follow Altman were asked to do leetcode?
I don't know about OpenAI specifically, but I have heard of interviews at other top ML research positions were partly based on leetcode problems.
Sure I can. By the time you get to Leetcode hard, these aren't just "can you derive the answer". The questions by design take 45+ minutes and have some weird quirk in it that is nominally related to the core concept being tested. These aren't necessarily meant to be done on the fly during an interview period.
>Not being able to see up the mountain doesn’t imply there are no climbers above you.
a better analogy is that youre on a road and you see a freeway above you. The people above you aren't "better", they are simply on another road, to another destination. But they aren't necessarily worse either. They could be on their way to a dead end job or could be a billionaire CEO.
That is to say, it's useless comparing yourself to other people you don't know. Everyone has their story.
So I'm not sure we're interpreting the same conclusion here unless you think trick questions are a mark of a "good engineer".
>You cannot argue against general rule by pointing out specialized exceptions.
Lertcode hards in interviews have been getting more common for a decade now. At what point do we stop pretending that more cats aren't losing their 4th leg and announce an epidemic?
It shouldn't be common, but it's becoming more common. I haven't seen any claims in any conversion even disagree with the notion, let alone provide hard evidence.
This sounds like you want to penalize students who studied for the exams. Or at least not reward them.
Like all interview formats, it’s a proxy for understanding if the prospect would be able to get the work done and be a good fit with the rest of the team. I’d say it’s a pretty good proxy for work ethic at crunch time as well.
If your complaint is that a normal person wouldn’t have the time to study these things in detail, why would a company want to hire someone who has external obligations?
s/exams/interview/g
External obligations like full time employment and a family?
Interviews are like exams but you don't have any clue what topics are on the test. If Leetcode was some licensed, standardized approach to getting some license to verify my ability to code myself out of a paper bag: I'd hate it, but I'd grit my teeth and study it. exams can be studied for.
But it isn't, so I can be studying leetcode questions and be hit by a dozen other topics. I don't have time to study everything, and the market right now isn't worth pinpointing specific companies unless you have a stellar reference.
>I’d say it’s a pretty good proxy for work ethic at crunch time as well.
I couldn't disagree more. Someone so prepared for interviews have the least skin in the game. Layoffs come and they didn't work > 40 hours but were otherwise excellent? oh well, get another job in a month because interviews are a breeze for them because they breathe DS&A.
>why would a company want to hire someone who has external obligations?
I'd love to know the answer here as well. Why do companies internally penalize workers who were laid off, but then try to "steal" currently happy employees but make them jump through these hoops? The logic seems backwards; interviews should be hard and depress wages for the laid off, desperate workers so you can get a desperate unicorn. But you want someone who lacks the time to study because of current job obligations to get to an offer faster. Their proof of work ethic is being employed to begin with.
Most companies are building products anyway.
We had one person who worked on [censored] 20 years ago, then was manager of [non-programmers, rest censored] - now wants to get back into coding - can this person still code? If so I want them, but if they have forgotten everything... I of course censored details for privacy reasons.
This is how I learned mathematics for example.
After you have learnt enough of the special cases (trees) you start to be able to connect them and see their relation (the forest).
I highly doubt you are sight-reading atomic bomb LC hards.
I've done tests like this for some companies. It felt a lot fairer and more closely resembling the actual work environment than live leet-code interviews, with biased interviewer(s) and a stress factor that's not a part of the actual job.
They will also be able to use ChatGPT on the job. And StackOverflow. And Google. If they know how to use tools available to solve a problem, that will benefit them on the job.
If you're testing them for what ChatGPT can already solve, then are the skills being tested worth anything, in this day and age?
Take-home LeetCode, even with cheating will still filter out a good chunk of candidates. Those who are not motivated enough or those who don't even know how to use the available help. You'll still be able to rank those who solved the task. You'll still see the produced code and be able to judge it.
Like other commenter points out, you can always follow up the take-home LeetCode. Usually, it becomes apparent really quickly if a candidate solved it on their own.
I wonder if either of the following could be cost-effective:
(a) Fly the candidate to a company office, where their compute usage could be casually monitored by an employee.
(b) Use high-quality proctoring services that are nearby to the candidate. E.g., give them 1-2 days in a coworking space, and hire a proctor to verify that thy're not egregiously using tools like ChatGPT.
Or alternatively, would it suffice to just have a long conversation with the candidate about their solution? E.g. what design trade-offs they considered, or how might they adapt their solution to certain changes in the requirements.
This is what privilege looks like. The inability to see barriers that affect others in a worse position.
Standards do not imply fairness only consistency.
You got a test that filters out people who lack the time to train for these tests. Basically, devs with a life (see family)
Is there a solution that you'd recommend?
Is the culture just so broken at these companies that it’s hopeless to expect people NOT to blatantly exploit the system for their friends? Why don’t people get fired for doing that?
How is this any less discriminatory than any other assessment based interview where you need to prepare? Non-assessment based interviews end up being vibes based which is much more discriminatory.
For example, at my current job, we hire web developers with Rails experience. Our technical interview process consists of either a pair programming session or an async/take-home task (candidate’s choice) which requires the candidate to implement a small feature in a Rails codebase. We do have some room to improve on objective evaluation of the candidate’s performance, but there is a test suite and a rubric which we use to evaluate their work. None of this should require that the candidate study, unless they’re coming in to the interview without Rails experience.
If you happen to have Rails experience but it was several jobs ago, the task we give you is basic enough that you should be able to Google what you need during the task to refresh your memory fairly quickly. In fact, I did this when I applied, having not worked with Rails in a number of years.
Edit: My main point is, even if you technically do have to "study" (really, just Google a few basic Rails concepts) if you're rusty, everything you do is preparing you for the actual job. Studying how to implement a hashmap, or computing the longest palindrome from a string of characters, or whatever other harebrained problem FAANG etc want to ask, is 99% of the time not really helping you prepare for those jobs.
For any given job, I believe there’s a way to design an assessment that allows the candidate to demonstrate skills directly relevant to the job, even if the job doesn’t have requirements that are as specific as “Rails developer with a few years of experience.” I have yet to see a single job listing in several decades in the industry where I think to myself “ah yes, the only way to find qualified candidates for the job would be to ask them to implement a quadtree.”
And my "implement a quadtree" example was hyperbole; again I'm sorry if you didn't recognize it as such. I do suspect we have wildly different definitons of what questions qualify as "relevant to the position," though, so I'm not sure how far we're going to get in this discussion.
because most other assessment based interviews are based on what you do on the job, likely stuff you've done at previous jobs as well. Less prep time when you spent thousands of hours already as a career.
Yes. Because interviewing is
1. hard, but no company has proper full time proctors. So "expert interviewers" are a rarity
2. not standardized in the slightest. So your performance varies entirely by the interviewer, their style, and their mood that day.
3. some weird blind test where you hope you studied the right topics. You're not getting the best out of a candidate, you're basically hoping they read your mind and know exactly what you want them to say.
>Is the culture just so broken at these companies that it’s hopeless to expect people NOT to blatantly exploit the system for their friends? Why don’t people get fired for doing that?
Sure. but that's a universal problem. "It's who you know, not what you know" is advice that has spanned centuries. I can't even blame the modern tech industry for that one.
That + the above issues with interviewing mean you're always going to go with a referral over a random applicant.
That isn't always true. When I interview HR gives me the exact questions I'm allowed to ask. These are vetted both to prevent me from asking something illegal, and also by research to get the type of things useful for interviews. Sometimes it is annoying - you can easially finish the interview with a great score but I have no clue if you can write code or not. However we are carefully trained on how to ask the questions and how to grade them.
I was mostly referring to technical tests, but I understand there are definitely some set of questions you need to ask no matter what. I don't really knock recruiters too much for repeating the usual "are you authorized to work in the US" kinda stuff even though it is the first question on their job application.
2. All tech companies I’ve worked at since around 2015 have completely standardized interview questions, sometimes also hosted in a GitHub repo where any employee is free to comment or even open a PR to request a change. Every candidate gets the same questions. This thing where “faang” interviewers just pull a random LC question out of their ass is complete insanity to me. An organized set of questions takes a senior engineer like a week to organize and commit to… And if your questions can be memorized and recited by rote memory and the candidate can do well on it without the proctor knowing, then your questions are BAD or your proctor is a moron.
3. I’ve never worked anywhere where I would describe the interview like this. Even the startup where I was the first engineer and there was no formal “test”, it was an hour long chat with the technical cofounder where he grilled me about coding skills and past experience/accomplishments. I won’t take a job if the interview isn’t asking me things that focus on my existing experience and skills, it’s a red flag about the company culture in general.
As for your “universal problem,” I disagree with fatalistic takes where you just throw your hands up and say “whatever shall we do” all the while YOU are the one benefitting from the system that cannot be changed. This is how simple-minded people think about the world.
2&3. I wish I could relate. I work in games, fwiw, so maybe it's simply more wild west than the other domains in such regardss. It's not even that I can't answer some of the questions, but if I'm asked a question on CPU architecture after I spent my time studying a bunch of C/++ trivia, I'm not going to answer the former as confidently since I'm pulling out knoweldge from a 3 year memory bank that I happened to work with at work.
>all the while YOU are the one benefitting from the system that cannot be changed.
I am? I'm a year out of a job and I sure wish I had a referral. My last studio shut down so those references scattered all over.
But for "whatever shall we do"... I don't have some magic perfect interview, but a few basic steps to save everyone's time:
1. we don't need 5-10+ rounds of interviews and 3 months just to make sure a candidate isn't a lemon. IMO if it's over 3 rounds (not including an HR call for basic alignment), there's some process that really isn't necessary. You dont need to speak to a CEO unless you're applying as a director or executive yourself. You don't need to meet every team member separately for their variation of an interview
2. Ghost jobs absolutely need to be regulated. I'm surprised it's even legal to lie to your shareholders and say "we're always growing" when half your postings have little intention of attracting a candidate. At the very least a job posting should delineate between urgent (hired withing 1-2 weeks of response), "normal" (2-4 week process), and "cold" (6+ week process, non-necessary role).
3. jobs should in fact give a general direction of what kinds of technical questions you expect to be asked for that portion. Who does it really benefit if one candidate happened to study leetcode (and the right kind of leetcode), but the other was focusing on system design, or questions about their specific industry problems solved? Or domain specific probblems?
Just a few ways to be honest, efficient, and bring out the best in candidates. Interviews shouldn't be so hostile to the labor the company clearly requires.
> if your questions can be memorized and recited by rote memory and the candidate can do well on it without the proctor knowing, then your questions are BAD or your proctor is a moron.
I agree in spirit. My only caveat sympathizing with interviewers is that most questions can indeed be "memorized", and arguably all questions/practices can be studied for. And there is some merit to trying to figure out how someone approaches a new problem. I think that approach is highly inefficient when you do in fact mostly want someone who "already memorized" the right domain knowledge, but I digress.
But people and their experiences are diverse. Without knowing their background (and again, some people seem to read the resume for the first time during the interview) you are never going to surprise 100% of your candidates with the same question bank. But that may be too much of a burden on trying to cater every inerview for every promising candidaet.
How do you catch them? Unless they’re idiots they’ll write facially plausible explanations of their hire/no-hire decisions.
It can make the process fairer but it's not a given. You can do the classic "no dogs, no blacks no Irish," and that was a standard across pubs in England. It certainly wasn't fair, unless you were racist.
If you're committed to fairness, then a standard will help. It gives you a clear point to fix and improve things and something you can use to measure if you're achieving your stated goals. That's definitely something you can't do if you're just making things up as you go along.
And yes, I made a deliberately provocative statement. I'm obviously not saying whiteboard tests are the equivalent of segregation.
...a standardized test? No. There are tests. They sure as heck aren't standardized.
Maybe they should be, since everyone seems to be doing the same thing.
They're more like a standardized test than a pure shoot-the-shit interview is, but still pretty far away.