Hacker News Reader

"Learning to get good at something in general" is the real thing these kind of tests proxy for.

There's a lot of things in life that are a pain in the butt, hard, and stupid that you just gotta do.

I went through the leetcode grind a few times, the last time I went through it I ended up taking a job at a startup that I was using as a "practice" interview before the FAANG ones because I could relocate out of California. I never would have interviewed there though if I wasn't laser focused committed to leaving my previous job for a better opportunity.

If you're the kind of person who grinds leetcode, you're _probably_ also the kind of person who spends the extra time reading all the documentation for redux. Or spends the extra time writing a tricky unit test before shipping something. That being said, there's still _plenty_ of people who fail leetcode phone interviews (e.g. myself) but go the extra mile, and FAANG companies still manage to let some questionable hires through the cracks. But engineer hiring is currently a filtering game, not a sourcing game.

You can either learn the rules of Monopoly and play or sit in the corner during board game night and complain you're not playing checkers. shrug

I think this is the most plausible explanation too, but is this true in practice? I would have thought most talented programmers don't want to waste their time on leetcode, so it would primarily select for people who can't succeed without dedicating time to it.

Have you seen this first-hand and compared leetcode grinders against say, prolific open source developers?

Most people in the world care about money, and practicing leetcode is worth way more money per time spent than basically anything else for a talented programmer, up until you hit the level where you can get into top paying companies.

You can get into well paying jobs without it, but then you need skills that are way harder to practice and usually you also need a lot of personal connections as those jobs often selects friends.

I'm not really convinced by this argument. Consistently improving your overall skills will lead you to outpace the competition significantly more in the long run than min-maxing, unless your industry is extremely bad at differentiating ability.

I suppose there's an argument to be gained that the marginal gain of Leetcode given thousands of hours of practice already is the better side of the tradeoff, but that's a circular argument. You're going to select your programmers based on whether they recently spent 30 hours doing Leetcode, rather than on what they got out of the 10,000 hours of regular coding practice they did beforehand? Doesn't seem like a good selection criteria to me.

That's why I asked whether OP had actually seen the effect he was describing and compared results directly. It's easy to come up with hypothetical benefits.

It is a bit tiring to explain the process to everyone, but here it goes:

Companies don't select programmers based on leetcode performance, no, they care a great deal about those 10,000 hours people spent. However if you fail that leetcode test you are out no matter what other experience you have. And it so happens that the set of companies paying twice the average for any experience levels tend to do leetcoding, so no matter what level of experience you have leetcoding is great. Although sometimes your experience might impress low tier companies but not high tier ones, so they downlevel you and ultimately your total compensation didn't move much.

If you haven't seen this process then you are either ignorant, don't live in an area with high tier companies, or don't have a lot of experience. Personally I did double my income by going through this process and I've seen many others do as well.

Now, lots of low tier companies started seeing high tier companies do this process, so they started copying it for some reason. But it doesn't work if you don't combine it with also paying significantly more than most others.

Ok, but companies do select based on Leetcode performance, in practice. It's the largest portion of the interview. It's useful as a filter, but it's treated as a benchmark. As evidenced by:

> Personally I did double my income by going through this process and I've seen many others do as well.

That's insane. That's a broken market. If that's happening, it's not a reliable signal.

Ultimately, if the market is rational then your actual ability will matter more than anything else. The only time that isn't the case is if the market is irrational. You can argue the market is currently irrational (although I would say that's not quite what's going on), but placing a long-term bet on it staying that way... Bad idea, IMO.

But I think the actual explanation for why high-tier (read: large) companies use Leetcode is a combination of the fact that they're monopolies, so they aren't punished for bad selection criteria (see Google's use of lateral thinking puzzles for years before admitting they were completely useless for predicting job performance), and that they need a replicable interview procedure that can be applied consistently by a workforce that fundamentally isn't very good at interviewing, thinks they are, and doesn't care if they get it wrong. It's the Big Mac of interviewing methods.

> The only time that isn't the case is if the market is irrational.

There are other situations, such as when measuring actual ability is sufficiently difficult or impossible (subject to the constraints you pointed out - via a process that's repeatable by a largely undifferentiated set of interviewers).

In those situations, you have basically no choice but to look for proxy measures.

Now, obviously, proxies can be gamed to various degrees. Leetcode is not actually such a terrible proxy, though. Here are some points in favor:

1) On average, better engineers will require less practice time to achieve similar results,

2) There is a certain minimum level of capability required to even "grind it out"; that minimum doesn't serve as a sufficient floor for being a net productive engineer, but it gets you a reasonable chunk of the way there, and that's why so much of the evaluation criteria for these interviews focuses on communication ability, since that's the rest of what they care about

Taken together, you get a process that is reasonably good at eliminating false positives and still manages to tilt the field in favor of more skilled candidates. On the candidate's side, it has the benefits of being generalizable across multiple interviews and also getting easier each time you do it.

While I'm somewhat sympathetic to the claim that the process favors candidates who have more free time, I think it's a largely overstated concern. First, this is true of any interviewing process wherein a candidate can improve their performance through practice. To a first approximation this is all interview processes. Second, it doesn't take _that_ much time. If you put in 100 hours (which is basically "made it a moderately important priority for 2-3 months") and you still can't clear any interviews, there are a few possible explanations.

1) You're failing at the communication side of things. Thankfully this is also something you can practice!

2) You fall into an unfortunate edge case, e.g. extreme performance anxiety, which isn't reflected in your day-to-day work. This sucks! I don't think there's a process that doesn't have unfortunate edge cases; all we can do is try to minimize them.

3) You're not able to learn the material well enough to generalize it to novel interview questions. This is the process working as intended.

I've also heard a reason that goes something like "I can totally solve those problems, just not in 45 minutes". Problem-solving speed is going to be positively correlated with other traits of talented engineers; obviously some otherwise talented engineers will still fall on the right side of the curve. Again, this sucks, but please bring me a process that effectively rules out false positives without bringing in some false negatives. If you can manage it there's a huge market opportunity there.

These are interesting points, but they very much read like post-hoc rationalisations to me. Let me take a different tact, I want to try challenging some of the more fundamental assumptions.

Do you really believe in the quality of Leetcode as an evaluation criteria, or are you trying to justify it? Google used lateral thinking puzzles for something like 8 years before realising they were completely useless (not just unreliable - they had zero predictive power). We have precedent for these companies using selection methods that are bananas, and smaller companies copying them.

>There are other situations, such as when measuring actual ability is sufficiently difficult or impossible

Sure, that's an example of why the market might be irrational. But then I'd ask: do you think it's extremely difficult/impossible to judge other programmers? I mean that seriously. I don't find that difficult. I can usually size up another programmer pretty quickly. If I sniff around a bit and look at some example code, my intuitive read will be much more three-dimensional and reliable than giving them two Leetcodes. You don't find that to be the case?

How much do you spend on hiring one candidate? I'm guessing tens of thousands, if not more. You can't invest a day into trawling that candidate's GitHub? "But not all candidates have a GitHub." Ok, but some do, so that's not the reason.

What I think is actually extremely difficult is a large, beurocratic organisation developing a replicable, second-hand evaluation system. It's a standardised test, and it has the same problems as most standardised tests.

> you get a process that is reasonably good at eliminating false positives

Right, it's good as a filtering mechanism. But if it's just a filtering mechanism, it shouldn't take up the majority of the interview.

Like, here is one of the assumptions you're making: most good engineers grind Leetcode before a new job. I don't think that's true. I think the number of people who grind Leetcode, even among the best candidates, is extremely small. Like <10%. If that's the case, it's inherently a very flawed criteria. If nothing else, you're filtering for the candidates that everyone else is filtering for - which means you're filtering not for capable candidates, but for candidates who are overpriced.

Here's another thing that's absent from these discussions: algorithms questions were developed before Leetcode existed. They're not designed for a world where you can game them. Leetcode is a bug in the system, but a lot of your justifications are, "it's fine, because the selection criteria is actually designed to select people who game it." Really? You want people to game the criteria? I don't buy it.

I don't consider myself particularly talented, but it's really not that much time to practice. At 10+ years of experience, I spent maybe up to 40 hours of practice the last couple of times I switched to interviewing mode. Pretty easy to spread out over 2 weeks to a month to get in shape. I don't consider that an unreasonable or unacceptable burden.

It's not that it's unreasonable, it's that it's a bad criteria because it's so easy. In 40 hours, a college graduate can get to the point where they're outdoing, say, one of the principal architects of AlphaGo? Then that selection criteria sucks. In any other situation we avoid metrics that are easy to game.

The argument that it selects for industriousness strikes me as a cope. It might, but it's also so easy to game that it obscures actual ability (unless you're just using it as a filter). Is that trade-off really worth it?

So, i was responding to your comment:

> I would have thought most talented programmers don't want to waste their time on leetcode, so it would primarily select for people who can't succeed without dedicating time to it.

I was saying that it's not that much of a waste of time, because it's not that much of a time investment for a good outcome (doing well at interviews).

If you want to change the argument from "it's a waste of time" to "it's trivially easy and easy to game", then why are there people who complain about leetcode and not wanting to do it?

It is both a waste of time and easy to game. The reason it's a waste of time is that it's easy to game, so some of your rivals will be gaming it. Which means you have to do it too, so it just becomes a dumb tax you pay every time you interview for a new job.

That's all from the individual candidate's perspective, of course. From the employer's perspective, you're still only going to have a minority of your interviewees having prepared their cheat codes. So you're evaluating based on ability to prepare a cheat code. Which IMO is retarded. If there wasn't a site dedicated to preparing cheat codes (which is how it used to be), then it would be sensible.

Hmm, i think your understanding of the state of things is slightly backwards. Because Google is so large, interviews so many people, has such a strong brand name, and pays so much more than everyone else, their interview process is going to be a very well-known quantity. Therefore, the questions that they used to use all were being gamed, so they kept on adjusting and tweaking and upping the difficulty, until they ended up with where they are now.

So the point is, how do you interview if you know that everyone is going to know all the questions ahead of time? Seems like google's solution is they'll just make their questions difficult leetcode level. So, then their thinking is, sure if you're able to grind leetcode to a point that you can pass our interviews, then we'll be perfectly happy to hire you. So you've "cheated" the system by learning how to program a wide variety of difficult algorithm-heavy programs. Worst case scenario, your resume is a complete sham, but at least you're able to write the code that they're hiring you to write, and everything else that you're lacking, well, they'll be indoctrinating you in the google way anyways, so they don't care about your past experience.

That doesn't sound like an effective pipeline to me. That sounds like a really bad pipeline, and a lot of post-hoc justifications to minimise the issues.

I'm hesitant to take Google's use of a tool as an indication that the tool is good, or that the process is refined. They used brain teasers for years before figuring out they had zero predictive power. Ignoring the implications of the scale of that mistake, one thing it indicates is that they aren't able to effectively evaluate different interviewing methods.

If they had no ability to effectively evaluate, how did they conclude that brain teasers had no predictive power?

I see google's process along the same lines as "democracy is the worst form of government except all those other forms that have been tried".

Now ideally, what you'd want is to be able to perfectly read someone's mind and someone's intention, to be able to tell instantly their strengths and weaknesses, to be able to tell which of those weaknesses are trivial ones that will get smoothed over a week into the job, and which of those are ones that are actually long-term issues that will poison your organization. It's possible that individuals with such judgement exist (Paul Graham has said that YC's success was attributed to Jessica's ability to judge founders) and that they work at google. But now you also need a process that can interview and evaluate a thousand people every single week. So what works for small companies where you can agonize over every hire or for exec searches, doesn't make as much sense for mass hiring of peons.

So given those constraints, everyone knows your process, everyone wants to apply, you need to interview a thousand people a week and decide to hire a hundred of them, and they really don't all need to be rockstar founder quality, but they all need to be able to produce something, you can start to imagine what type of interview process you might end up with.

> they've trained themselves to look at a program and see the data structures rather than the code, and in my opinion that's a better way to think about a program.

I don't see how this is mutually exclusive to writing readable code. I'm not good at these but I picture good code in your definition to be utilizing data structures in an intuitive way instead of coming up with some novel but wacky performance trick.

Yeah, it makes no sense to me. It's like saying if you stare hard enough you can see into the matrix.

For me, i'd sacrifice raw performance for using Python because i love its simple datastructures and protocols and the compbination of code and interpreter is great... and more often then not, it's not much of a sacrifice since compute time is often more affordable than creative programming time...

I think as somebody wfo too as competed in ICPC albeit not at world finals, I just want to add an important caveat, getting to world finals has a lot to do where are you from, for example if you are from a historically tough university or place such as India, Belarus or China. It is exponentially harder since due to how ICPC works the rules make it a piece of cake if you are from an unnamed college in norther America but much harder if you are from an established IIT in India or from somewhere like university of Tokyo.

Except there’s not much competitive programming bench strength in India (happy to link to ample evidence). Really facetious to compare it with China where the bench strength is real, and those thrown by the wayside could have otherwise killed it.

> It ("getting to world finals") is exponentially harder since due to how ICPC works the rules make it a piece of cake if you are from an unnamed college in norther America but much harder if you are from an established IIT

Rephrased as I understood it in context, it says: "unnamed colleges in northern America has better chances at getting to WF than India, Belarus, or China".

Which is a surprising statement to me. At my time (2008-2012, followed it a few more years thereafter), the heavy hitters were from Belarus/Russia, China, and India to a smaller extent. I remember getting news that Shanghai Jiao Tong really dedicates time to training, almost like a sport. Petr Mitrichev is Russian and tourist (forgot his real name) is AFAIK Belarusian. At least Mitrichev coached in grassroots programs for IOI which then translated to ICPC performance. Maybe India wasn't as dominant but I believe they were at least a considerable contingent in the Asia-regional levels.

So it seemed to me the competition format favored these countries over US universities, which focused on research. Not to mention China and Russia loved hosting big ICPC events, if not the World Finals itself.

Do elaborate on your statement please. I am curious why you'd make that claim.

You can check out the rating of teams going to world finals in previous years.

https://codeforces.com/blog/entry/73791

https://codeforces.com/blog/entry/64909

The problem is that certain regions just have too many slots relative to their density of competition programmers.

For example in less competitive regions, a rating of around ~2100 gives you a decent shot at world finals.

But if you were in a region with a lot of other good schools, then even if you had a rating of 3000 you might not even be able to represent your own school, let alone your region. See MIT's result where their top 3 teams dominated their region but their school can still only send at most one team to world finals: https://nena20.kattis.com/standings

Isn't this how most international competition works? I'm most familiar with the olympics and this is exactly how it works there. People from uncompetitive countries for their sport have an "easier" time going to the olypmics.

lol my team even made to this stage here - https://nadc21.kattis.com/standings- and we did practically nothing, haha

it's Macalester btw

The idea is that the bar for getting there is much higher - if you're from China or "Northern Eurasia", you get to the finals only if you dedicate time to training and can defeat those "heavy hitters" since you have to compete with them for scarce slots long before the final, but if you come from an unnamed college in northern America, you have a chance to get to WF without being at that level.

>look at a program and see the data structures rather than the code, and in my opinion that's a better way to think about a program.

that sounds pretty difficult for me to understand

when you read the code, analyze and understand it and due to your experience you see that something is bad (e.g poor api), then how can you "lose focus" from it / ignore it?

how it ain't just lack of experience or simply not caring?

It is a question of priority, I think what GP is saying is that programmers used to these contests will focus on getting their structures right first, and will take care of logic on a second time, if they have time for it. In programming contests, they don't have time.

And I think it makes sense and that is a hierarchy. Data structures are the most important, followed by logic (algorithms), followed by comments.

The reason is that if you change your data structures, you need to change your algorithms, and if you change your algorithms, you need to change your comments. Data structures define everything, comments define nothing but themselves.

Yeah exactly, thanks. I think I should've said "and focus on the data structures" instead of "and see the data structures"