Hackerrank was broken – but now it's harmful | Hacker News Reader

Hackerrank was broken – but now it's harmful

segfaulte.mataroa.blog

59 pointslunarcave67 comments

I've said it before and I'll say it again here - the best job interview I ever had for a software engineering position was about 10 years ago. I was shown a piece of code (directly from their codebase) which was about 50 lines long and we had a free wheeling technical conversation around it in which the interviewer asked me the following questions:

- What does this code appear to do (both technically and in the context of the business domain of the company)

- Are there are any red flags, edge cases, etc. that this code might not cover?

- If I were to rewrite it in the using modern libraries/tooling/best practices, what might I have done differently? High-level - we're not looking for you to physically write syntactically correct code here.

This is the same interview process that I now use on prospective applicants. In a single 30-60 minute dialogue it helps me answer questions such as:

- Can they size up and summarize existing unfamiliar code bases? In enterprise the ability to deal with legacy code is often more common than writing new code.

- Can they re-interpret old code using a modern lens? For example if we're discussing JS/TS (requires vs imports, callback hell vs async await, XMLHttpRequest vs fetch). For python (nested lists vs dataframes, simple numpy regression vs pytorch, etc)

- Are they able to communicate and convey their thoughts in a coherent and logical manner?

This is brilliant. I'll be stealing this for my next interview.

This works when "game recognizes game". But when you have HR or shitty developers doing the interview it doesn't.

The author says "whiteboard tests" are broken, but it seems like they're arguing that online coding assessments are broken, not in person interviews using an actual whiteboard.

Doing an in person interview on a whiteboard sidesteps the AI issue. As someone who's done a large number of remote interviews, there are some clear signs that some candidates try to cheat online tech interviews. I wonder if the trend will fuel more of returns to the office, or at least a return to in-person interviewing for more companies.

> or at least a return to in-person interviewing for more companies.

This has been broken for a while now, and companies still haven't reset to deal with it. The incentives to the contrary are too large.

The disincentives are huge though. Hiring a bad employee is a very expensive problem and hard to get rid of.

Isn't it as simple as going on pip for fangs, a short conversation for a founder of a startup and a few weeks notice pay?

That comes after the decision that you can't fix the situation, which comes after you discovered that the hire was bad, which comes after a number of visible failures. That's a lot of wasted time/effort, even if the firing itself is simple.

The cost of hiring, firing, rehiring approximates the position’s yearly salary.

Depends on the country I think - in Australia at least it seems like you can sue for unfair dismissal if you're angry about being kicked out, so HR departments only seem to get rid of someone as a last resort.

In my area they just tell you to leave. No warning. No severance. Midwest US.

The process is anything but simple at large companies. Even if the new hire is a complete fraud and can barely write code it'll still take an average manager 6-12 months to be able to show them the door. And it'll involve countless meetings and a mountain of paperwork, all taking away time from regular work. And then it'll take another 6 months to get a replacement and onboard them. That means your team has lost over a year of productivity over a single bad hire.

In which country?

In France, for instance, you have a (typically) 6 months long no questions asked window to fire a new hire, if they prove a bad employee. Presumably, if you haven't found out in 6 months, you wouldn't find out by changing the interviewing strategy.

Is using AI cheating when it's part of the job now. Is not using AI signalling inexperience in the llm department.

Yes, obviously. Cheating is subverting the testers intent and being dishonest about it. Not just what a lawyer can weasel word their way around.

It’s not dishonest, it’s just business. I’m under the exact same burden of truth as the company interviewing me; zilch.

Fair enough! In this case, it seems that policy "hard-fail any candidates who use cheat, using AI or otherwise" is working as expected. Interviews are supposed to be candidate's best showing. If that includes cheating, better fail them fast.

It's cheating if you don't say you're using it.

At some point I assume that it’ll be so normal that you’ll almost have to say when you’re not using it.

I don’t need to say that I’m using a text editor, instead of hole punched cards. It’s also quite common to use an IDE instead of a text editor these days in coding interviews. When I was a student I remember teachers saying that they considered an IDE as cheating since they wanted to test our ability to remember syntax and to keep a mental picture of our code in our heads.

I wonder if OpenAI/Google/Microsoft, et al would hire a developer that leaned heavily on ChatGPT, etc to answer interview questions? Not that I expect them to have ethical consistency when there are much more important factors (profit) on the table, but after several years of their marketing pushing the idea that these are ‘just tools’ and the output was tantamount to anything manually created by the prompter, that looks pretty blatantly hypocritical if they didn’t.

Amazon uses Hackerrank and explicitly says not to use LLMs. In that case it would be cheating. However, given that everyone is apparently using it, I now feel dumb for not doing so.

They made tools to make us redundant and are upset we're forced to use those tools to be competitive.

That's actually a valid question. It looks like it was an unpopular one.

Personally, I despise these types of tests. In 25 years as a tech manager, I never gave one, and never made technical mistakes (but did make a number of personality ones -great technical acumen is worthless, if they collapse under pressure).

But AI is going to be a ubiquitous tool, available to pretty much everyone, so testing for people that can use it, is quite valid. Results matter.

But don't expect to have people on board that can operate without AI. That may be perfectly acceptable. The tech scene is so complex, these days, that not one of us can actually hold it all in our head. I freely admit to having powerful "google-fu," when it comes to looking up solutions to even very basic technical challenges, and I get excellent results.

Copy pasting code from ChatGPT doesn't mean you have any kind of understanding of LLMs.

So now there are job applicants not only pretending to know DSA by using ChatGPT, but also claim they have "experience in the LLM department".

It's not part of the job now, unless you're too inexperienced to estimate how long it takes to find subtle bugs.

Depends on what kind of developer are you trying to hire, maybe.

If your coding assessment can be done with AI and the code that the candidate is expected to write can’t be, doesn’t that by definition mean you are testing for the wrong thing during your coding interview?

Every people says these, but what is the best objective way to know the candidate is good for position? Leetcode is still the best option imo.

Give them a real world simple use case where they have code and they have to fix real world code by making the unit test pass.

Never in almost 30 years of coding have I had to invert a b-tree or do anything approaching what leetCode tests for.

Well actually I did have to DS type code when writing low level cross platform C in the late 90s without a library.

But how many people have to do that today?

And how is leetCode testing the best way when all it tests for is someone’s ability to memorize patterns they don’t use in the real world?

Yeah its weird, because the whole point of having a system for hiring involving common questions, rubrics, etc is because at the end of the day you can either show that scoring well on the interview is correlated with higher end-of-year performance reviews or not show that and alter your interview system until it does.

Like you guys can keep posting these articles that have 0 statistical rigor. It's not going to change a process that came about because it had statistical significance.

Do remember, Google used to be known for asking questions like "How many piano tuners are in NYC". Those questions are gone not because somebody wrote a random article insulting them; they're going because somebody did actual math and showed they weren't effective.

Yes, because of Google’s rigorous hiring process they have had many successful products outside of selling ads against search…

I’ve done my stint in BigTech, most developers are not doing anything ground breaking

Absolutely. We've switched from (much simpler than hackerrank) coding tests to debugging problem. It relies on external API, so we get to see candidate's train of thought, and naive methods of cheating using ChatGPT are trivially detectable.

But this is arms race of course. I have no doubt LLMs could solve that problem too (they might be able already with the right prompt). And then we'd have to make to even more realistic... How does one fit "here is a 1M lines codebase and a user complaint, fix the problem" in a format of 1-hour interview? We'll either have to solve this, or switch to in-person interviews and ban LLMs.

Is it already that time of the week?

People were complaining about whiteboard coding interviews in the 1990s and they are complaining about it today. Meanwhile the tech industry has managed to hire millions of incredibly smart people and done pretty well for itself in that period. The interview process isn't going to change. There is no reason for it to change. Your options are to either suck it up and brush up on basic data structures and algorithms or interview at places that don't test for it. Just don't hold your breath waiting for the world to change for you.

That’s not what this article is about. It actually suggests people who love whiteboard interviews are losing here.

> Your options are to either suck it up and brush up on basic data structures and algorithms or interview at places that don't test for it.

The article is about the fact that if you did just this, you're at a disadvantage because people use Chat GPT to cheat on these and get ahead.

The article is bizarrely conflating whiteboard interviews with online coding tests and even admits such:

Some people have (rightly) pointed out that I'm interchanging whiteboarding (writing on a physical whiteboard) and online coding tests interchangeably

Your point being?

I'm not commenting about the white boarding.

I hired hundreds of people without coding sessions, because although you are right, they also hire a lot of bad people, who are just good at selling themselves. As bigger the company gets, as easier it is to deal with the unnecessary cost of wrong hirings, it's just a number game.

So no, I didn't wait for the world to change for me, I changed it myself ;).

AI is likely to shape the future, gradually embedding itself across industries, fields, processes, tools, and systems. Similar to the introduction of electricity, where identifying practical applications was inevitable, trial and error will be an essential part of its evolution. While concerns about its impact are understandable, it's important to recognize that each generation often worries about the capabilities or standards of the next. However, such concerns are typically rooted in current frameworks of thought, which may not remain relevant in the future.

Right now, generative AI looks to be more like microwave ovens than electricity -- usefully novel but not obviously revolutionary or all-pervasive.

Even with the remarkable convenience and new opportunities that microwave ovens brought to kitchens, it still made a lot of sense for commercial kitchens and culinary schools to keep focused on more traditional fundamentals and their mastery when hiring and training candidates.

Same applies to our field for the foreseeable future.

I like this metaphor.

I suspect that GenAI will eventually reach a point where it's actually really useful, but we aren't quite there yet.

>Since everyone started using AI, more candidates started clearing the first round with flying colors. The platforms had to recalibrate to let in their target percentage.

Is there any proof or data on this? Not saying it's wrong but I'm curious how big the effect actually is.

I'm not convinced it would be caused by AI. More people are laid off, so more unemployed people can grind leetcode all day, and more people are competing for fewer jobs.

The dirty secret of hiring processes is that the main goal of the earliest stages is to get from a lot of applications to something less than a lot, not to screen for fit. This has long been true, and it is now super true.

Every job posting gets flooded with hundreds of applicants, if not into the thousands. Most of those people are coming through the front door with no one to vouch for them and probably nothing on their resume that makes them a must-see candidate.

Most managers and HR teams probably don't even explicitly think about it this way, but by pure pragmatism have evolved processes that act as flow control.

The unacknowledged result is that the company will reject 90+% of applicants, regardless of fit, under the assumption that the filtering process will allow enough good people into the actual interview rounds that the team will be able to find someone they want. From this perspective, Hackerrank is not broken, it's doing exactly what is required of it by companies.

I say all this because people who are in job search processes should frame the process accurately in their mind. It hopefully will help with not taking the process so personally or not getting so infuriated with it. It may also help you strategize how to find your way into the roles you want, if crushing these tests isn't your strong suit. People who are vouched for get to bypass all of this. The more confidently you are vouchced for by a trusted party, the more benefit of the doubt you get in the hiring process.

One might ask is there a better way to do this? Probably so. But if it were easy, it would already exist.

There would need to be an easier and cheaper way to filter applications.

One option for candidates is networking, this gets you in through the "vouched" side door.

This potentially means that the company should be encouraging employees with more than financial incentives to find candidates and recommend them. This means networking workshops, time swaps for attending networking/recruiting events, understanding the need to make synchronous contact with people who might be good candidates.

If you are interviewing based on what you do, and what the job application (which has been mutated to get through HR's posting requirements, but don't understand why and how you should be interviewing you are more likely to bring your past experience and biases to the interview. This is bad for your company and for candidates.

From hiring perspective all of that just sounds like tons of more work to find any candidates.

The end target is automation. The cost of doing business is the cost of doing business until then.

And then if you don’t know the “right people” you will never get a job.

It’s like VCs who want to pattern match for someone who looks like Zuckerberg

> I say all this because people who are in job search processes should frame the process accurately in their mind. It hopefully will help with not taking the process so personally or not getting so infuriated with it.

For people who consider honesty a part of their identity, it's infuriating that the system has a selection bias for dishonest people. It's not selecting a random 10%.

> It may also help you strategize how to find your way into the roles you want, if crushing these tests isn't your strong suit. People who are vouched for get to bypass all of this. The more confidently you are vouchced for by a trusted party, the more benefit of the doubt you get in the hiring process.

Networking is still selection bias, but one that favors white men. Racial minorities have fewer social connections to hiring managers, and women in tech have fewer friends working in the same industry.

> For people who consider honesty a part of their identity, it's infuriating that the system has a selection bias for dishonest people. It's not selecting a random 10%.

I don't think that's quite right. There may be ways to get temporary advantage from being dishonest, but I think that it is career-limiting for most people.

If by dishonesty, you mean using AI tools, I think that the reality is that we should consider all workers to be AI-augmented, going forward. We may as well not pretend that that's not the case.

> Networking is still selection bias, but one that favors white men. Racial minorities have fewer social connections to hiring managers, and women in tech have fewer friends working in the same industry.

I agree, and as a Black guy, I relate. Being mid-career now, I'm on the other side of that, where being well connected is immensely helpful.

I'm not saying it's a good system, but people still need to understand the game they're playing.

i screen candidates all the time (not involved in technical rounds) and notice chatgpt cheating all the time but its easy to spot because the candidates read off answers in mechanical ways right after a flurry of typing

I recall having to implement A* to search a nxn character grid in my AI course a few years ago. It took me close to a full day to wrap my head around the concepts, get used to python (we usually worked in C++), and actually implement the algorithm. Nowadays, an LLM can spit out a working implementation in seconds.

I think that's a big part of the issue with tests like Hackerrank - LLMs have been trained on a lot of the DSAs that those questions try to hit. Whereas, if you ask an LLM for a truly novel solution, it's much more likely to spit out a garbled mess. For example, earlier today, Google's search AI gave me this nonsense example of how to fix a dangling participle:

> To correct a dangling participle, you can revise the sentence to give the dangling modifier a noun to modify. For example, you can change the sentence "Walking through the kitchen, the smoke alarm was going off" to "Speeding down the hallway, he saw the door come into view".

LLMs have effectively made it impossible to test candidates for crystalline intelligence (e.g. remembering how to write specific DSAs quickly) remotely. Maybe the best solution would be to measure fluid intelligence instead, or front-load on personality/culture assessments and only rigorously assess coding ability in-person towards the end of the interview cycle.

So if I asked you how A* works in an interview you you be able to explain it. Johnny ChatGPT would not.

In most cases it really doesn't matter. You wanted A* and you got it. Understanding isn't important if the product works.

Don't onsites or virtual onsites also have coding rounds? would be pretty easy to figure out who is using llm and similar tools.

IME you screen people online, get some reasonable candidates, get them onsite and they can't code to save their life. But it's your problem then.

Seems a problem that will sort itself out if companies are in fact hiring under-qualified cheaters.

please bring in person interview. I am sick and tired of cheaters. They rig every thing ....

TL;DR: best case LLMs break algo-whiteboarding interviews so visibly we stop using them (they're already invisibly broken).

Some engineers are genuinely better at using ChatGPT and friends to solve problems, and that's a skill worth including in evaluation of potential software engineers. I would advocate interviewers go with the flow, allow use of LLMs, but have candidates do it on the "shared screen" to see how they think - just like using a web browser in interviews.

I've been a professional programmer since 25 BAI (25 years before AI). I see junior engineers who use it disastrously to create piles of messed up spaghetti code, and junior engineers who use it skillfully and judiciously to create good code and solve problems far above their years of experience. Both of them will use LLMs in their daily work. One of them will destroy the codebase, and the other will far exceed the expectations we had for a "2 yoe eng" only 2 years ago. I want to identify the latter, and filter out the ones that aren't good with ChatGPT and will use it to make messes.

I want to identify experienced Senior Engineers who kick ass. They can do it with or without ChatGPT, I don't care which, and I don't want an interview format that doesn't let those who use ChatGPT show their true strength. I simply don't care how well you can recreate (and pretend to have rediscovered lol) PhD thesis algorithms with no internet access on memory alone: that is not a relevant skill in 2024.

I won't cry if "allow LLMs but watch how they use it / think" doesn't scale to "no skin in the game" offline tests where interviewers try to get candidates to spend an hour without the company matching that hour with an hour of employee/interviewer time. In fact, if (and it probably will) this kills offline testing, I think that's a win for labour.

What ChatGPT "cheating" reveals, imo, is how weak and artificial "whiteboarding" interview questions are: they claim to be problem solving tests but they're actually algorithm memorization tests. Interviewing without allowing use of LLMs is like requiring "no web browsing during interview": it artificially biases toward test-specific skills and away from real world strengths. If an LLM can straight up solve the problem, its not a very good match for skills required in 2024: its now a problem that SHOULD be solved via LLM.

It turns out that LLMs are better at rote memorization than we are. Good! The "whiteboarding algos" interview technique was already a bad match to real world SWE skills except in relatively rare job roles, and its a good thing that its artificiality caused it now be irreparably broken by ChatGPT. Bye bye, won't miss you ;-)

Most bog-standard software engineering work is boring things like good variable naming, effective use (and non-overuse) of encapsulation, designing a schema that won't make you hate life in 5 years, wise selection of dependencies, etc etc. Testing live with a candidate, watching them use ChatGPT, seeing how they think, this is what gets you a job-relevant-skill signal.

I think its time for the federal government to get involved, and it already has leverage

If your company benefits or plans to benefit from QSBS tax treatment, then evaluate engineers on things they’ll be doing in the company:

the application development process, sprint ceremonies, asynchronous communication and tooling

they wont be implementing data structures often, and if they are reinventing the wheel they are wasting company time, if your company needs a more efficient data structures for its super scalable problem - it should trigger a process audit to be honest.

looking forward to Leetcode Prohibition Act of 2029

First let them get rid of non compete agreements. Haha.

2025 lets go, new administration is up for anything and has no clue what theyre doing

The center is unguarded I bet I could get the 119th congress to pass anything since all the moneyed interests went to far extremes of their parties and will never get their causes passed the filibuster

I don't understand how QSBS has anything to do with interview practices. Are you saying the IRS should somehow audit any QSBS company to make sure they're not implementing any data structures or algorithms? Like, new companies must be audited if they try to use computer science?

Control behavior by regulating the intermediary, constitutional under any governance system, my favorite approach to anything is to look at what the thing relies on, and controlling access to that thing.

We can even add whistleblower bounties to current and prospective employees. The IRS already has a bounty program, probably doesn't even need Congress for this one. Its under the Treasury Department right? Can probably still Scott Bessent on linkedin about implementing it on Day 1

So, yes. You got it. Play devil’s advocate all you want, I’m ready. More likely you’ll have to put your reservations in during the comment period.

"Let's effectively prohibit all non-CRUD apps" that's hilarious. "If your startup needs to solve hard computer problems, you're going to get an IRS audit". wild. It's like the 90s cryptography export laws, except it applies to domestic development too!

it can promote training employees again instead of “screening against false negatives” or whatever dumb interviewing rationale needs to be eradicated

Reply on news.ycombinator.com