The impact of AI on the technical interview process | Hacker News Reader

The impact of AI on the technical interview process

coderev.app

> The system design ones were somehow even more outrageous given that system design is often the deciding factor between a mid/senior level and a staff/principal level engineer. Yet watching the IGotAnOffer video series successively reveals just how formulaic and predictable the response and banter is designed to be.

Oh my god, yes. Every architectural interview I’ve had in the last five years has been a rehashing of the same concepts, pulled straight out of Designing Data Intensive Applications. Anyone who has read that book once or twice can easily pass. Half the interviewers even use the Twitter example presented in the book, with a few arbitrary extra requirements tacked on.

To be fair, DDIA is a pretty deep book, so if you read it, took something out of it, and read it the second time (!), you probably deserve an offer.

There's very little room for deviation as well because it is so formulaic. Present a novel and interesting system? It may not be realistically solvable in a 30-60 minute session to a satisfying degree because the candidate doesn't have familiarity.

From the candidate side, you can take a risk and make it more interesting by presenting more novel solutions, but the interviewer might be looking for very specific response patterns; "signaling" of a sort. So both sides just stick to a narrow range of scenarios and responses.

Once you've done one or two, it's hard to see how you couldn't do every other one after it.

> pulled straight out of Designing Data Intensive Applications

I mean, because it works. Why would you create a novel system just for the sake of it?

I think the root of the problem is actually an org design problem.

Big companies want consistency in their process and their hires. There's a lot of benefits if you can create a Google type hiring pipeline you're just constantly interviewing candidates and you worry about where they go once they've passed the interview, like:

- Since all candidates at the same level are passing essentially the same interview you can move people from team to team easier. You reduce the risk of a team with lower standards causing you problems when their engineers are moved to other places in the company. You know that most of your hires have a certain consistency in their skills. This makes things like reorgs a lot easier.

- Your recruiters and hiring managers save time because in most cases they don't need to meet to set up unique hiring pipelines per team. You just have one big hiring pipeline and teams with open spots only step in at the last minute.

- You reduce the risk of being sued for bias or discrimination. If interviewers are just pulling from a bank of questions that are somewhat equal it's hard for a candidate to make a case that they were singled out and given a tougher question because of their identity.

The problem with that is that consistency breeds predictability. The only way to get a big pool of interview questions that are about the same difficulty is by repeating the same patterns between questions. Once candidates know those patterns, candidates can just focus their studying on that and the interview just boils down to pattern matching.

I think it would help if companies approached system design interviews less as an opportunity for the candidate to show off all of the fancy performance optimizations they know and more as an opportunity to demonstrate that they can actually design and lead implementation of a system.

That is to say, real world system design is more about understanding that most systems don't need most of the stuff in Designing Data Intensive Applications and avoiding premature optimization.

> reveals just how formulaic and predictable the response and banter is designed to be

Accurate. Interviews have to be formulaic and predictable. That's how you combat bias in the system.

Kahneman's book on Noise (in decision making) has a whole chapter on interviews. With careful design to avoid bias you can get to about 75% accuracy measured as correlation between Yeses and Job Performance. Without having repeatable interviews judged against a pre-defined metric, your signal becomes meaningless. You're basically flipping a coin so why even bother.

Or worse, you get people passing interviews because they went to the same school as the interviewer or whatever. In that case your un-designed un-formulaic interviews become literally worse than random.

Yeah, I think interviews are to get some signal of competence and ability to prepare / offer some filtering ability even if they're a bit arbitrary or banal.

Even before AI tools, anyone can write code. It's not impressive, and as an interviewer you're not going to gain deep insights on whether they're a developer you want to work with by watching then squirm with some boring leetcode exercise.

We've landed on a hybrid approach of reviewing a small homework exercise they were sent prior in an interview and find it a lot more useful to ask them to explain their approach and use it as a jumping off point for other questions. When's the right and wrong place to use arr.reduce()? Why did they need event.preventDefault() in that form submit handler?

Nothing's perfect, but I do think asking people to explain their code to be much more resilliant to those who treat coding as a rote exercise, or just blindly ChatGPT-ed their way through.

> We've landed on a hybrid approach of reviewing a small homework exercise they were sent prior in an interview and find it a lot more useful to ask them to explain their approach and use it as a jumping off point for other questions. When's the right and wrong place to use arr.reduce()? Why did they need event.preventDefault() in that form submit handler?

I applied this approach recently and it was... depressingly useful. Sent out 3 short tasks which should take an experienced programmer less than 30 minutes to solve in total: one "find, fix, and explain the problem in this code" and two "write the ~50 LOC to implement these exact specifications". These went out only to those whose resumes looked like a potential fit.

Something like half of the responses we got back didn't even compile/run. Mostly due to hallucinated library functions.

One candidate had decent responses, so we went on to the full interview, asking (as you mentioned) if he could explain his approach. He was having some trouble so I just asked him to explain how a sync.WaitGroup (which he'd used in his code) actually functions, and we got treated to an intensely painful few minutes as he desperately tried to spin some BS about how important concurrency is, and how Go's so powerful at concurrency, etc etc etc. but nothing about what a WaitGroup actually does.

I don't really know how to solve this shit going forward.

Just need to keep interviewing more people. Ironically, AI might make (competent) devs more valuable, not less, as lots of them will simply be filtered out because they use AI as a crutch, thus the overall pool of software engineers decreases (and supply is constantly increasing). It's akin to the COBOL situation at banks, they don't have enough competent COBOL devs (as new devs don't learn the language and older ones die off) so they need to pay out the ass to get them to work on their systems.

This is s side comment, but banks don't actually pay out the ass to get people to work on COBOL. That's a myth. They actually just pay people (sometimes programmers, sometimes not) to learn COBOL. (And pay them fairly unconpetetive salaries, because COBOL is not really all that useful.)

They do pay a lot of money to the people who program COBOL and maintain their archaic IBM mainframes. But that's mostly because it's basically a giant proprietary stack (hardware-OS-softare).

> They do pay a lot of money to the people who program COBOL and maintain their archaic IBM mainframes. But that's mostly because it's basically a giant proprietary stack (hardware-OS-softare).

I meant more so this.

I actually think you've found the answer on how to move forward:

    > He was having some trouble so I just asked him to explain how a sync.WaitGroup

If understanding of concurrency is a key selection criteria, introduce some obviously broken code in a code review and see if they can find it and explain why it's broken.

The reason I asked him to explain sync.WaitGroup was because we were trying to discuss the homework problem which was intended to demonstrate knowledge of concurrency. "His" code used WaitGroup correctly throughout! But he wasn't able to explain how the code worked... like at all... so after 5 minutes of getting nowhere I tried to narrow down the problem space a bit and asked about WaitGroup specifically.

When I interview for C# candidates, I have a series of questions that I like to ask 1) what are `Func<T>` and `Action<T>` 2) write a function that takes a `Func<T>` or an `Action<T>` (any function; can be trivial), 3) write a function that returns one of `Func<T>` or `Action<T>`.

The reason is that over various interviews, I've noticed that candidates that can answer and write the code are going to be able to read, understand, and write the level/type of code that I expect.

I bring this up because I think when teams use projects for assessments, there are clearly certain things that you look for or you would see and be able to tell that this candidate knows what they are doing.

Rather than doing all the extra work, to me it seems like you can short circuit that process by just directly focusing on the 4-5 things that -- if you saw them while reviewing the code -- would tick the right boxes. My point here isn't necessarily `WaitGroup`, but that maybe there are some topics that the process could be reduced down to that would yield the same learnings.

It looks like you are doing a pretty good job of filtering with this approach.

Yeah, I'd say it's working, but if I can use an analogy: it's always taken some care and inspection to pick out the un-bruised apples, but now half of the bin is also made up of pretty damn convincing plastic replica apples too, and even though I'm getting good at spotting the plastic ones it's an annoying additional hurdle to the task.

> Even before AI tools, anyone can write code.

If only that was true. As an interviewer, I had quite a few instances when candidates couldn't write any code (especially during initial screening). And sometimes you wonder how did they get to the onsite.

I have a lot of things to say about modern leetcode style interviews, but there clearly must be some place in the process, where candidate is actually writing some code in front of an interviewer.

> anyone can write code

I used to give the most boring, not-leetcode question, about how to lay out a minesweeper board of arbitrary size with an arbitrary number of mines (not any gameplay code, purely the initialization of the layout), and at least 95% of the candidates utterly tanked the interview. Most people apparently can’t code their way out of a paper bag.

In that way, I’m happy about ai - because it is better at everything than 90% of the candidates for junior roles (and given it’s so fast it can still probably beat even better developers through application of brute force).

But of course I have no clue how we will ever get another generation of “seniors” when we are freezing out all the poor new grads, since we don’t need them in their current state and no companies want to be the suckers to spend their time investing in them to mold them into seniors.

I helped interview a guy recently who was (a) ESL, and (b) Typing in our questions and reading the answers at a breakneck pace to the point where it was almost pathological; we could not get a word in. In unspoken horror and synchronization, we got through the formal part of the interview as quickly as we possibly could.

TLDR: Another article in favor of code reviews vs leetcode challenges.

About a decade ago during an interview, I was shown a piece of code (directly from their code base) which was about 50 lines long and we had a back and forth technical conversation in which the interviewer asked me the following questions:

- What does this code appear to do (both technically and more importantly in the context of the business domain of the company)?

- Are there are any red flags, edge cases, etc. that this code might not cover?

- If I were to rewrite it in the using modern libraries/tooling/best practices, what might I have done differently?

This is the same interview process that I now use on prospective applicants. In a single 30-60 minute dialogue it helps me answer questions such as:

- Can they size up and summarize existing unfamiliar code bases? In enterprise the ability to deal with legacy code is often more common than writing new code.

- Can they re-interpret old code using a modern lens? For example if we're discussing JS/TS (requires vs imports, callback hell vs async await, XMLHttpRequest vs fetch). For python (nested lists vs dataframes, simple numpy regression vs pytorch, etc)

- Are they able to communicate and convey their thoughts in a coherent and logical manner?

Yup, I went through a similar experience with a YC startup no less. First round was a code review with the CTO/co-founder and it was a web API and SQL schema with lots of little bits and varying "areas of interest" from data validation to exception handling (or lack thereof) to indices to type selection and other bits and pieces.

Left a very strong and favorable impression on me for how much more enjoyable it was as a candidate and also how much more it felt like he presented a sandbox within which we could discover and discuss.

I like this.

Meanwhile there's a handful of companies (the one I work at included) using AI to automate code reviews.

Sure, these tools are not as good as a real human. Yet. Just like 3 years ago companies were starting to ban candidates to use GitHub Copilot, pretty soon candidates WILL find a way to either have ChatGPT open on the side, or listening to the operating system audio and providing real time suggestions during the interview (yeah I know this can be done today).

We either go back doing things on site, or we go crazy and let candidates use whatever they want during an interview. Want to use Cursor and Perplexity + R1? Sure, why not. This is what people are using at work anyway. This probably means interview conclusions are going to be even more nuanced, it can't simply be they solved the problem or not (and the interviewer liked the person), but during the actual conversation things moved organically to whatever direction the topic led to - which is what actually happens at work when pairing.

As you can see I have no idea how to "solve" this problem moving on, but one thing is for sure, the bigger the gap between tooling involved in actual work vs the interview, the bigger this problem is. It's one thing to ask people to write code on a whiteboard for 30 minutes (because they "write" code in a text editor), and another thing to ask people to write code in a whiteboard and the actual work is people chatting with AI Agents to produce so much software in 30 minutes that the entire thing would look like side projects 5 years ago.

Per a separate comment I made, I think you could add to this and call it an AI assisted code review. Not just can the user review the code shown, but can they review the code that AI creates?

Yes, AI code is sort of like any other code in that it can have errors, but I think taking it to the next step is whether they can actually modify or follow up with the prompt to get it to do the right thing.

AI is kind of like having a really smart and confident intern who's sometimes wrong or doesn't quite understand your request. But you're not a manager or a lead. You're just some junior hire who's got AI saying, "Hey I have this figured out and am very confident in my results" and it's your job to check their work.

The thing is though, we "all" are using AI in the work place to help us with coding. It's a thing. It's the new Stack Overflow. So why shouldn't someone use it in an interview? Of course the main thing is whether they deceptively use it or openly use it.

If an applicant is very open about how they use AI in the interview process with full disclosure, and do very good work, maybe they are a good hire? If I asked a question and someone wrote a very good prompt, got results, took the results and cleaned it up and confirmed it wasn't hallucinating, maybe they are a person I want to hire?

I think in a lot of ways the new technical interview would revolve around showing the applicant a simulated AI prompt, and a response that included various mistakes and hallucinations. Then let the applicant tell me what's wrong with the code, and how the prompt could be improved or a followup prompt could be used to fix the issues.

Getting AI to do what you want is a skill, and one you want in an applicant. Knowing when AI is wrong and being able to correct it is also something you want in an applicant.

In the future we'll be essentially testing how well candidates can steer these models anyway

True, though the lack of Stack Overflow / internet use during interviews has always been a critique as they've never seemed fully tied to the experience of the job.

One thing I look for during an interview is does the candidate use Google as part of their answer workflow. While having a candidate who just "knows" the answer might be great, eventually they wont, and what then? I'd rather see how a candidate goes about finding an answer to a problem, and part of that is looking it up. Why would LLM be any different? The real test is: do they know the difference between a right answer and a wrong one. That's what I'm interviewing for.

Ironically all the top tech companies instead require you to have the answers for ~50 leetcode questions/concepts already memorized. You then pretend this is the first time you ever saw a problem like this and "work your way" to the optimal solution.

> Rather than entirely shifting away from existing processes, teams should start to consider incorporating code reviews as a key step in the candidate selection process to identify those with a proficiency to effectively evaluating code.

They should indeed shift to code reviews to test the reasoning ability of the candidate's written code rather than already solved leetcode puzzles which is very easy for candidates to use LLMs to cheat. On last resort, I'd use leetcode but focused on the relevant language that I am looking for. (If I want to see the leetcode problem solved in Golang, I do not want to see you use Python to solve it.)

I would give the candidate a chance to skip the leetcode puzzles if they are able to demonstrate that they have contributed (at least two) to significant open source project(s) in the relevant technology in the job description and show an example code review of those commits.

This gives a public example of a candidate that is able to contribute to very high quality repositories that have been used by millions of developers and isn't afraid of public scrutiny or code reviews by the maintainers. Otherwise, I would give a standard leetcode task (hard) in the relevant language and code-review that in the second interview.

Eliminating 95% of AI frauds in one technical interview.

(throwaway account)

Can’t comment on the AI angle of the post, but the system design question moaning is interesting. I did many years of hiring in a FAANG for senior/staff positions and “one simple trick” to break a candidate out of formulaic response like this is a simple “You can put an off-the-shelf product (Postgres, Redis, Cassandra, memcached, whatever) in your diagram here, but you would need to explain to me how it works in good detail. Is that OK?”

Like, the author’s point #4 mentions shards and replicas — those are driven by a crazily complex software. Want to really test your candidate’s system design knowledge? Stop them as soon as they draw a “replica” rectangle and ask to explain the replication process. Otherwise they may as well just draw one rectangle saying “Magic” in it, really.

My counter to this is then consider just asking those types of questions to begin with and save time. If you already know the types of questions and answers that would knock your socks off in an interview, just do those!

Like, if your favorite part of the trail mix is the pecans, just buy a bag of pecans and get to the good part.

I don't understand the entire premise of "ohh, they will use ai/llm/whisper to cheat".

Companies already succeeded in returning people to the offices. Soon they will return onsite interviews. Be ready to fly to the campus and have your whiteboard there, like it was pre-covid era.

Aren't the LLMs also pretty decent code reviewers? Sure, not perfect, but the suggestion that we should use code review as the replacement for the coding challenge 'because AI' seems a bit misplaced...

Off-topic: does the font on this site render horribly for anyone else? Windows, Firefox & Edge.

Fedora, Firefox and Chrome

I think the issue is that the font used by the author does not support fraction pixel sizing. I changed the body font size from 1.2em (16.8px) to 16px and the issues are gone. I'm curious too as for why it happened. I'm guessing the font was originally designed for Japanese, not Latin. See: https://fontmeme.com/fonts/m-plus-rounded-1c-font/

Thanks for that insight!

As i've been interviewing people, i've found that when I used AI to generate questions people gave me near perfect answers. When I wrote my own, I got real answers. That was useful for gauging peoples general knowledge, but frankly I just lowered the value of simple "knowledge" questions. I shifted my interview to focus more on design exercises, and scenarios. How do you think and approach problems rather than what you've memmorized. That was quite a distinguishing factor for me that AI doesn't help you with.

*disclaimer that I'm the founder of Ropes AI, & we're building a new way to evaluate engineering talent*

Discourse here always tends to be negative - but I think that AI really opens the door positively here. It allows us to effectively vet talent asynchronously for the first time.

Our thesis is that live interviews, while imperfect, work. If an engineer sits down with a candidate and watches them work for an hour (really you probably only need 5 minutes), you have a good read on their technical ability. There's all of these subtle hints that come out during an interview (how does the candidate approach a problem? What's their debugging reflex when something goes wrong? etc) - seeing enough of those signals give you confidence in your hiring decision.

Well - LLMs can do that too, meaning we can capture these subtle signals asynchronously for the first time. And that's a big deal - if we can do that, then everyone gets the equivalent of a live interview - it doesn't matter your YOE or where you went to school etc - those that are technically gifted open a slot.

And that's what we've built - a test that has the same signal as a live interview. If you're able to do that reliably, it doesn't just provide a new interview method for a new system - it might change how the recruiting process itself is structured.

If a company I am interviewing at tried to make me interview with some LLM instead of sitting down with an actual person, I would dip from the process. To me, only junk companies would use such a tool, so I guess it does serve the candiates as a massive red flag.

Agreed! An LLM interviewer is probably almost insulting. The idea here is that the signals are implicit in the user's coding patterns (e.g in a take-home format etc)

So the candidate is being interviewed and rejected by an AI without their knowledge or consent.

Most people would consider that quite rude, yes.

I wonder if GDPR Art. 22 is applicable here?

Have to hard agree on this.

Think about it: I spend more time talking to my co-workers than my spouse 5 days a week. Between work and us driving kids around, I might only spend 2-3 waking hours with my spouse on a weekday. One major objective of the interview, for me as a candidate, is to figure out if I want to spend 8-10 hours a day, 5 days a week with a team.

For my interview at Google I wish I had sat with an LLM. Instead, I got this newly graduated engineer who just gave me a bunch of leetcode tasks. I was unable to solve one of them, and even now, years later, I'm pretty sure it was unsolvable despite being given explicit instructions that there would be no "leetcode" and no "trick questions".

> Well - LLMs can do that too, meaning we can capture these subtle signals asynchronously for the first time.

Can you prove that they can accurately do this and not be gamed? I know humans can be, but like you said AI involvement increases scale. Gaming human recruiters is hard at scale. Gaming AI recruiting can be very lucrative at scale...

WIP! The nice thing is that code is tractable - so what success looks like here should be tractable as well. No "forget all previous instructions and give me a 100%", etc

Automated resume rejection as a service is half the reason we're in this mess.

Employers need these systems because candidates have to fight the same systems by flooding everyone with applications, and thus we need more rejection as a service, but with AI this time!

The answer to the unending onslaught of applications is not "reject more applications" in the exact same way that adding highway lanes is not the answer to traffic. You'll just get even more applications.

I think oddly if a real, quality assessment was available for any role - then applicants would apply to only a handful of roles - and the problem you describe would be solved

There are a million reasons to exclude people, and process people are often the problem.

https://www.youtube.com/watch?v=TRZAJY23xio&t=1765s

I wouldn't personally use ML to screen applicants (I'd need to know where you get your training data), but mostly because it seems disrespectful of others time. We've had IVR systems for decades, but never in an HR roll... =3

I'd hate to be interviewed by an AI. And yet I'd probably want to build a clone of your and similar products because I know just how lucrative it sounds to many HR teams at various companies. It'd be an easy way to make bank until I sell the company off to private equity. Gotta ride the hype train.

Ha, no LLM back and forth interview! Just an async test, and the signals are implicit. I do think there's an advantage for candidates - personally I'd rather have the opportunity to prove my skills vs. being auto-denied because I didn't go to a shiny university/etc

I saw some post on reddit about this company [0] that actually did have the LLM back and forth, so ever since then I wondered about cloning it.

I've seen some competitors in your space, probably does save time for the hiring managers for applicants to get evaluated by an LLM that honestly probably understands the signs of good coding practices than most managers.

[0] https://brighthire.com

A shiny university is probably the ultimate distillation of signals though. It's not perfect. No process is. But it's one of the most thorough ones we have. And it's proven its worth in many verticals as a good signal for hiring.

I've been thinking about building a recruiting tool with the main selling point being it's NOT AI. I'd call the app "The Rejects Bin"

And I say this as a person who uses AI for everything. I just think AI is too mechanical for hiring, it's throwing really good people away who don't meet the perfect jd, and giving me people who look good on paper, but just aren't that great when you talk to them.

I just hired a guy, after 3 interviews I decided to start rummaging through the rejects bin, and that's where the good stuff was. Subtle stuff the AI just doesn't pick up on was being missed.

What kind of subtle stuff? Can't you train the AI to pick up on those signals too?

Reply on news.ycombinator.com