Hacker News Reader

The real issue with any of these AI innovations is they really point out places where humans have already started to behave like an AI in the first place.

Ever since the emergence of leetcode style interviews I've been shocked at how many people can reproduce leetcode examples, but still fundamentally have no sense of algorithm design outside of the context of a job interview.

Programmers with their sights set on acing a FAANG interview will just keep repeating leetcode problems until they start to memorize the common patterns (not the problems themselves of course, but the structure of these type of problems). What's disturbing to me is that I recall far more interesting discussion about algorithms in the era before leetcode dominated everything.

The common solution isn't to understand algorithms better, but to become a leetcode solving robot.

So it's no surprise to me that AI can pretty easily replicate humans that have tried to turn themselves into robots.

We see similar patterns in the art that AI can create. It's very good at replicating a kind of art style of designers trying to turn themselves into design robots.

Absolutely spot on. I actually do algorithm design, usually over a period of weeks (at least), and leet code is a joke for the serious algorist (I’m sure I’d fail an interview based on it). Nothing has so clearly illustrated the robotic nature of the leet code expert quite like this result has.

Do tell!

What sort of job leads you do design algorithms?

Mathematical Biology/Bioinformatics. We have to think very carefully about every step in the process of extracting information from large, diverse datasets--often writing things from scratch, combining/transforming things in novel ways, and implementing new mathematical ideas efficiently enough to be computable on large datasets.

I see several argue that if you are poor at leetcode then you cannot be a serious algorist. Can you explain why that argument is wrong?

  > So it's no surprise to me that AI can pretty easily replicate humans that have tried to turn themselves into robots.

this is such a great insight... i feel like it could even somehow explain a lot of politics and many other phenomena.

> c) Embrace it

asking "here's a chatgpt solution to the problem... what's wrong with it?" would be a solid process imo.

This would be way more fun

Well, if they still have access to ChatGPT, they can ask that of it as well. It may or may not give a valid answer, just as it does with code.

A big if. A lot of ChatGPT discussions seem to take for granted that it'll always be available/free/priced low enough that ~everyone has access to it. Seems more likely that at some point OpenAI will close it up and put it back behind an API.

You can use text-davinci-003 from the API now and it works better for many things.

The company itself could have access to the AI for the interview, since idea is lowering the playing field.

But even more, it seems likely someone is going to be willing to supply LLM access at not that much more than the cost of computation.

I know OpenAI's business is selling access to their stuff at a premium but since current AI is not much more than brute-forcing of massive public data, it doesn't seem like this premium will be justifiable for long.

And you can give every candidate a new version of it.

I love it!

Particularly, ChatGPT can give apparently correct solutions that are wrong in subtle ways.

Also reading, understanding, reasoning about, and fixing code other's wrote is way closer to on-the-job performance.

Yep, the future of SWE is debugging AI.

This is already the present for translators. They often are called to proofread machine translated text. And this is often more difficult than rewriting the whole text.

I would expect A, because it is an option.

Like in mathematics or other tests students are not allowed to use the internet or an advanced calculator in order to test wether they truly comprehend the stuff.

If you think that your FAANG-interviews are any good, then just keep them in the format you already have, by making sure applicants cannot use AI during the test. I would have on-sites with pen+paper, whiteboard or a prepped/supervised machine, whatever.

Of course, applicants could use AI to train for the interview, but that is not a problem, as long as you test their comprehension.

Would training with such an AI help their comprehension? Or hurt it?

My money is on "hurt".

Given the trends towards remote tech jobs, it could be difficult to convince your companies engineers to show up to the office for what’s usually just a weed out leet code interview.

Proctored tests are common for certification exams. Perhaps there's a future where candidates go to one of these exam centers for one of their technical interviews.

Or just a live (remote) programming interview, with no use of Codex or Copilot.

I'm not a fan of making the process more dreadful than it already is. Forcing people to go to an office and use pen and paper or a whiteboard would do exactly that.

I hope companies (including mine!) see the writing on the wall and stop trying to fight the future.

This isn't accounting for even just pretending we are testing candidates on anything remotely indicative of on-the-job performance.

Algorithm interview and competitive programming type questions are probably some of the easiest for GPT to solve because there are a massive number of problems and solutions publicly available for training.

The real benefit of AI is somewhat shown in this paper, it effectively solved the problems through brute force generating millions of possible solutions. For real world problems it would be interesting to let GPT generate a bunch of different solutions and push them into a test environment and see which works best.

The biggest problem I see is black swan events where AI coded systems work great until something goes wrong and no human truly knows how all the pieces fit together.

I get the same results from asking for code. Near 100% success with an initial demo, and then pretty tragic performance when making a series of changes to actually fit the concept. I got it to write 3d rotation code, and to switch to using rotors instead of quaternions but when I asked it to rewrite the combine_rotors() method it rewrote it to combine Enigma rotors. I explained its error and it apologized but just wasn't able to go back to the original code and work with it anymore.

Interviewing is going to be shaken up but I think some methods are more timeless. I've been giving the subject code instead of asking them to write it for a while now. Largely because I wanted to talk more and watch them type less.

We start by discussing the coding challenge guidelines and I leave them vague. They need to understand the goal and what I've left out and suggest those guidelines themselves. Once we agree though, I give them the code. "Here's what's running now."

Then I update the goal and we discuss changes. I get them to "whiteboard" certain things, like what a query looks like with their proposed changes or whatever, and we discuss big-O, etc. This, imho, is how whiteboarding is actually used - not to write whole programs but to provide examples and pick them apart.

I feel that this would work even if they were using an AI in another window. We're trying to select for developers with common sense and domain knowledge, and who can clearly discuss engineering tradeoffs. Actually making the changes (the coding itself) was a big part of the job and that's decreasing, but imho all the other requirements remain. They'll still need to know how to handle the issues I talked about above, of trying to get the model to write the right code!

Our coding exercise cannot be solved by ChatGPT and I believe is much more effective at evaluating coding ability than leetcode-style questions. We ask candidates to design an object-oriented booking system which requires 3 or 4 different classes to implement. ChatGPT cannot easily do this without heavy prompting from the user at which point, they'd be better off just writing their design down themselves. We want to evaluate candidates based on the kind of work they will actually be doing - not brainteasing O(n) solutions to contrived algorithm questions and so far it's worked very well.

> cannot be solved by ChatGPT

That's great! For now. But tomorrow's coming fast.

I agree. Our interview has 3 coding rounds, and one of them I call "clean code". For that I ask a straightforward question that requires candidate to define some APIs and write some classes / functions. Then I ask several follow ups, adding or changing requirements.

This is by far my favourite question, the one closer to on-the-job coding. It also lends itself well to deep conversations with candidates.

But alas I don't own the process and still have to work within the parameters of the company. Whenever possible I ask this kind of questions, but other interviewers will default to leetcode-style rounds.

That seems like it's likely been trained on various examples of FAANG questions that are posted to the likes of leetcode, with solutions often presented. The push for harder version was clever, and that it fell over is no real surprise.

> c) Embrace it. [...] My money is on c)

Agreed.

I think developers who don't will be rare in a few years time. Just like developers who primarily rely on assemblers (versus compilers) have basically become extinct. Like we sometimes, very rarely, need to inline some assembly, we will sometimes need to use the ol' gray matter to figure out a novel algorithm or something.

I believe that avoiding learning these tools could be a existential issue for your present-day job. You don't have to come to depend on them, or use them daily, but you do need to understand how best to use (and not use) them.

This is why I ask questions in a business domain. It requires the programmer to think not only about solving a straightforward clear problem and only worry about Big-O. Instead they need to figure out the problem by asking thoughtful questions about possible business concerns and think about which ones to optimize for.

Ambiguity that requires follow up questions for successful isn't going to be addressed by something focused on solving a problem that "thinks" it has all the information to solve the problem.

> I've conducted hundreds of FAANG-level coding interviews, [...] I always loathed this interview format,*

That's impressive perseverance. Did that wear on you?

Myself, I've only conducted dozens not hundreds of FAANG-style coding interviews. I also loathe the format, and it does indeed wear on me. I hate every second.

Damn. What is your success rate? A lot of times it seems to mee that luck and your ability to withstand stress are one of the main factors. Is it something you can teach yourself?

If by success rate you mean inclined rate, it varies a lot by level. I'd say it's around 30% for interns (which is by far more lenient) and closer to 5% for Senior level. However it's also common to downlevel senior candidates rather than downright reject.

From my side of the table I can confirm that of course luck plays a role. Nevermind the variability of the coding problems themselves, but the variability on the interviewers. Even in my company, that has a formal process and pretense of objectivity, interviewers quality and expectations vary greatly.

In short, I'd say what people on the hiring side hate to hear, but interviewing is ultimately a numbers game. I wouldn't take rejections too seriously. The best you can do is to improve your chances with proper prep (leetcode, system design) and interview broadly.

The illusion of objectivity is strong. We have an official interviewing process. It has a big survey that turns into "metrics". We have a debrief where we talk about the "metrics" as though it was an "evidence based" process. Then you'll hear someone say "Well, he found a solution for the balanced tree problem, but he just didn't seem all that confident doing it, you know?"

> Then you'll hear someone say "Well, he found a solution for the balanced tree problem, but he just didn't seem all that confident doing it, you know?"

This sounds like yet more evidence that it's really just pledging a frat.

I hate leetcode style interviews, but I've done hundreds of them. The irony is not lost on me. Some might say I'm part of the problem.

Thing is, in a small way, I'm trying to change things from within. I try to make the process as palatable and fair to candidates as possible, while working within the constraints of the system. When I train new interviewers some of my top tips are:

a) The purpose of the interview is to determine whether the candidate is a good fit for the company, and the company is a good fit for the candidate. b) The interview is an imperfect proxy for this.

It then follows that interviews shouldn't overindex in the coding round. If I have helped someone to get an offer that wouldn't otherwise, I'm satisfied.

BTW that's also one of the reasons I'm publishing this. I hope to push the point that leetcode-style coding interviews are outdated and should be burnt to the ground.

i think you're correct about option c, primarily because we could never force everyone to adopt the other 2 options.

option c makes me nervous, but right now I can't see an ai correctly dealing with the ambiguity of the data sets i typically look at.i do a lot of "asking for clarification."

We're just going to move to on-site interviews only. ChatGPT can't enter the equation then.

Neuralink blocks your path.

Neuralink is not realistic in its current conception, they won't easily solve the problem of input to a brain

Neuralink doesn't work.

>ChatGPT can't enter the equation then.

In a few years, ChatGPT will be conducting these interviews.

LMAO, no. What would prevent someone from just looking up the answers using ChatGPT on another machine?

I expect that the reason it gave the correct answer to your first question is simply that it already saw the problem and memorized the solution - there is some empirical evidence that deep neural networks are able to memorize much of their training data.

Possibly. The problem itself is not in leetcode, but it almost certainly has been leaked somewhere. However the program was able to make a few changes to the code with some prompting, which hints a little bit more smarts than just regurgitating an answer.

Nevertheless the point is moot. I've invented completely novel questions (promise!), and saw them leaked online after asking them twice. The process is fundamentally flawed and large language models are just making that glaringly obvious.

Your option d) that you don’t even enumerate (scrap leetcode) would be best.

> Interview cheating has always been there, but it is generally so rare that it isn't a real concern.

You're the person they're trying to fool, so if they cheat successfully you would never know. You've never seen overt displays of bad cheating, which is different.

Think you haven't seen a stick insect in years? Likely you have, but just didn't notice it...

Cheaters will always be more motivated than those trying to detect them - because everything is on the line for them.

>You're the person they're trying to fool, so if they cheat successfully you would never know. You've never seen overt displays of bad cheating, which is different.

I don't think it's widespread at least, since in my experience people that do well in technical zoom interviews do not drastically decrease in apparent competence when we move to in person rounds.

People definitely get told interview questions by recruiters though, if you count that as cheating then it is everywhere.

By the way ChatGPT is not even the best model OpenAI has for writing code.

By the time someone gets an on-premises interview, you'd notice if they're using a bot or not? Or are all the interview rounds remote these days?

In my experience (with FAANGs at least) the interviews have been virtual since the pandemic began. Haven't heard of any plans to return to return to flying out candidates for in-person interviews.

We've been doing remote-only interviews for years now.

If a company allows ChatGPT to be used on the job, anything but c) seems foolish. If the company doesn't allow it, c) seems like a bad idea.