Hacker News Reader

Comment by cogman10 | Hacker News Reader

I'm starting to see this come up in coding interviews.

I had one candidate that struggled MIGHTILY on a coding problem that was easy (kept running into syntax errors) but then pulled out an esoteric solution for the second problem in no time flat.

I can't prove that he was cheating but it very highly suggestive that he is.

I hate it, but I'm starting to think I have to do an AI detection question now. It's not hard, just ask someone to do something impossible. However, I don't like the fact that now I need to be "tricky". I've never believed in making coding challenges hard, I just want to see if you can write code.

View on HN

I saw something similar in a recent interview. I brought in a candidate who did not meet all of the typical resume check boxes, but who had a long history of success on highly technical problems. These type of candidates can bring very new perspectives to our team, but they’re often a long shot in interviews.

In this case the candidate chose to use a language that I don’t know well and they said they weren’t especially familiar with either. Then they struggled a lot to even talk through the question and it was a grind to work through each line of code. I couldn’t offer much help, and they were very clearly unsure at each step.

But when we clicked “run” on the code, we were both surprised to see that it ran on the first try. Not only that, but it already worked for all of the “but what about…” follow up edge cases. And the candidate didn’t know why… “oh, I guess the language just works that way.”

I don’t know if they were cheating, but the entire thing was far outside any other interview experience I’ve ever had — and fits the pattern of what I’d expect an LLM to produce.

this is me using ocaml

Seems like there's a silver lining here. It reinforces the idea that the answer to the problem is not necessarily the important part. When you can trivially provide a correct answer, it places more emphasis on the skill of being able to actually explain and work through the logic as a way to distinguish yourself from other candidates.

You didn't hire them, right?

This may be an unpopular point of view, but I think in situations like this, you should just pass on them. They are applying to tens of jobs and your company is interviewing tens of candidates, so there's no real reason for either of you to continue this process if they struggled on the easy problem and had questionable reasoning on the hard problem. However, if the candidate could very clearly explain the esoteric solution, there is a good chance that it's just similar to something she has done before and you should not be suspicious.

I am assuming that you had a compiler or IDE in this situation providing a useful error message. If you were whiteboard coding and you were telling the candidate "you have a syntax error but I won't tell you where" (like several folks at a former employer of mine were fond of), you may be the problem, not the candidate.

We ultimately did pass.

> I am assuming that you had a compiler or IDE in this situation providing a useful error message.

Yup. I prefer to give candidates compilers/IDEs and even access to google if they ask about it. Like I said, I'm not trying to trick them or anything, I just want to know if they can code.

> If you were whiteboard coding and you were telling the candidate "you have a syntax error but I won't tell you where"

I'd hate being on the receiving end of that and would never pull that sort of stunt. The software we use allows both sides to edit the code and I pretty regularly will go in and silently fix syntax errors for a candidate so as not to let a compile/refresh loop get in the way of actually solving the problem.

Like, what does "You have a syntax error" even prove? I'd let you write pure pseudo code so long as it makes sense.

I just don't like "mind games" in interviews.

I still make syntax errors all the time.

I've been writing code for pay for more than 20 years and will forget basic syntax and keywords for a language I wrote hundreds of lines in yesterday without other code to crib off of, IDE assistance, and/or reference material (google, whatever). Like, I'll forget if it's "else if" or "elsif" or "elseif" or "elif" or whatever. Or I'll mix up how to access hash/list members. How to designate a constructor. Stuff like that.

I'm a pretty good practical programmer, good at noticing potential edge cases, writing code that's changeable without being overcomplicated. This based on feedback across years of professional work.

I also have adhd and literally brain damage and have worked professionally in about a dozen different programming languages. I can't write a for loop or declare a static method or w/e in any of them without googling the syntax or using my editor's hints & autocomplete.

That's fine and all, but not what was happening.

They had google access and editor hints. Further, I start every interview with "please, ask any questions, we aren't trying to trick you we just want to see if you can code." And I mean it. If someone has a minor problem with remembering syntax or how to declare it I don't really care and will happily let them know how to do that.

Heck, I've even had interactions where a candidate was like "I think there's a method that does x for this" in a language I was unfamiliar with, so I googled up what x was and shared it with them mid interview. "Oh yeah, looks like this is what does x for your language".

There was also some pretty weird behaviors with the camera/screen that caused them to need to touch it fairly frequently (I'm guessing to take over keyboard control from the AI software they were using).

What they were doing, repeatedly, though the interview was writing

`a[i]` then having to change it to `a.get(i)` because they were working with a `List` in java. I get maybe doing that once or twice, but in the course of the interview they did it every single time they needed to pull something out of the list. (and each time needed to touch their screen and take a few minutes to correct it). This is why I strongly suspected cheating. It extended a question that normally takes 15 minutes to 35 minutes. Then the question I ask that usually takes ~45 minutes they completed in 15 with an optimal solution that relies on a data structure I literally only learned about because the interview hints for the question are like "Hey, java has this data structure that makes everything easier".

Yeah I get it. This is one of those things that's really hard to convincingly describe online because each individual thing is explainable eg I will write a[i] every time and my format-on-save will change it to List.get(a, i) every newline or whatever.

But you were there and I wasn't and if the overall situation seemed suspicious to you that's evidence at least as strong as any specific thing. Intuition can certainly pick up on weird behavior even if it's hard to explain how after the fact.

Reply on news.ycombinator.com