michael_mroczka

It's a good question. In short, yes, I believe it is reasonable to assume custom questions will generally perform better than verbatim or modified leetcode questions. In general, while ChatGPT could handle modified questions well enough for interviewees to pass their interviews, it still choked through them and required more coaxing to get an answer than verbatim questions.

Custom questions, by definition, aren't available online, and with no direct tutorials to pull from, the LLM has to make more inferences about the problem and will find the question more challenging.

As for asking ChatGPT novel DS&A questions, I think it is harder than maybe you'd think it is. Any question you'd think to ask likely has a tutorial for it online somewhere, so unless you happen to make up questions like this professionally (I'm paid to do this), or you have a large unique question bank that doesn't exist online (few people have this) then my instinct would be that the questions you're giving it aren't as unique as you think they are.

As a practical example, just give it a log file and tell it to pull out specific pieces of information from it. ChatGPT struggles to write code that can dynamically check obvious boundaries for humans. Recently, I had a list of times in a CSV that I asked it to pull for me ("1pm", "3pm", "9am"), and it wrote code to just grab 3 specific indices from the string. It didn't consider the need to check for 4 indices ("10pm"). It didn't think to start the check based on where commas were in the CSV, and it didn't consider looking for "am" or "pm". It just sliced a specific set of indices in the string. That's mostly because it's used to getting questions working for specific examples, but fails when you ask it to incorporate simple broader tasks into the coding interview question.

Exactly. People end up throwing the baby out with the bathwater on this. "Data structure & algorithm interviews aren't perfect, so let's not ask people to code at all." It's an absurd overcorrection, but most people think these interviews are about demanding optimal code and perfection when they mostly just are making sure you're not using arrays when you should be using hashmaps... and that you know what a hashmap is, I suppose.

The short answer is, "Yes." They are very flawed, but one of the most reliable ways to avoid "bad" hires.

The longer answer is: Fundamentally, you need to address the fact that there exist a huge number of people in this industry declaring they have masters degrees/phds or years of industry experience, but when pressed they can't write even the simplest of functions.

While we called it out explicitly, some folks seem to miss that "Custom" questions are still fundamentally DS&A leetcode-style questions. I completely agree that "leetcode style" interviews are flawed, but most people don't have a better answer for this problem that still guarantees the person you're hiring actually can code.

We are optimizing for coders that make good choices quickly, and if they can code efficient code to toy CS problems, then you at least guarantee that 1) they can actually code and 2) they can code simple things quickly. Non-coding interviews allow you to hire people who can't do these basic things and therefore guarantee their performance is worse in a production environment.

I'm not sure if you read the post, but with some of the new cheating tools that exist, they overlay the GPT responses in front of your screen with concise bullet points. You wouldn't even need to look away from your screen or interviewer to cheat. The bullet points are also small enough to where it is incredibly difficult to tell that someone is reading anything - even if they have a webcam enabled and are looking right at you. This, coupled with some interviewers that don't care a lot about the process, it is getting easier for cheaters to slip into places for sure!

I AM the author, and I also am in favor of "open book" interviews. I'm not against ChatGPT use in interviews, but if you're doing it secretly in an interview that clearly is meant to be "closed book," I think it's fair to say you're cheating.

I'm the author of this post. Happy to answer questions if you have any. This was such a fascinating experiment!