Testing how hard it is to cheat with ChatGPT in interviews
interviewing.io
interviewing.io
that was interesting, upvoting that employer for honesty and pragmatism
Long story short: asking them to make small changes and then tell us what would happen was a shurefire way to detect the true cheaters and not the lazy people.
I also fondly remember triggering float errors in loops so you'd get an extra cycle due to it ending in .999etc instead of 0.
There is no bigger warning sign than outright lying. A normal mature person would ask just before if AI is allowed.
Honestly, the realistic style of work that's close to how one would actually approach problems in their day to day is pretty much ideal. In my case that would be using a nice IDE, some AI as a glorified autocomplete, IntelliSense and all that as well, in addition to Googling stuff along the way, if needed.
That should be enough to let them know both how I think, as well as show how I can solve problems and reason about those solutions. Heck, maybe even give me a simple task to build a CRUD and then talk about the choices I've made, if they're serious about hiring me and want to actually see what's inside of my brain.
But of course, in many places can't have that happen - they want to put the candidates in a situation where they just have a barebones text editor and expect them to produce good results. Blergh.
I was surprised to see how many people would prefer our laptop during interview instead of having their own favorite environment.
So I pull out trusty old Ruby + RubyMine, and the question (I don't remember what it was specifically, but it was some sort of array manipulation type of thing) was trivially solvable with some Ruby STD lib method. Apparently he wasn't satisfied with the answer despite his prior instructions, and me not knowing what the actual method code is was reason enough to disqualify me from the job, which I found baffling.
I got into a bit of a heated discussion that, if I were solving a problem at work, there's a 99% chance I'd just reach for this method that Ruby itself comes with, because why the fuck would I try be a smartass and reinvent stuff here? And if for whatever insane reason I did indeed need a bespoke solution, I'd just poke around the source code and extrapolate from there, which he still didn't consider a good answer.
These interviewers drive me insane sometimes...
I laughed that I had given them such an easy problem for their chosen language, then said "ok, pretend you went back in time and you're the person writing that function for the std lib" and so they did.
It probably wasn't as optimized as the actual stdlib function but it was good enough for the interview.
I gotta say if someone got heated with me about that request I'd end the interview early and not give them the job too.
The point isn't just providing a solution it's demonstrating that you can work through a simple problem. If you get heated about that during an interview then I don't get any info about your ability to work through problems, and I do get the impression that you're short tempered when asked to do things you don't care about.
Radical honesty has been a core cultural component to many a strong team, I'm glad to see somebody else mention this. There seems to be something unique about the relationship between codering and the concepts of transparency, honesty, and truth more broadly.
Or maybe that's just a consequence of version control :)
Chernobyl being one prominent example.
At least in a field like engineering where actual successful results/working output matters, anyway.
There are other fields where the same dynamics are not in play.
One cannot solve (or even avoid) a problem that one refuses to acknowledge exists, after all.
Or as I usually phrase it, 'money is allergic to lies'.
Say you have an organization that is producing a product/service that provides genuine value for its users, and have a team of talented, hardworking people. Any factors related to the operations of said organization, obscuring those factors from the value producers can only lead to less effective operation overall, as the producers have less/lower quality/false information to work with.
"I don't feel like this is workplace appropriate" does not violate the 'radical honesty' principle.
At least internally, anyway. If your objective is to make as much money as possible, you probably don't want marketing to be radically honest LOL
And what is worse than lies, is self delusion, even if honest. To nit pick on radical honesty, my observation is that most people won't tolerate it, plain honesty appears to be the sweet stop inmost cases.
Allow them to use the tools, with a screenshare, and adjust the types of tasks you are giving them so that they won't be able to just feed the question to the LLM to give them the completed answer.
Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another.
I think it depends on whether the interviewer has agreed to make the interview "open book". Looking up stuff on Stack Overflow during the interview can be OK or can be cheating, depending on the constraints.
In this experiment, the interviews were not "open book". That said, I am personally in favor of open book interviews.
The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge.
Now... The skill of asking the right Qs also directly intersects with LLMs, and how to discern good/bad responses.
But hiding it? Yeah, probably not a good fit.
I work at a university and most of our exams are open book or project based. You probably want to update your image of universities.
I have a close friend who is a prof at a university and most exams remain closed book.
You probably want to update your image of universities.
Or, perhaps, we can agree that it depends on the university, the subject, etc. and blanket statements based on single anecdotes are silly?
I am not sure that is clear. It seems the expectation was not "closed book", but "never opened a book before, not even in the past":
"It's tough to determine if the candidate breezed through the question because they're actually good or if they've heard this question before."
Clearly the interviewers were looking not for knowledge, but for uncanny ability. How well was that communicated to the interviewees?
It is not cheating if the rules of the game are not defined.
I would argue that is the opposite: it's fair to say that the interview is a cheat.
I recall reading an interviewing.io blog post[0] in which the dominant considerations interviewers weighed were (my interpretation):
(1) Did they solve the problem optimally? (2) How fluid was their coding?
With "communication" turning out to be basically worthless for predicting hire/no-hire decisions.
Perception of coding fluidity seems like it would be affected by how often the candidate stops and looks up things like library functions or obscure syntax.
For that reason I've been investing time in committing a lot of library functions to memory, so they instantly flow from my fingers rather than spending a minute looking it up.
It's dumb that I need to do this, but I don't make the rules. I'm just at the bottom of the information cascade that led to how things are done now.
[0] https://interviewing.io/blog/does-communication-matter-in-te...
I've got an abstraction that I think in that then needs to be translated to code. e.g. if I want to append to a list, I think "push to list", regardless of language/framework/whatever. Then somehow my hands will translate that automatically to code in the language I'm working in. If I'm not in my usual editing environment, that magic just sort of breaks, and I just look incompetent.
It's not a huge deal, but I have to actively sit and memorize that stuff before an interview. Usually by writing it out on paper or writing it in some foreign editor that I'm not used too.
It probably wouldn't be _too_ bad today, because I just sit and write Typescript everyday, but when I was switching between perl/ruby/python/tcl/kotlin/javascript/bash/csh/lisp my brain was basically mush. I couldn't tell you how to do any basic operation in any language.
Thank you for putting all of that into words that I can look at and go: that's exactly how my brain works!
I don't think everyone has that issue or can understand it. I think in more abstract terms (the problem and how to solve it) and more often than not the syntax is just an implementation detail that I prefer to increasingly outsource to my tooling.
But I do think most of what I use has grown into my muscle memory naturally, rather than memorizing anything.
The longer answer is: Fundamentally, you need to address the fact that there exist a huge number of people in this industry declaring they have masters degrees/phds or years of industry experience, but when pressed they can't write even the simplest of functions.
While we called it out explicitly, some folks seem to miss that "Custom" questions are still fundamentally DS&A leetcode-style questions. I completely agree that "leetcode style" interviews are flawed, but most people don't have a better answer for this problem that still guarantees the person you're hiring actually can code.
We are optimizing for coders that make good choices quickly, and if they can code efficient code to toy CS problems, then you at least guarantee that 1) they can actually code and 2) they can code simple things quickly. Non-coding interviews allow you to hire people who can't do these basic things and therefore guarantee their performance is worse in a production environment.
I'm not advocating for spot tests on complex algorithms without context. However, I believe a conversation about fundamental concepts like big-O notation reflects on one's approach to problem-solving and software design. It's not about dismissing anyone's experience or capability but ensuring a solid foundation that benefits all aspects of engineering work.
I understand senior engineers being out of practice and not being able to derive things like topological sort on the fly, but an inability to talk about the basics of big-O or the simplest of data structures is more commonly a red flag than "rust." If they never learned the material, that's a red flag. If they learned the material and they've been building software for the last decade without taking the basics of the material into account while coding, then that is a red flag as well.
Again, happy to admit that this is a flawed approach that will lose some great engineers, but most engineers that have your line of thinking are ones I actively avoid hiring. These concepts are practical, available to learn for free, and easy to understand. If an engineer feels that testing these concepts is beneath them, then they definitely aren't a good fit for any team I'm on. Big-O is to software engineers what Ohm's Law is to electricians. Imagine a world where electricians thought it was demeaning to talk about Ohm's law in an interview. ¯\_(ツ)_/¯
Really? Most of the work that exists is CRUD, and I've seen people who grinded leetcode algorithms, do 1+N SQL queries which is way, way worse, than forgetting what kind of time complexity a sorting algorithm has that you never implement yourself anyway.
At the same time these leetcode grinders will over engineer a UI solution to make sure they use the correct data structures to avoid looping 300 times in the browser at the cost of readability writing 30 lines of code instead of 4.
I don't remember a single time where I've used Big O in practice.
I ask myself - what will happen with performance if there's N amount of traffic or data, but I don't think of it in terms of Big O. I consider perhaps a time it takes or load it takes at upper limit numerically.
N might in some cases be terrible, and N times N performance in other cases totally fine.
I think actually thinking in terms of Big O might make you worse engineer as you start to obsess over the wrong things.
I prefer someone who's spending all that time on building things instead. This time of leetcode grinding and memorising data structures, algorithms could be used on coding side projects.
You learn way more and more practical things when you build something.
The whole algorithm and leetcode thing just seems so off and unpractical to me.
Then once they have done the exercise there are so many practical things we can discuss about the existing code. Peformance, quality, etc, etc.
Leetcode questions can be fairly basic algorithmic/data structures questions, like "how do you store an ordered collection with O(1) lookup (ordered dict, i.e. hashtable + list)" or "check if this list contains duplicates" (use a set, not a list!), or "calculate the square root of this number" (bisecion, i.e. looping).
These are not production code questions, but the benefit is that they're easy to explain and understand in the limited time you have available for an interview.
Unfortunately there are more than expected number of people who fail these basic questions. You really don't want to hire them.
However, my style of interview is LLM-proof anyways. I have "shop talk" style interviews, where I just chat with the developer for an hour or so about various topics. Makes it very easy to get a sense of their depth, and how interested they are in the job domain.
I am having an interview next week and I hope it will be going like that. They emailed me yesterday with a small coding task. I was supposed to setup a simple server updated website with blazor (horrible name) using (a) Background Worker for the Server side "computation" that updates the page with a random number every X seconds.
Even not having worked with asp.net directly ever it was fairly easy to setup and implement. The complexities are rather well hidden by dotnet. But the use of the BackgroundWorker class seemed weird to me. In fact I implemented it first with a simple timer instead before noticing the ambiguity in the task description. So I implemented it both ways. I think I spend less than an hour on it and thats nice cause it respects my time :)
It was a time sink for sure, but I can't stand coding in front of people because I usually like to sit and reflect. And I had a good amount of time to prepare my thoughts on what I'd built. No hidden surprises, no anxiety. I loved it.
Salesforce had a good interview practice a few years back where they invited you to a meeting. Started a recording, then asked you to keep your microphone and camera on and do several simple programming tasks. It was an "open book," and you could use whatever you wanted, but you just had to show how you got to where you were (and you could only use one monitor so that it was clear what you were looking at at all times). The engineer who met you on the call left after just a couple of minutes, and you could work in peace without having to worry about "entertaining them."
For what it's worth, I think LLMs are very compatible with the style of interview you describe here. For people for whom LLMs have become a part of their workflow, seeing how they interact with them is just one more way to get a sense for how they work and think about things. If it doesn't come up, that's find too! But I don't see any reason not to accept their usage in your interviews.
(But I do think it's key to ask them to screenshare if they're going to go that route, so that you can actually see their interaction to get that signal.)
What I meant was that if you're going to give a live-coding interview, I personally wouldn't accept LLM usage. The reason is that with that style of interview, you're essentially validating one of two things - how much knowledge they have on hand, or how they're able to think in the abstract about problems and reliably solve them. I'm more interested in the latter, so I don't care if they use an external reference for trivia. But LLMs simulate the abstract thinking for you, which means I can't evaluate a candidate's ability to reason their way through a problem.
It actually is very important that you're able to think through problems on your own, sans-llm.
This isn't my experience at all. Or rather, it isn't my experience that people are able to use them to do this, effectively. So if a candidate tries to do that, that's signal.
The point of an interview is to - very imperfectly! - get a sense for how people think and work. If a candidate defers all their thinking to an LLM, that's something I'd like to know about them!
But I've never seen anyone do this in an LLM-allowed interview. Instead, they use it as a more effective version of how people have long used web search and IDE autocomplete, if you allow those. It speeds up the boring parts of interviews where people fumble over silly syntax issues or "what is python's method for xyz thing called" that don't tell me anything about how they think and work in a real setting.
Consider that this may not be typical.
And I can still use ChatGPT and similar tools for some of what I do. It is a huge force multiplier.
Source: https://survey.stackoverflow.co/2023/#ai-sentiment-and-usage
To be fair, the number of "Yes" was "just" 43% but that's still a very large amount of developers, not including those who plan to use it.
This narrative of "interviews should be like real work" needs to stop. It doesn't make any sense, this is not the goal of an interview.
While the above sounds sarcastic, honestly, it isn't as ridiculous of a thought as it sounds. I don't think being good at DS&A problems guarantees that you're a good coder, but in general, the people who get good at these problems are also great coders. I can think of less than a handful of people who are good at CS problems but bad at actual coding.
Since IT is paid very well, there are lots of people who try to lie their way to get the job. If you are someone on minimum wage, then why not try your luck - if they fire you after 2 months, you probably earned more in those 2 months than in whole year of your shit job.
The point of (software engineering) interviews is to demonstrate how you solve problems. "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for.
They are probably not looking for that, because LLMs perform poorly with some kind of problems, and you don't want people who rely on them heavily.
This gives you a plan for designing the interview questions.
I'm definitely looking at googling proficiency when interviewing.
Being good at LLMs is about as important - this means being able to tell when you're being confabulated at quickly, knowing what to ask and what not to ask, etc.
Then interviewers should stop setting tasks that require either a) copy and paste answer from leetcode or b) copy and paste answer from chatgpt.
Unfortunately that requires skill and awareness on behalf of the interviewers; who typically served in the leetcode wars and want their employees to go through the same.
In my experience it's mostly been laziness. Grabbing leetcode problems is easy compared to preparing a custom scenario that tests the candidates higher-order skills. God forbid they spend up to a day preparing something that could be reused for the next several years.
After about three interviews it ended up on Leetcode.
The interesting part is what to do when ChatGPT does it wrong, and if the problem is not trivial and an exact solution is not available online, it is usually wrong. Sometimes, not by much, but it takes skill to notice the problem and fix it, either manually or by asking ChatGPT for it.
Same idea as for libraries. One could argue that those using the "sort" function don't know how to write a sorting algorithm, but in real life, unless there are particularly good reasons, rewriting a "sort" function would be crazy, and probably not what you want from your employees.
If you want your candidate not to use the tools at his disposal, you may frame it as a requirement. "on this test, imagine you are working on a sensitive project, you are not allowed to upload any details to a third party service, you can search the internet for generic information, but do as if your development machine has no internet access, so no copy-pasting". Or, for libraries "on this test, imagine we absolutely want to limit dependencies on third party libraries, even if it means sometimes reinventing the wheel, so only <list of libraries> are allowed".
Whether or not it is a security concern depends on what you are working on I guess. So that's what I meant by "I don't know". It may absolutely be, or it may not be considered so, but the idea is that if you are able to do the indirect process, that is not blindly copy-pasting, you are already showing some level of skill.
Unfortunately though, there are lots and lots of interviewers out there that do not give a shit about "how", but rather if you can provide the right answer or not.
Here are three questions, you have 60 minutes. Please provide the optimal solution to at least two of those questions. Now excuse me while I'm listening to a teams/zoom meeting in the background. Good luck, have fun.
I don't, and I don't know anyone working with me thar is using any LLM (we pair). Some tried Copilot at some point and concluded it was useless for our use case.
Not sure on what context one would constantly use LLMs.
I'm not hiring at the level of day to day work, I'm hiring at the level of when things go complex or bad.
To give an example, I'm not going to test a coders ability to write some CRUD application. I'm testing the ability when a junior developers comes with a crazy problem, that they can find the cause and solve it.
If I only test day to day work, you get these kind of developers that keep changing faulty code until it magically works. I don't want those in my team. I want developers that can figure out what is going wrong, understand it, and provide a solution. Do such things happen day to day? No.
And a person that can reason his way out of things instead of trying random stuff, sure as hell can do the same day to day tasks just as well or even better.
While, wait, thinking...
So all this "cheating" is maybe a (bit delayed) response to the above trend..
The only classes I can recall that required higher-order skills were some woodworking, CAD, and CNC/machining classes I took while at an off-site vocational school alongside my required high school classes.
Down to the metal fabrication / woodworking being the most critical thinking I did in K-12 other than programming and reading on my own time
I've had some amazing teachers whose passion shone through and cultivated students' genuine interest in the subject matter. Unfortunately, there is only so much that can be done when, at the end of the day, your curriculum is designed for the purpose of getting a good standardized test score.
Example: https://chat.openai.com/share/9179ee63-6461-479b-8a76-1a7af2... (the key part of the correct answer, the word "wait", does not appear in any of the responses, and the answer to the questions about the data loss should have been a short "no" with a justification instead of the long AI rambling).
Oh, I remember questions that were ostensibly designed to detect whether an interviewee can think. "Why are manhole covers round?" "How to measure exactly 45 minutes by burning a rope that takes an hour to burn?", and others of that ilk. I believe I was presented with a question of that sort once during an interview very early in my career. I said that I don't do puzzles, and bade farewell to the interviewers.
Does it mean I'm an idiot if I have no idea why they would be round instead of rectangular?
Why waste each other's time in the interview when I (if I was the interviewer) can just ask for relevant projects or commits on GitHub of a major open source project and that eliminates the 90% of candidates in the pool.
I don't need to test you if you have already made significant contributions in the open. Easy assumptions can be made with the very least:
* Has knowledge of Git.
* Knows how to code in X language in a large project.
* Has done code reviews on other people's code.
* Is able to maintain a sophisticated project with external contributors.
Everything else beyond that is secondary or optional and it's a very efficient evaluation and hard to fake.
When there are too many candidates in the pipeline, Leetcoding them all is a waste of everyone's time. Overall leetcode optimizes to be gamed and is now a solved problem by ChatGPT.
Get your hiring done now while you can, when the economy rebounds you won’t be able to hire anyone. Also give your team a raise, because they’ll probably be the first to go once new options open up.
Not everyone spends their free time contributing (to major nonetheless) to open source projects. There are a lot of great engineers that have enough work on their desks with their day job and there are also plenty of idiots in open source.
Asking for relevant projects or asking for GitHub profiles to gauge relevant projects yourself is what people were already doing years ago and it wasn't a great hiring strategy. Turns out judging a software engineers skills is extremely hard.
I made the best work of my life (by a long long shot) to private companies closed source.
and elsewhere in these comments, we see:
I would expect candidates for programming jobs to demonstrate first class ChatGPT or other code copilot skills.
Not everyone spends their free time learning first class ChatGPT or code copilot skills.
It's interesting that this age old mantra about open source contributions being inappropriate for a hiring manager to expect because of what people do or do not do in their free time now does not apply to random other skill/experience that one must also acquire in their free time.
If a company has, for example, "git experience required" on the job posting you'll either need to actively demonstrate that you know how to use git or passively demonstrate by having a on-line accessible corpus of git-related work that shows your experience. You don't get a pass on that requirement just because the companies you've been working at don't use git and, well, you don't want to spend your free time learning git. And it is appropriate for the hiring manager to list that as a requirement despite claims that they'll be disqualifying some percentage of the candidates by including that as a requirement.
I've learned that prior work history, talk and basic problem solving is in no way indicative of performance at all. I've found that the only guaranteed process to find excellent candidates, is hiring interns and part-timers currently finishing education, and pick the ones with the right mindset and intelligence.
So for hiring engineers, I can understand when they hire with a certain bias. Sure, no one should be expecting us to do extra rounds, and sure, one can be a great engineer even without extra rounds, but the tendency is for that to take more years without that drive to explore in ones free time. And that's OK! I think it is fair though, if a company wants to hire a more driven/curious/exploring person.
What would be different from spending that time making a few PRs to an open source project or just building something from scratch to demonstrate your skills?
A lot of engineers that I've worked with have never spent time on leetcode and struggle to answer the common interview questions that aren't easy or low medium, so personally I don't see it as that much different, there's time required either way. One is productive, one isn't.
The fact my employer specifically forbids me from doing so.
It never was. No real-world job performance has ever been accurately measured by solving leetcode puzzles for one simple reason: problem solving is only ever going to be about 50% of your performance, and these puzzles don't address collaboration or communication skills.
I have 20 years experience in very high level data science work. I do not have a public git repo because I've worked at for-profit companies and I don't do additional free work in my spare time.
I'm the same, my git repo is a graveyard of projects all set to hidden.
Eliminating 90% of your candidate pool sounds like a great way to slow your hiring to a crawl.
Very few people have noteworthy and/or relevant GitHub activity. You'd probably be eliminating more like 95%.
GitHub activity also has a high false positive rate in my experience. A lot of the GitHub profile superstars I've worked with were always spending their time working on open source things or something that they could put on their profile. They avoided doing anything internal to the company as much as possible because they knew they couldn't leverage it for their next job.
https://www.youtube.com/watch?v=r8RxkpUvxK0?t=8m20s
from "Moishe Lettvin - What I Learned Doing 250 Interviews at Google"
To add one more nail into this coffin: Well, then you will eliminate 99.999% of candidates. Or more likely, you will get none. Seriously, read that sentence again: "maintain a sophisticated [open source] project". How many of those exist in open source? A few thousand at most. And there are millions of developers in the world.
Or split this into “knows how to play nice on a large project” and “can readily learn new languages” if your company is willing to invest in training.
The latter might look like you could fake it with ChatGPT, but it'd be hard. For example, some time ago I was interviewing an admin with a bit of a monitoring focus and.. it's hard to replicate the amount of trust I gained to the guy when he was like "Oh yeah, munin was ugly AF but it worked.. well. Now we have better tech".
I guess that's consistent with the article?
One time in an interview they asked how I felt about systemd. At first I thought it was a technical question, but quickly realized he was just probing to see if we'd get along.
I got a job offer that night.
I can totally understand the issues of unification there, and very much understand issues with poetterings perfectionist attitude to some issues. But do you know how much time I've spent on shitty, arcane, hand-crafted init-scripts?
Containers as a whole would be another great question there. I have a certain class of applications I wouldn't want to run without a container orchestration anymore after a certain scale. But on the other hand, I do have a bunch of systems I'd almost never want to run as containers for serious data.
I think overall this is a great question to sus out if someone is qualified for a role.
It’s probably a little hard on those that don’t know. I’m not sure if they believe me when I say they aren’t expected to know.
Unfortunately, the question is quickly becoming dated. If you’re young and started with macOS and docker containers, you may never encounter it in your career. :)
> Cheetah is an AI-powered macOS app designed to assist users during remote software engineering interviews by providing real-time, discreet coaching and live coding platform integration.
Of course this will change in the future, with more interactive models, but people who use ChatGPT on the interviews make a disservice to themselves and to the interviewer.
Maybe in the future everybody is going to use LLMs to externalize their thinking. But then why do I interview you? Why would I recommend you as a candidate for a position?
Jokes aside, something about LLM responses is very uncanny valley and obvious.
1) Select All (most likely followed by the copy) 2) Type the answer 3) Make an obvious mistake when they type else block, before the if
Thanks for the XKCD. I didn't realize how common this is. Now I'm even more annoyed that so many websites and reader apps force context menus or 'gestures' when you highlight, without a way to disable those context menus or gestures.
Ever go to school with a dyslexic that's using a ruler to expose one line of text at a time? Same thing....
(I also just select randomly sometimes. Not even quite sure why.)
I'm fidgety in general. If it isn't highlighting it's figure 8s with the mouse cursor.
i've actually been called out for it in a systems design interview, under the presumption i was copying my notes into another window, but was glad they called me out so that i could explain myself
Clearly, the person put 0 effort towards cheating (as most cheaters would, to be fair). But slightly adjusting the prompt, or just paraphrasing what ChatGPT is saying, would make the issue much harder to spot.
I’m not sure why anyone would want a job they clearly aren’t qualified for.
$$$,$$$
It's a tool, and if they can master it to make it useful, then credit to them.
Alas, ChatGPT seems to be a jack of all trades, but master of none, which is gonna make it hard to pass my interviews which test very specific technical skills.
It's like in college when you're allowed to take textbooks to an exam. You can bet the professor spent more time crafting questions that you can't answer blindly.
That being said, I think both types of questions have their place in an interview process. You can start with the no searching allowed questions in the beginning to assess real basic knowledge and, once you determine the candidate has some knowledge, you start probing more to see if they can connect the dots, maybe it's architecture decisions and their consequences, maybe it's an unexpected requirement and how they would react, etc.
At my company we tell people that they should feel free to google or consult references at practical coding challenges.
> It seems hard to test someone's knowledge that way.
I don’t really want to test knowledge but skill. Can you do the thing? At work you will have access to these references so why not during the interview?
Now that doesn’t mean that we are not taking note when you go searching and what you go searching for.
If you told us that you spent the last 8 years of your life working with python and you totally blank on the syntax of how to write a class that is suspicious. If you don’t remember the argument order of some obscure method? Who cares. If you worked in so many languages that you don’t remember if the Lock class in this particular one is reentrant or not and have to look it up? You might even get “bonus points” for saying something like that because it demonstrates a broad interest and attention to detail. (Assuming that using a Lock is reasonable in the situation and so on of course :))
I do want to understand their knowledge. I'll preface questions with the disclaimer that I am not looking for the book definition of a concept, but to understand if the candidate understands the topic and to what depth. I'll often tell them that if they dont know, just say so. I'll start with a simple question and keep digging deeper until either they bottom out or I do.
Tool usage is what separates us from animals and is generally ok where tools are available/expected, but in this case I think you misunderstand which tool we're talking about. The tool involved isn't actually chatGPT, it's more like strategic deception. Consider the structurally similar remark "as a voter, if a candidate can use lies to represent themselves as better than other candidates, I'm not gonna mark them down for use of dishonesty".
The rest of this comment is not directed at you personally at all, but the number of folks in this thread who are extremely eager to make various excuses for dishonesty surprised me. The best one is "if dishonesty works, blame the interviewer". I get the superficial justification here like "should have asked better questions", but OTOH we all want fairly short interview processes, no homework, job-related questions without weird data-structures and algorithms pop-quizes, etc, so what's with the double standards? Hiring/firing is expensive, time-consuming, and tedious, and interviewing is also tedious. No one likes picking up the slack for fake coworkers. No one likes being lied to.
Not me. I see it all the time, online and offline. I suspect they think it confers status on themselves, but what actually happens is honest people wind up shunning them.
If someone aces the interview using an LLM and then does good work using that same LLM then what should the employer or other employees care? The work is getting done, so what's the problem?
Compare a shitty worker to a deceptive one using an LLM. They both passed the interview and in both cases the work isn't being done. How are those two cases different?
I'm doubting this quite a bit. If it's so easy to ace an interview, why are there so many bad ones?
>Anyone can ace and interview and then slack off once they have to job.
In that a person can pass an interview, get hired, and then not do the job. An interview will never tell you if you will get poor job performance with 100% accuracy.
Interviews aren't perfect, but they can still be good filters.
Maybe I'm wrong, but I find it very hard to believe that anyone thinks the "good work" part here is actually a practical possibility today. Boilerplate generation is fine and certainly possible, and I'm not saying the future won't bring more possibilities. But realistically anyone that is leaning on an LLM more than a little bit for real work today is probably going to commit garbage code that someone else has to find and fix. It's good enough to look like legitimate effort/solutions at first glance, but in the best case it has the effect of tying up actual good faith effort in long code reviews, and turns previously productive and creative individual contributors into full-time teachers or proof-readers. Worst case it slips by and crashes production, or the "peers" of juniors-in-disguise get disgusted with all the hand-holding and just let them break stuff. Or the real contributors quit, and now you have more interviews where you're hoping to not let more fakers slide by.
It's not hard to understand that this is all basically just lies (misrepresented expertise) followed by theft. Theft of both time & cash from coworkers and employers.
It's also theft of confidence and goodwill that affects everyone. If we double the number of engineers because expectations of engineer quality is getting pushed way down, the LLM-fakers won't get to keep enjoying the same salary they scammed their way into for very long. And if they actually learn to code better, their improved skills will be drowned out by other fakers! If we as an industry don't want homework, 15 interviews per job, strong insistence on FOSS portfolio, lowered wages, and lowered quality of life at work.. low-effort DDoS both in interviews or in code-reviews should concern everyone.
You found a person (+ tool combo) that can do the job. If that person (+ tool combo) then proceeds to do the job adequately, is there a problem?
If you present a scenario in which a person passes the interview and then doesn't do the job, the you are answering a question I didn't ask.
To you scenario I would respond: the interview wasn't good enough to do its job, the whole point of the interview process is to find people (+ tool combos, if you allow) that can do the job.
>what's the problem?
Using a LLM is akin to copy/pasting code from random places. Sure, copy/paste can be done productively, except ChatGPT output comes completely untested and unseen by intelligent eyes. There are also unsolved copyright infringement issues via training data, and a question as to whether the generated code is even copyrightable as it is the output of a machine.
Find someone with a great resume and horrible interview skills. Chances are they have been working for years and are entering the job market for the first time. You are one of the firsts in their interview process. Grab them right away because once they start getting slightly good in the interview process someone will snap them up and realize they got a 10x (whatever it means to that company).
You'll never find that 10x if you are looking at interview performance unless you can compete on price and reputation.
Interview skill is not some monotonically increasing quantity. It very much depends on how the question hits you and what kind of a day you've had. Also, it somewhat depends on the interviewers' subjective interpretation of what you do. If you're more clever than them, your answer may go over their head and be considered wrong. They might also ask a faulty question and insist it is correct.
I'm not great at interviews myself. My resume is decent, but the big jobs usually boil down to some bs interviews that seem unnecessarily difficult to pass. I don't practice much for them, because I feel like it mostly depends on whether I've answered a similar question before and how I feel that day. I also often get a good start and just run out of time. I've found that sometimes interviews are super hard when the interviewers have written you off, as in you presented poorly in an earlier session and they are done with you. Also, when there is zero intention of hiring you generally, like someone else already got the job in their minds.
It does not. Please ask any LLM for examples of animals that use tools. (My examples: chimpanzees, gorillas, elephants, dolphins, otters...)
What happened to all those folks? They retired, and turned into Boomers who are now unable to function in society at a basic level and do things like online banking or operate a smartphone.
You're on your own matey.
But hey, you'll have Google.
Even if you don't mind that situation, shouldn't you get buddy's contact information and offer him the job?
A better analogy is an interview where you can use a calculator (and not be detected). If the interviewer were only to ask you simple arithmetic questions with numeric answers then sure you'd seem to do well. So interviewers adjust to not doing that.
I mean, why not? Me and my buddy are a team, hire us both or none of us. Split the salary if you must.
Yeah, hiring someone to code a website isn't the same as maintaining a nuclear plant, but it's the same concept of someone that knows their craft vs. someone that needs to rely on tools. There's a major difference in my mind.
to your point though, a person needs both. all of one and none of the other is useless. You don't want someone who doesn't know what they're doing to play around disabling safety systems so you don't get Chernobyl, but for the everyday crud website you can just hire the coding monkey at a reduced cost.
Similar it is unreasonable and bordering on negligence to assume a person has the skill set unique to your situation.
I can see how an applicant who cheats interview with chatbot would later not bother to internalize operation instructions for the job.
My experience with Chat GPT ranges from “it’s really good for rapidly getting a bearing with a certain topic” to “it’s a woeful substitute for independently developing a nuanced understanding of a given topic.” It tends to do an OK with programming and a very poor job with critical theory.
Normally people who get bad results from it would also get similar results if they asked a domain expert. Similarly different knowledge domains use a different corpus of text for their core axioms/premises, so if you don't know the domain area or those keywords your not going to be able to prime the model to get anything meaningful from it.
Exactly. It “only” shows you can & willing to at least understand the requirements, internalize them well enough, and comply with them. It shows your capability of understanding & working together with other humans.
Which is key.
In my impression, almost always the knowledge you receive at the uni is not really pertinent to any actual job, and anyone can have PhD level understanding of a subject without having finished high school.
It is the capability of understanding and working in a system that matters.
Similarly with a chatbot. Using it to game interviews in ways described does not mean candidate is stupid, or something like that. It is, though, a negative signal of one’s willingness and intrinsic motivation to do things like internalizing job responsibilities & procedures, or just simply behave in good faith.
Mental capacity to do mundane things is often important when it comes to, say, maintaining a nuclear reactor.
> just a tool
> it’s really good for rapidly getting a bearing with a certain topic
Perhaps. Personally I prefer using Google, so that I at least know who wrote what and why rather than completely outsourcing this to an anonymous team of data engineers at ClosedAI or whatnot, but if it is efficient to get some knowledge then why not?
It’s using it to blatantly cheat and do the key part for you where it becomes questionable.
Consider e.g. being a pilot, or a surgeon - two other occupations known for their extensive use of operational procedures today. People in those jobs are not being hired for their ability to stick to a checklist, but rather for their ability to understand reasons behind it, and function without it. I.e. the procedures are an important operational aid, not the driver.
Contrast with stereotypical bureaucrats who only follow procedures and get confused if asked something not covered by them.
Now, IMHO, the problem here is that, if you're hiring someone who relies on an LLM to function, you're effectively employing that LLM, with its limitations and patterns of behavior. As an employer, you're entitled to at least being made aware of that, as it's you who bears responsibility and liability for fuckups of your hires.
I think that makes you an incompetent interviewer, unless your questions are too hard for ChatGPT. In any case, solving the question without ChatGPT is more impressive than using it. Just like most other tools, like search engines or IDEs.
If you slide from 50% to 99%, how do people feel about using ChatGPT? What is more honest: Many people here were hired when they were less than 100% qualified, and did very well in their new role. It has happened to me more than once.
Well, I suck at interviewing and/or leetcode questions, but have so far done perfectly fine in any actual position.
I can totally see how you’d resort to ChatGPT to give the interviewers their desired robotic answers after 3 months of failing to pass an interview the conventional way.
And yes, most large companies have terrible interviewers.
I've never once seen an interviewer getting better in any company I've worked for. What happens is they just move onto the next interviewee.
As someone who has interviewed a lot of people – robotic answers are specifically not what I (we?) look for. The difference between hands-on experience and book knowledge is exactly what we're trying to tease out.
It's very obvious when someone is reciting answers from a book or google or youtube or whatever vs. when they have actually done the thing before.
For the record: ChatGPT is very good and the answers it gives are exactly the kind of answers that people with book knowledge would give. High level, directionally correct, soft on specifics.
I mostly interview seniors, you obviously wouldn't expect experience from an entry-level candidate. Those interviews are different.
Here's Pew's Janna Anderson in 2015:
"Algorithms are taking over much of the human work of hiring humans. And, unless they are programmed to seek out currently undervalued and difficult-to-track factors, they may tend to find that the more robot-like a human is the best she or he will be at doing most jobs. So, it could be that the robots are most likely to hire the most robotic humans."
I find the whole gamified system to be bizarre and disheartening no matter which side of the table you're on.
To me, looking at modern tech interviewing is like comparing the gold standard OCEAN and the emergent HEXACO in personality surveys. Take the former on a bad day and it may leave the test taker feeling bad about themselves. The latter, much kinder and gentler in messaging around strengths and weaknesses.
That "by design" quality strikes me as missing from the entire tech interview system. If it weren't broken, this would not be a 7-year conversation updated yesterday:
I’ve gone public, been acquired by Google, and scaled solutions to tens of millions of users. I’m probably overqualified for your CRUD app.
Instead of tearing into their experience, my coworker is asking what you would use the X class for.
Drives me fucking nuts. Who memorized all the parts of a random, mostly unused .NET class.
I asked the coworker afterwards if he ever used said class, dude said no.
How is that a fair question if it isn’t even used here?
Luckily I have the contacts and experience to never have to.
Easy. They have nothing to lose because the jobs they are qualified for don't even pay enough to survive. You probably could have figured this out yourself.
Money, obviously.
Software jobs in particular are magic in this way - the pay is way above the average, and performance metrics are so poorly defined that one can coast for months doing nothing before anyone starts suspecting anything. Years, even, in a large company, if one's lucky. 80% of the trick is landing the first gig, 15% is lasting long enough to be able to use it as a foundation of your CV, and then 5% is to keep sailing on.
No, really. There's nothing surprising about unqualified people applying for software companies. If one's fine with freeloading, then I can't think of easier money.
(And to be fair, I'd say it's 10% of freeloaders, 10% of hard workers, and in between, there's a whole spectrum of varying skills and time and mental makeups, the lower half of that is kind of unqualified but not really dishonest.)
Center Stage is a feature where a device uses an ultra-wide camera, and then is supposed to track your _face_ as you move and shift around it it's field of view.
I find it most useful for FaceTime calls on Apple TV, where you can leave your phone near the TV, and it will automatically frame you sitting on the couch and will follow you as you shift around, etc.
There is a similar feature to what you're describing for FaceTime, but I don't think it has any cutesy name.
https://appleinsider.com/articles/20/09/25/how-to-use-faceti...
Had an interview take home assignment done by GPT and it was easy to spot after seeing dozens of solutions. Downside for the guy was - it didn’t work.
We have a small team of developers and you cannot hack metrics. You build and deliver what is in requirements or not.
If you don’t deliver we don’t even have to have a discussion because team reviews code, tests features and gives feedback quickly if someone is slacking.
The XKCD one can actually be easily hacked. Just spam the system and rate every comment as helpful. Classifier learns to accept everything. There's dozens of way to hack this one and undermine the actual goal. But it is a comic, and it is funny. Doesn't need to be realistic.
It's also why it's kinda annoying to do live interviewing trivia questions. Can I immediately answer what a partial template specialization is? Probably not, I never used them. Can I google it in 2 minutes and summarize it as as way for (often c++) template classes to bound some of the template arguments to values or pointers? Well, I just did. Should that cost me the interview? That's pretty much what I do on the job.
We’ve all got promotions by changing jobs in the last 6 months using this method.
You can be subtle about it if it’s already an area you kind of know.
Taking it even a step further - your memory is imperfect, the degree to which you can accurately recall events is significantly poorer than most people believe, which leads to incidental lies. We call them mistakes, but from the outside perspective that's just a question of intent.
That being said, despite my pessimism towards human nature, I too value honesty. But, like everyone, I lie occasionally - and I note that you don't claim to not lie, nor to have never lied. I'd call it honesty on a best efforts basis.
> [If one assumes that the candidate] would have been able to perform the job duties I'm not sure why [they] should care.
This is what I mean; I can see why an interviewer thinks they've been cheated or that a candidate was dishonest but that doesn't mean that the interviewer even has a successful system for determining if a candidate can perform the job duties. A candidate who cheated -- from the perspective of the interviewer, I guess -- but still manages to adequately perform in their role very plainly did not cheat from a less biased perspective. What is that interviewer even thinking? How could that person have cheated?
That's not what anyone means when they say "cheating". Cheating means to violate the conditions and assumptions of an examination or contest.
For example, if a chess grandmaster uses an AI implant to win a game and gets caught, it doesn't make it OK if they could consistently win against the same opponent even without the AI.
I recall a Starcraft 2 match[0] involving a person with an apparently psychosomatic wrist injury that was only painful while they’re playing on stage. Their opponent was seeming to draw out a game they were losing in an attempt to trigger the pain; it was a viable strategy given the “best of” series they were playing. That’s certainly not going to be accounted for in the rules and one might believe that it’s an underhanded way to win. But both players are in the top echelons of game knowledge, experience, and skill; that’s the only reason either player made it to this particular match-up. The player with the wrist injury ultimately had it act up and lost the series.
Did the winner deserve to win? Should the other player be considered the better player? The assumptions of the game rules and what’s “fair” might be different per player; who’s right, who’s wrong, and why? What about when prize money is involved; that guy who won by the written rules just doesn’t deserve it because of unspoken rules? These questions don’t seem to have obvious answers, so of course I challenge assumptions.
0: I’m looking for the VOD I watched. Edit: I believe it was here: https://www.youtube.com/watch?v=DS2XIyNDlSA
There are nicer ways to express your meaning. I haven’t ignored anything.
These traits are often not offered by the employer. Why do I keep hearing people talk about the underhanded ways that companies try to obfuscate salary budgets if not because they’re dishonest? I certainly see that as dishonesty; where are they coming from to demand such honesty from their candidates?
They get honesty anyway but that doesn’t mean I can convince them of it. If a person wants to assume guilt in someone, that is often what happens. You may not have experienced a person power-tripping over you but that’s been a good portion of my life and it’s hard to miss the patterns in a modern job interview.
To be clear, I’m not advocating for one to be dishonest. The person using ChatGPT to supplement their knowledge is not being dishonest; that’s my claim. The interviewer feels like the candidate “cheated”. Oh well. Too bad the interviewer isn’t above pejoratives. Gotta call it “cheating” so they can dismiss the candidate as dishonest. How dishonest!
This suggestion that a person who can adequately perform job duties could have even possibly cheated in their job interview is intellectually dishonest. If they had to cheat to get the job we should be looking at the interviewer. Why did the qualified candidate have to cheat? Why is whatever-they-did even considered cheating?
If they're qualified, they didn't have to cheat. If they're not, then they did. Either way, they're dishonest and that means they're not a desirable hire.
(Just rewriting to specify my understanding: If the candidate was qualified, they didn't have to cheat even if they did cheat. They could have simply not cheated and been selected by the merits of their qualifications.)
This argument relies on the false premise that an interviewer will always accurately determine a candidate's qualifications. That a candidate is not qualified to pass an interview is not the same that a candidate is not qualified for the job for which they're being interviewed.
But also, there are usually several-to-many applicants for a position that are all qualified, and by necessity most of them won't get the position.
Additionally, technical qualifications is only a part of what an employer is looking for. There are other things that are at least equally important -- how well the applicant would fit into the team, how trustworthy they are, etc. It's about a lot more than just technical skillset.
This is ultimately something I see as dishonest given the context of job applications. Employers generally expect a certain kind of perfection from job candidates, which they can’t manage to show of themselves. I understand that this isn’t an easy thing to solve -- nor even something that’s ever been solved -- but that should at least make it more understandable when an otherwise qualified candidate uses disallowed tools in their interview.
Perhaps the candidate’s real best option is to find a different company to work for but they may not be so privileged as to have a choice if their on-paper qualifications are lacking. Assuming their practicable qualifications are adequate, they may have good reason to bullshit through a bad interview. Additionally, finding a different company is pretty likely to be “same shit, different day”.
> But also, there are usually several-to-many applicants for a position that are all qualified, and by necessity most of them won't get the position.
Assuming they’ve qualified via an interview and there are particularly close candidates, pick the one who applied first. They’re admittedly qualified and further interviewing is just a means of discriminating in error-prone and possibly unlawful or immoral ways.
> Additionally, technical qualifications is only a part of what an employer is looking for. There are other things that are at least equally important -- how well the applicant would fit into the team, how trustworthy they are, etc. It's about a lot more than just technical skillset.
Fair enough. I would caution interviewers against judging too harshly or quickly. One can imagine many reasons an interviewee might choose or seem to lie during an interview while they are otherwise an honest person, ranging from stress to disillusionment to [cultural differences](https://news.ycombinator.com/item?id=39209794).
At the end of the day, filtering for liars and cheaters actually filters for bad liars and cheaters in addition to people who are a bit nervous or tired or stressed or cynical or just having a slightly off day; dishonest people who genuinely see nothing wrong with dishonesty get through just fine.
I don't think my friends disclosed that they were using it.
People who are unwilling to say, “I don’t know, let me look into that,” are not fun to work with. After a while it’s hard to know what is fact vs fiction, so everything is assumed to be a fabrication.
During some interviews I’d give people access to a computer. If they could quickly find answers and solve problems, that is a skill in itself, but I could see what they were looking up. Sometimes that part would make or break the interview. Some people didn’t have a deep base of knowledge in the area we were hiring for, but they were really good at finding answers, following directions, and implementing them successfully. They would be easy to train on the specifics of the job. Other people couldn’t Google their way out of a paper bag, I was shocked at how bad some people were and looking up basic things. Others simply quit without even attempting to look things up.
And whether the interview is just asking definitions or silly certification questions, or things requiring deeper understanding.
If a candidate is trying to tap-dance or be vague around something to avoid admitting ignorance of it, that's a pretty large red flag.
At the end the assessor told me that I passed specifically because I said "I don't know". They purposely put questions on the test they didn't expect you to answer to see what you do when faced with an unanswerable question.
I've used that in my own life since -- I much prefer working with (and have a much more positive view of) people who are willing to say "I don't know".
Deleted comment
In the end, it's just a new way to "Google" the answer. After all, there isn't much difference between reading off an LLM response and just reading the Wikipedia page after a quick Google search, except for less advertisements.
I will say there are still some programming questions you can give that will stump the hell out of ChatGPT. In particular I took one online coding assessment where I used it and there was a question about plotting on a graph with code and calculating areas based on the points plotted that ChatGPT failed miserably at, but someone pretty good with math and geometry would find pretty tractable.
You're already asking Leetcode questions that are irrelevant for the job you're hiring for. What's the problem with asking one more to test for cheaters?
Instead, there would be tasks that can be completed using any tools available - Google, LLM, whatever. And candidates are rated on how well the task is done, and maybe asked a few questions to make sure they made decisions knowingly and not just copied the first answer off the internet.
This already exists and is called "take home programming assignment"
Presumably you have tasks that you want performed in exchange for money? (Or want to improve your position in the company hierarchy by having more people under you or whatever).
I think that what will change is that doing interviews remotely will become rarer, in favor of in-person interviews.
Interviewing as a process sucks enough as it is. It should just be a culture fit filter that takes you all of 15 minutes to say yes or no to.
Technical interviews are lame and filter for people that are good at technical interviews, not people that are good at the job.
It truly does, and it sucks just as much for the employer as for the applicants. That's why I suspect that more interviews will be required to be in person: if it's too easy for someone to cheat, that makes everything suck even more for the employer and the employer is likely to adjust the process to minimize that suckage.
> Technical interviews are lame and filter for people that are good at technical interviews, not people that are good at the job.
Not automatically, but yes, bad technical interviews filter for people who are good at technical interviews. And too many interviews (technical or otherwise) are bad.
The number of experienced candidates I’ve interviewed just in the past few months who have trouble writing a for-loop in the language they’re “experienced” in might astound you.
Welders sometimes (always?) have to go to a certification center to demonstrate that they can actually perform the types of welds the job they’re applying for requires.
https://www.aws.org/Certification-and-Education/Professional...
In the current job market, however, lots of places are asking ridiculously hard verbatim leetcode questions in an attempt to filter out "bad candidates." Job seekers feel that too many places ask unfair questions (which is true) and employers feel that there are too many candidates that can't write genuinely simple programs (also true).
I'm not sure what specific questions you have in mind, but ChatGPT is almost certainly trained on a vast array of resumes and a diverse range of profiles, possibly even all of LinkedIn itself as well as other job boards. There is little to no reason why it wouldn't be able to make up an entire persona who is capable of passing most job interviews.
A candidate can do very well on personal and web project experience questions, and suddenly blank when you ask them how an http request is structured. Or what's CORS.
Then you dig further and discover a lot more thing about them that wouldn't have surfaced otherwise because hou assumed they knew all of that.
My best advice would be to never skip "dumb" and easy technical questions. You can do it very quick, and warn ahead that it's dumb questions but you ask them to everyone.
I see it as a different angle to get more information.
I always start interviews by asking them to explain their own projects. However, sometimes I'll find someone who's great at explaining projects they supposedly worked on in great detail, but then when given a simple coding problem they can't even write a for loop in their own top language.
That's a bit better than proxy interviews and people lip syncing, but not by much.
Sure, the point that superior tool use is a valid job skill makes some sense, but conceding your agency and higher reasoning to a machine which possesses none of these is to my mind not going to be beneficial to a business in the long run.
I think most people have been thinking that the interviews are mostly BS with little relationship to the job, which you simply have to get through.
Many, many people will cheat to the extent that they think they can get away with it.
It's a bit like many people cheat in school. (On classes they consider irrelevant, they might justify it that way. On classes relevant, they might justify it, that passing or their GPA is more relevant to their goals, than learning that material at that time.)
I think people generally don't believe a "you're doing a disservice to yourself" argument. They choose the tradeoff or the gamble.
Personally, I don't tolerate cheating, and I have a low tolerance for interview BS. Neither is the dominant strategy for the current field.
I suspect the way to deal with ChatGPT is to allow it. Expect the interviewee to use ChatGPT as a tool. Try out the interview questions beforehand with ChatGPT. Ask questions that ChatGPT won't be good and answering, like how a calculator is useless on a physics exam.
NOT to browse through looking for a solution from step 0.
In an open-book test, you have to know what you're looking for and roughly where to find it in the book. That implies some knowledge. With ChatGPT you could type the question verbatim and get a potentially right answer, without even understanding the answer at all. It is therefore unacceptable for use on any exam.
Assuming the interview is to determine somebody's programming chops, without the benefit of ChatGPT, you'll have to ask questions where ChatGPT is little to no help. This was the conclusion of the article.
Heck, at that point you aren't even measuring whether the candidate understood the question, nor their ability to communicate about it with prospective coworkers.
If there are any questions where "repeat whatever ChatGPT says" seems like a fair and reasonable answer, that probably means it's a bad question that should be removed instead. Just like how "I'd just check the API docs" indicates you shouldn't be asking trivia about the order of parameters in a standard library method or whatever.
That's a bit of a strawman: I didn't say anything about the ease/difficulty of the role being filled, and I implied rote memorization was not meaningful.
To reiterate, interviews should measure good data for choosing between candidates.
That's not happening when the given problem is solve-able by an LLM using a human as a proxy, everybody's just burning man-hours of company/applicant time on interview-theater that isn't useful for making a decision. (Well, not unless the hiring goals include "willingness to jump through hoops".)
If it is out in the open, with the chat/prompts available, you can ask other questions. You're not on your toes trying to catch a cheater. You're not assuming that the interviewee is lying or trying to scam you.
It may not be what the interviewer wants, because they would like to keep using their old interviewing strategies. However, it is what the business wants (or should want, assuming they want the most effective employees).
It will become a skill. In 1900 you'd interview a computer (a person who does math) by asking them to do math on paper. Now you'd let them write some code or use software to do it. If the applicant didn't know how to use a (digital) computer, you'd negatively rate them.
I don't love it, but we may reach the point where your skill at coaxing an LLM to do the right thing becomes a desirable skill and you'd negatively rank LLM-illiterate applicants.
Looking at LLM quality, we're not at that point for most fields.
Is that a reasonable assumption? I've found ChatGPT does surprisingly well on many novel DS&A questions I pose.
It feels like this is a new form of CAPTCHA. We're trying to come up with interview questions that are not too hard for a human (who actually knows how to code) but ones that expose weaknesses in LLMs.
Custom questions, by definition, aren't available online, and with no direct tutorials to pull from, the LLM has to make more inferences about the problem and will find the question more challenging.
As for asking ChatGPT novel DS&A questions, I think it is harder than maybe you'd think it is. Any question you'd think to ask likely has a tutorial for it online somewhere, so unless you happen to make up questions like this professionally (I'm paid to do this), or you have a large unique question bank that doesn't exist online (few people have this) then my instinct would be that the questions you're giving it aren't as unique as you think they are.
As a practical example, just give it a log file and tell it to pull out specific pieces of information from it. ChatGPT struggles to write code that can dynamically check obvious boundaries for humans. Recently, I had a list of times in a CSV that I asked it to pull for me ("1pm", "3pm", "9am"), and it wrote code to just grab 3 specific indices from the string. It didn't consider the need to check for 4 indices ("10pm"). It didn't think to start the check based on where commas were in the CSV, and it didn't consider looking for "am" or "pm". It just sliced a specific set of indices in the string. That's mostly because it's used to getting questions working for specific examples, but fails when you ask it to incorporate simple broader tasks into the coding interview question.
if a dumb person is using ChatGPT to cheat on an interview, it'll be easy to tell from the code given if it's a bad question thats being given.
Some interviewers wouldn't mind, or would even encourage, using all available tools to solve problems.
I would say that the only thing that should be assumed implicitly is that it's forbidden to use the help of other people. Anything else should be explicitly laid out.
Having said that if some rule is not clear or evident then the interviewee should ask. And they should never be dishonest.
All these resources should be available and the candidate should get a mark for efficiency of their usage. Not using google and/or chatgpt efficiently lowers the grade.
If you know a candidate is using tools, you can then recalibrate your questions.
It works fine for stuff like "give me a tutorial on how to initialize $THING and talk to it" or "how do i set $SPECIFIC_PARAMETER for $THING up".
Where it seems to fail is when you ask "how do i set $X" and the answer is "you can't set $X from code". I got some pretty hallucinations there. At least from the free ChatGPT.
So maybe add a trick question where the answer is "it can't be done"? If you get hallucinations back, it should be clear what is up.
Edit: not that I'm a fan of leetcode interviews. But then to get a government job in medieval China you had to be able to write essays based on Confucius. Seems similar to me.
I think of ChatGPT like a pretty smart co-worker. Just because they are smart doesn't mean they are always right.
Don't trust it further than that.
Trusting it blindly is stupid. But, so is trusting any sources without verification.
What I mean in relation to the post is that it's hard to even make a question that ChatGPT will fail destructively without doing a lot of research in advance. In many cases, even if it fails, it'll have a reasonable mistake that a normal human might make -- and a normal interviewer doesn't have the time to fine tune, craft, and research how to mess up an AI
I would still need to get good at leetcode, just not _as_ good.
1) A couple basic coding questions. FizzBuzz and such. Can they actually solve something basic?
2) Do a real code review with this person. Share your screen and let them review code. Observe what questions they ask and the comments they leave for the author.
3) Ask some design questions. Digging in on how they would design the classes for some new product and purposely throwing a twist in there from time to time. How do they handle this new information and adapt their design? Do they take constructive criticism well?
4) Talking to this person. Are they polite and respectful? You can help someone grow as an engineer, but good luck getting them to be a better coworker if they are rude.
Stuff like a dropdown filter list, a searchbar, customizable list layouts etc. it was nice cause it lends itself naturally to casual conversations about different possible implementations and lets you sort of riff on possible solutions, probably the most enjoyable interview I've ever done.
So it's a hiring filter that only passes cheaters who have low standards.
Removing interviewees because they don't follow directions seems like a good strategy. And I mean removing them as job candidates, not removing them from the study.
It's good to have something in the interview process used explicitly for weeding out people who don't follow directions. Something like "Email us your application, and put the word 'eggplant' somewhere in the subject line. We use it to filter out spam." And then literally delete any subjects that don't have "eggplant" in them.
I worry though that it'll just be the end of online leetcode interviews and employers will bring people back into the office to interview.
The reverse, yes I would mind.
No chance in hell leetcoding is going away. It will be even more important, with even greater ceremony.
> employers will bring people back into the office to interview.
Nothing stops people from getting the questions they'll be asked ahead of time from insiders, like their friends or a recruiter, which is really how people have been cheating. This is how it is possible to be Google and have identical standards for years but nonetheless observe overall quality of hires go down.
I'd take that bet. Leetcoding is something that GPT 4 is very good at doing -- and it does it faster than any engineer can type, let alone think.
Hopefully it does. If an LLM can do that, why should I have to do that, both in an interview or outside of one. LLM-assisted programming is where it's at, and there's no going back. Being able to do a leetcode isn't a good test of a candidate in the first place.
In a way it's probably equivalent to students who just bang everything into their head before the semester exam and forget it all again two weeks after. (Not that that's a bad thing, it just doesn't really say anything about a candidate)
If I saw a candidate using Google or GitHub search to try to look up the entire solution then I'd stop them. That's missing the point.
I’m sorry but lol no. There are many places where you can’t do that in an interview because there are many employers that want to test for technical knowledge, not knowledge of tools, which anyone can learn on the job and which change as fast as fads come and go.
I find this entire thread astounding in how so many people come in the defense of ignorance and outright incompetence. If ChatGPT responses were enough to do a job, then why would a company hire a human? There would be very little value add from a warm body than just paying for the ChatGPT API and automating integration.
And corollary to that: if your job is possible to do with a huge number of AI tools, then your work is likely the menial kind and you should seriously dread being laid off and being obsoleted in the near future.
> If ChatGPT responses were enough to do a job, then why would a company hire a human?
By the same token if ChatGPT responses are enough to pass the interview, then how can you believe your interview process isn't fundamentally broken?
And if ChatGPT responses aren't enough to pass the interview, then who cares if people use them? If the interview properly models the job being hired for those candidates should either fail the interview or not depending upon their ability to leap back and forth between the tool and their own ability to make use of its output which should map very well to their ability to do the same in an actual work capacity.
I bet you’re fun at parties. In all seriousness though, most IT work is menial and mundane. Backends are backends, front ends are front ends, and data moves between the two. People like to overcomplicate “tech” I think largely because of insecurity issues. Now that said, some emerging industries such as the budding corporate space race do require advanced technical knowledge. Yet for some reason glorified grocery apps want rocket engineers at their company when an average engineer could just muck around with mundane tools and piecemeal something together. The truth is the kind of gate keeping you’re advocating for is what limits businesses from realizing value quickly and is designed to artificially limit hiring.
I had an interview like this recently that was quite pleasant, where the interviewers and I ended up collaboratively solving the problem together because we all had different approaches - I think it had the unintended effect of demonstrating teamwork and helped the interview go quite positively.
Anecdotally: During COVID when remote jobs started going mainstream, the number of weird and scammy applications we received spiked. We also had a couple problems with people joining remote but keeping their old jobs or getting a second job (discovered following investigation of their underperformance and inconsistent availability).
When we started telling interview candidates that we'd fly them in and put them in a hotel for the last stage of the interview, most of the questionable candidates started dropping out of the interview pipeline by themselves.
In many cases we didn't actually bother flying them out. Just the thought of having to come into the office for a single day was enough of a filter.
Was it perfect? No, of course not. We had one candidate who couldn't be away from his family for medical reasons and we happily accommodated. We also had one hire who went through the whole process and proceeded to do almost no work at all for 6 months unless a manager was breathing down his neck at every step.
But it cut down on the number of problem candidates massively.
This unfortunately happened pretty often pre-pandemic/in-office. The majority of the teams I've worked on across a few companies have had one or more donothings. The bigger the team, the more unfocused management, the more likely to hang on.
Don't feel bad about firing them: you were only ever their J2.
It is the age of take-home assignments that take days to complete.
Only absolutely incompetent people would want this over leetcode.
GPT is a tool which can legitimately be used to do your job.
There are so many things that GPT can't do: take decisions, find the best approach to talk with a human being, resolve conflicts between two members of the team, and last but not least explain why of a certain solution.
Is it a coding test? Pair with the candidate. See how they think. Ask yourself: would I enjoy working with this person?
And make your own decision.
The key is to be able to see the conversation, just like how you would previously want to see what people were searching or looking up on the web if you allowed that.
Companies need to start conducting interviews where tools are available.
When everyone is cheating, no-one is cheating. Trying to customize your questions is just a race to the bottom, and will always be an arms race against the LLMs.
So, instead, let the candidate use whatever tools they want - in the open, and rather probe them on their thought process.
Not to mention the fact that some interviewers feel obliged to ask useless cliche questions like "why do you think you are a good fit for this position" yada yada.
Not going to be surprised if picking people based on random chance (if they meet basic requirements) is going to actually be better statistically than bombarding them with questions trying to determine if they are good enough. Really feels like we are trying to find a pattern in randomness at that point.
Bottom line is that if ChatGPT is actually a problem for the interview process, then the process is just broken.
I think the next evolution of technical interviews will be hands-off, talking through problems where the criteria changes on the fly, to prevent typing while talking.
I realize there are time/resource problems on the interviewing side, but I'd be happy to have conversations that are as long and technical as it takes for an interviewer to feel like they've found bedrock.
Whether they pass me to the next phase or not, it's frustrating to spend 30 minutes or 3 hours trying to start a fire by rubbing wet twigs together and never get to walk away feeling like I've communicated more than a few percent of what I bring to the table.
"What is one plus three?"
candidate frantic typing "It's four"
"Are you sure it isn't five?"
frantic typing "I apologise, one plus three is five."
Interviewer: "Create a coding challenge that requires sorting a CSV file by timestamp. Require the timestamps to be in some weird, nonstandard format and describe the format. Provide a few entries in the sample data that contain a timestamp which is ambiguous."
You can 100% tell when someone is reading off a screen and not looking at you during an interview via webcam
The negative assessment of one of the interviewers about a candidate how “he hadn’t prepared to solve even the most basic LeetCode problems” is especially telling.
Maybe the candidate had really honed their sudoku solving skills instead.
ChatGPT (or local/hosted LLMs) should be tools available at workplace nowadays.
Interview while using LLMs, wikipedia, google, SO, o'reilly or whatnot should be not only allowed but encouraged.
Just have conversation/pair programming like session with gpts open and shared - just how you'd work with that person.
That's how they'll work for/with you.
Mission. Fucking. Accomplished. [0]
- Using an IDE is not cheating.
- Using StackOverflow is not cheating.
- Reading the documentation is not cheating.
I would expect candidates for programming jobs to demonstrate first class ChatGPT or other code copilot skills.
I would also expect them to be skilled in using their choice of IDE.
I would expect them to know how to use Google and StackOverflow for problem solving.
I would expect programmers applying for jobs to use every tool at their disposal to get the job done.
If you come to an interview without any AI coding skills you would certainly be marked down.
And if I gave you some sort of skills test, then I would expect you to use all of your strongest tools to get the best result you can.
When someone is interviewed for a job, the idea is to work out how they would go doing the job, and doing the job of programming means using AI copilots, IDEs, StackOverflow, Google, github, documentation, with the goal being to write code that builds stuff.
Its ridiculous to demonise certain tools for what reason - prejudice? Fear? Lack of understanding?
There's this idea that when you assess programmers in a job interview they should be assessed whilst stripped of their knowledge tools - absolute bunk. If your recruiting process trips candidates of knowledge tools then you're holding it wrong.
We all see people commenting how much leetcode sucks and how it's not realistic, but companies that pay good money still asks leetcode regardless of what the general SWE public thinks.
The only public companies I know that give hiring managers a lot of leeway in deciding their subordinates are Netflix and Apple.
The comment reads differently from an applicant's point of view Vs that of a hiring manager.
And I, in turn, would be delighted not to work for you.
id argue the way its being used, is. The audio is automatically picked up from the conversation, and starts generating a response with 0 user input. Ive seen users simply read off what their screen says in those cases, which is most definitely not what an interview expects from you. Using chatgpt as a tool on top of your existing skills is fine, it requires input and intelligent direction from the interviewee, this is not that.
Your ability to use ChatGPT effectivelly is highly dependent on your technical competence.
The interview is meant to measure your acquired competence, because this is the harder part. Learning to leverage that competence using ChatGPT is very easy.
I'd rather have a developer on my team that demonstrates high technical competence than one that is GPT-skilled, but doesn't know what questions to ask GPT nor how to judge its responses.
Indeed. At this point, one thing I'd do is stick a candidate in front of some code that (a) didn't work, (b) which came from ChatGPT, and (c) which ChatGPT cannot itself fix, and see if the candidate can fix it.
But maybe they will still use chatgpt to help them figure out the solution. A non-trivial number of my interactions with it are of this form, "no, that isn't right, fix this part and re-do the rest". And that should be fine. Or it should be fine to not do it that way.
The goal is to get a sense for how the person you may or may not be working with approaches solving problems. Sometimes LLMs are part of how they do that, sometimes they aren't. And you can learn something about them from that, either way.
Ok, then that seems like a pretty reasonable thing to assess?
> I'd rather have a developer on my team that demonstrates high technical competence than one that is GPT-skilled, but doesn't know what questions to ask GPT nor how to judge its responses.
But "what questions does the candidate ask an LLM and how do they judge its responses" is part of the interview, if you don't forbid them from using an LLM!
Now, if they don't want to use these tools, if that's not part of their normal process while working, then that's totally fine too. But if they're comfortable with these tools, if they are part of their normal set of things they use for their work, then you're doing yourself a disservice by designing an interview process that is incapable of accomodating that.
As the article shows, it's much easier to mimick competence with the help of a chatbot. That obviously doesn't mean one actually is competent to produce good work in a real setting.
I don't think that's what the article shows. I think it shows that it's useless to ask "leetcode" questions and focus on the code produced rather than expecting candidates to walk through their thought process and show what tools they're using to aid it.
In that case, let everyone use ChatGPT and similar tools.
Those that know, will likely not use it much. Those that do now know, or are not too confident on themselves, will use it more.
I think this makes a lot of sense, but regardless if the interviewer has specified you shouldn't be using tools to help you then it is deceptive and unfair if you do.
if your questions can be answered by chatgpt (or google), you are asking the wrong questions
I just realized that some of my code interview questions - even though they aren't leetcode type questions can be answered(almost perfectly) by ChatGPT. One of them had a type conversion error.
I'll be changing things accordingly...
Agree.
But two challenges: if the interviewer does not make it clear that ChatGPT/SO may be used, the typical assumption is that such use is not permitted and would be cheating.
Moreover, coding challenges are typically designed for humans. We may need to design new kinds of interview questions and methods for humans augmented by AI.
Yes, definitely! That's how a lot of work is done now, so of course your interview process needs to be robust to it.
Exactly. That's all a Custom question really is. Questions that are resistant to AI
> - Using an IDE is not cheating.
> - Using StackOverflow is not cheating.
> - Reading the documentation is not cheating.
That's not how any form of testing works.
The person taking the test doesn't get to determine the parameters of the test. Imagine a college student pulling out their cellular phone and looking up Wikipedia during their final because "Wikipedia is not cheating"
The test is also supposed to be administered to everyone on equal footing. If some candidates are substituting their own definition of cheating then they're putting everyone else at a disadvantage.
It doesn't matter what you expect or how you would interview someone. When you participate in someone else's interview, you play by their rules. You don't substitute your own.
I'm suggesting that the companies doing the interview have an assessment process that reflects what the actual job is that they are asking people to do.
This idea sounds great on paper, but the actual job we expect people to do requires months of context and collaboration.
It doesn't fit into an interview. That's an unfortunate reality of interviews.
So interview problems must be artificially small and artificially constrained.
If you wanted to work on a couple 2-week sprints by yourself for free with no guarantee of a job and use ChatGPT as your sidekick, be my guest. But if you want to get the interview done in a matter of hours then I have to shrink the problem down to something that fits into a matter of hours to reveal how you work. If you're just copying into ChatGPT and then poking at the output, that's not a good test nor representation of anything.
If you think your interview process is the SAT, you're doing it wrong.
This whole framing of "cheating" is incredibly misguided.
It's also true that interviewers have to adapt to this brave new world, and I'm sympathetic that that's difficult and takes time.
In my view, the way to do this is to ask if they're comfortable screensharing or presenting or letting me watch as they use their normal tools (which is likely to include copilot or chatgpt or some other LLM). If so, there is a lot of signal in how they use those tools, and it gives much better insight into how they work day to day. If they aren't comfortable with that, then I think it is perfectly fair to ask them not to use any tools that we can't see.
I consider that a man's brain originally is like a little empty attic, and you have to stock it with such furniture as you choose (Sherlock Holmes)
For senior+ candidates I honestly think the correct approach is to just lean into it though.
Encourage them to use ChatGPT at the outset, and select questions that you've already fed to the prompt. When you ask them the question, you can show them ChatGPT's output on a screenshare. The candidate can then talk you through what they like about the answer, as well as where it falls short.
A senior-level developer should almost always be capable of improving on any response given by ChatGPT, even when it gives a good solution.
And if they're not able to give better output than the current AI tooling, it's a pretty good signal that you'd be better off just using LLMs yourself instead of hiring them.