AI is the reason interviews are harder now
softwaredesign.ing
softwaredesign.ing
>> gpt-4 is at least as good as i am at code reviews, so i don't think this solves the problem this post is about
These two comments don't seem consistent.
Honestly, "i've found mistakes in its code reviews" vs "it's absolutely superhuman at is avoiding red flags" is possibly not self-consistent. But I think you mean glaring mistakes?
Which if I understand you correctly, than I'm not sure how you get the first conclusion unless you are saying you're mediocre (which is fine). But really it just makes it seem like you and the machine complement one another, rather than compete, which makes me not understand the original comment in context.
But I mostly agree with Copenjin. Interviews are less about on the spot skill checks so much as learning how someone thinks and problem solves. Honestly, you could play a boardgame/videogame/cardgame with them and it would be an effective interview (asking them relevant questions during the game, and maybe even better if it's a relatively unknown game so they can zero shot it). The reason for this is that in real world work you are more concerned with how someone adapts to changing environments and thinks through situations. To see when they'll ask for help, what they might get stuck on, and how they strategize.
Your employees will always be gaining new skills. And honestly, it is easier to take a lower skilled person who's more adaptable and driven and turn them into a great employee than it is to take someone who's got skills but will stagnate. But ymmv depending on the job and requirements. Sometimes you just need to fill a seat.
by 'red flags' i inferred copenjin to be referring to things like getting aggressive or defensive, rather than making dumb mistakes, but i could be wrong about that. i guess there are also some mistakes that are so dumb that they'd be a red flag, and i have to admit that gpt-4 is somewhat subhuman at avoiding those
if you play a board game with someone you can assess their general intelligence and capacity for logic. all else being equal, having more general intelligence and knowing how to think logically do make you a better programmer. (if you just want to assess general intelligence, a much faster pair of tests would be reverse digit span and reaction time.) but those are far from the only things that matter, they're not enough to be a great programmer, and they're not even among the most important factors. other important factors in programming include things like knowing how to program, knowing how to listen, and being willing to ask for help (and accept it), which a board game generally will not test
I agree with you. Which is why I say that they complement. But your reply to Baron implies that what they were suggesting wasn't a solution. I agree with the sentiment of the post to do things in person. But what I take from Baron is that it is much harder to fake the process with GPT because the actual part of the code review isn't so much about finding the bugs, it is you watching someone perform the code review (presumably through screen sharing and a video chat). You could have this completely virtual, but I think you're right to imply that the same task could be then optimized.
But at the end of the day, I think the underlying issue is that we're testing the wrong things. If GPT can do sufficient, then what do we need the human for? Well... the actual coding, logic, and nuance. So we need to really see how a human performs in those domains. Your interviewing process should adapt with the times. It is like having a calculus exam where you test someone and ban calculators but also include a lot of rote, mundane, and arduous arithmetic calculations. That isn't testing the material that the course is on and isn't making anyone a better mathematician, because any mathematician in the wild will still use a calculator (mathematicians and physicists are often far more concerned with symbolic manipulation than numerals).
> other important factors in programming include things like knowing how to program, knowing how to listen, and being willing to ask for help (and accept it), which a board game generally will not test
I agree the game won't help with the first part. But I thought I didn't need to explicitly state that you should also require a resume and ask for a github if they have one. But I did explicitly say you should ask relevant questions. And I'm not entirely convinced on the latter, which are difficult skills to check for under any setting. There's a large set of collaborative games in which do require working and thinking as a team. I was really just throwing a game out there as a joke, being more a stand-in for an arbitrary setting.
At the end of the day, interviewing is a hard process and there are no clear cut solutions. But hey, we had this conversation literally a week ago: https://news.ycombinator.com/item?id=40291828
"Can you scroll back a bit? Now, can you show me the docstrings for Foo.detachBar and Bar.attachFoo?"
Are you bad at code reviews? Is the code you're reviewing fairly standard?
GPT misses nuance. It can't reason while you still can. It certainly can do certain tasks better than you but certainly humans can do better at other tasks (specifically in the creativity side, logic, and when it comes to nuanced thinking). But if you're always focused on being quick (quantity over quality) then yeah, I think GPT could replace you. Otherwise, I don't know how anyone comes to this conclusion.
gpt-4 makes stupid logic errors a lot, and i agree that sometimes it's bad at nuance, inappropriately applying heuristics in a context where they're inapplicable. it's much more creative than people are, though, and people also have those same flaws
I've seen it notice logic errors in proprietary non-standard code that a human missed. It may not be able to literally "reason" through your code but it can follow the logic pretty well. I've even been able to have it pretty accurately comment the "intent" (vs. function) or spaghetti code with some reasonable accuracy.
I quite liked it, though I wasn't a fan of how bad the PR is itself, since I ended up having way too many comments, some of them being massive "wtf is this and why would you ever do it this way?" Types of things that IRL I would've refused to review without a proper rewrite.
We were already well past the point of futility before AI became part of the system. The only real way to get a job is by the equivalent of mass spamming and luck (which also means low chance of fit if hired) or by human connection and familiarity (the one true way for anything in life).
Every job I've had since grad school was through personal connections. (And luck.)
I suspect the state of SWE (and adjacent roles) for the past decade+ leads to a lot of SWEs believing that sending out a few applications will result in a great offer in a week.
I'm not sure how much AI has to do with making spray and pray less effective or enabling various cheating in interviews--presumably leading at some point to only allowing in-person interviews after an initial screen--but it probably decreases the time spent on the initial filter and you start relying more on heuristics like where they went to school.
I help immigrants for a living, and the biggest underlying challenge for recent immigrants is the lack of a network. They truly start from scratch, so for a year or two they are often completely alone.
Networking is absurdly effective, since all nodes stake their reputation every time they recommend another node. However it only helps those in the network. This leaves a lot of good candidates out. It can end up creating sort of exclusive club.
I wonder if this could be the beginning of the end for purely remote interviewing; seems like in-person interviews would be less likely to be gamed in this way.
Before the pandemic the team I was in used to conduct interviews in person. We'd give people a sheet of paper with a bunch of questions that they had to answer. No leetcode type stuff. Just basics.
With that we had one person cheat with a phone.
Doing remote interviews from then on we saw at least one person who was very probably cheating. They would get an answer, pause, then give a response, but then when we asked follow up questions they would seem not to understand their previous answer. They also had 'Audio Visual Issues' at the start of the interview. We figured they had someone else listening in on the interview and giving them answers. No AI needed.
One of the worrying things was that they did it fairly badly, had they done it well we probably could not have detected it.
Perhaps the ideal interview now would be one in person with a computer that wasn't connected to the internet where you told a candidate exactly what tools and references they would have and then gave them a coding problem.
For these jobs they were pretty heavily cloud related and also showed another probable problem, people are getting other people to get their certifications for them. If you have an AWS Associate Level cert you should be able to explain what S3 is without blinking.
I had two interviews one after another. The first interviewer asked me a basic question, and then kept drilling "ok but how does it work in more detail?". This was extremely exhausting but really refreshed my knowledge.
Then the second interviewer asked me exactly the same question. I was able to give him perfect textbook answer with lots of details because of the previous interview. I also have a tendency not to look into the camera/someone's eyes when talking, I only do this when I'm listening. The interviewer accused me of cheating and ended the interview there and then.
I’ve been on both sides of this. Occasionally mistakes on the recruiting side means two people get scheduled to do the same facet.
I’ve also been asked literally the same question multiple times. If you don’t let them know, then obviously you are going to come across like you are suspiciously over prepared.
Perhaps the questions got leaked on Glassdoor or something. Either way the interviewer is justified in rejecting you if something seems off and they can’t get a good read of your actual thought process
I worked in one "unicorn" once and all I did was getting protobufs from one API, putting it to database and taking it back from database and putting it to different API; it was so boring. I solved my boredom by contributing to some in-house framework, after which I was told I am out of my line and should go back to copying data between APIs.
Are people at Facebook actually solving these hard CS problems in daily life? Why are these interviews even a thing?
The typical day-day is always something like 'munge the data from this input stream so it can be processed by this analysis service and then uploaded to snowflake' or whatever.
Very rare you need any DSA (other than being aware of memory/performance implications of certain datastructures, etc) let alone DP.
Surely that's an attitude thing more than a true limitation? Just because the hill is steeper for you doesn't mean you can't still get to the top.
It's just very difficult. I'm not saying people shouldn't try difficult things - I did, and I do and I will. But also being realistic about your abilities and chances is important.
Now not all FAANG is the same, for example Microsoft (who isn't part of FAANG apparently?) where I live isn't that hard to get into. Hard yes but is less notorious than the others. But I'm talking in general here.
Let's say you have a 1% of getting hired. And getting 2 offers is still getting hired, right? So the odds of getting hired after N interviews is:
1 - (1 - P)^N.
(because you have to be in the "not hired" 99% bucket every time)
For 5 interviews with P=1% => 4.9%
For 10 interviews with P=1% => 9.6%
For 20 interviews => 18.2%
50 interviews => 39.5%
It takes more than 68 interviews to get to a 50% chance of getting hired. And of course, the number of interviews can be infinite and you still won't have 100% odds.
Is it fair to set the bar higher than what it was set for you to be hired into a company? Yeah, I think so.
The problem though is that leetcode challenges are shallow, and don't measure the applicant's ability to understand complex issues or algorithms. Most code is not solved once in 45 minutes, but iterated on multiple times.
"I solved a leetcode issue, so you have to solve one too if you want to be hired on" I think is the kind of gatekeeping you're talking about.
It’s why so many in the tech industry are H1B. There’s no lack of domestic talent but domestic talent doesn’t need a visa to stay in the country. So, they’ll not go through the insane process and just take a normal job that doesn’t have as insane of a hiring bar.
And if so, maybe it’s because a lot of companies are “hiring” but not really. They’ll hire a senior for a junior position but otherwise just keep waiting for the “right fit”.
No interviewing is absolutely not harder (from the perspective of the hiring company). Have been interviewing candidates or hiring manager for 25+ years at this point. It's always been hard to run a good interview process or do a good interview [1] and hard to hire good candidates no matter how good the process (because good people are rare and always in demand). It doesn't appear to me to be harder to interview now than it used to be.
If anything it's a bit easier than during the dotcom bubble for example but for different reasons. Now I get fewer candidates in total but more candidates who are at or close to the skill bar, so discernment is harder but the consequences of making a suboptimal choice are not quite as extreme (because a lot of candidates are pretty close to each other), whereas then I would be spammed with a deluge of candidates the vast majority of which were totally hopeless and extremely few of which were competent at all so the temptation to lower the bar was extremely high.
[1] ie the task is a challenging one. It requires difficult tradeoffs, and time, skill and practise and application on the part of the interviewer. There is no one-size-fits all method which works for all roles, candidates and organizations. Interview processes tend also to degrade over time so any time when you think it's going well is a local maximum and it will slide from there and you'll need to rethink, change it up etc in a month or two.
I hope for them that they they don't focus on this (j/k, we already know that some do).
Step 1 is an online HackerRank test that takes about 30 minutes. Its purpose is to filter out the really bad candidates (if you never did interviews, you can't imagine how bad the majority of candidates are).
Step 2 is a remote video interview where I go over some code that I write step by step, and then have all these weird bugs that they need to solve. I basically test for "A junior has this strange problem, can you figure out what it is?".
Practically it tests knowledge of the event loop, closures, event propagation, React peculiarities, ... .
We do remote video interviews, and sometimes I have candidates that are obviously cheating. It's always funny how chaotic and incomprehensible their explanations are. And in the end, don't even come to a solution.
I never liked homework tasks, because you can always cheat, for example your spouse might be an awesome coder.
If any experienced interviewer has more tips, I'm always happy to hear them :).
I think same thing applies here. The interview process must evolve to factor in gen ai. If a candidate is effective coding with gen-ai, does it matter that they are not effective without it ?
Several people have mentioned code reviews, +1 to that.
Presumably these questions are rooted in some kind of developer reality, they're asked to gauge technical expertise and suitability.. what's the real difference between someone with a magic box that gives them the right answers versus someone who can derive the answers themselves if the questions suitably emulate real life scenarios?
why should a company care about 'natural' problem solvers aside from the context of IP ownership and so on?
It sounds like it really just turns application questions into a voight-kampf test for no real good reason when the real point is to ascertain whether or not a candidate can get the job done.
I think this kind of stuff is just a gut reaction from a humanity that realizes that it's not the cleverness of the interview questions that are the problem, it's that machine tools are now at near human levels in the majority of mundane stuff a developer does every day -- the reactions are born from a panicked realization that it no longer makes sense to employ the lower end of the developer skill spectrum.
As an aside, I really hate the 'art' of US essay writing. The AI has now learned to emulate this equivocating waffle that sits somewhere between not wrong and not even wrong. I do wonder now - with better tools available could we instead teach people to first act as an editor for other peoples content, perhaps even AI content. In effect have the people act as the discriminator in the GAN. I think it might be a faster and more thorough way of learning that embraces the help of AI while ditching the awful equivocating waffle that is currently being taught.
Have you ever taken one of these interviews? One of the primary criticisms of the practice is how disconnected they are from the realities of the job itself, requiring studying specifically for the interview process to actually pass.
But I get it. It’s so expensive for everyone involved if you hire the wrong person and need to correct it. Asking a candidate to put more work up front seems reasonable when the offer can easily be worth $250k or more for what is essentially a day of zoom calls.
What’s a better way? Hire as a contractor and convert to full time after 6mo of performance evals? That’s also problematic.
And more recently some interns, also (but more forgivably) below that level.
It's certainly raised the minimum necessary standard beyond some humans.
They're not near the developers you want to hire. But they are better at what should be done than some people who have developer jobs.
My brain just shuts off during some interviews and I can't recall even the simplest of things. I forgot the name for ternaries in one of my interviews, despite these being things I use more or less daily.
I'm not sure what it is really. I do fine in exams/tests. I don't have any kind of anxiety on the job or otherwise. I don't even really feel anxious during interviews either, but my brain just goes poof and I can barely form a coherent sentence anymore out of the blue, never understood it.
My problem is candidates that keep being able to talk in ways that to a non-technical manager would sound as if they plausibly know their stuff, but then struggle to offer up any kind of detail when you dig into specifics, while still being articulate yet not giving me any reasons as their answers remain superficially coherent yet descends into technobabble.
I've had candidates telling me they've gone blank. That's fine - I'll find something else to ask about, dial it back, slowly circle back to the problem from another angle, and if I'm fine with everything else we'll discuss e.g. giving them a problem to solve without me there, outside the interview setting. I've had people similarly go blank in front of a whiteboard. That's fine - I ask them to forget about that, and talk through the problem instead.
The candidates I consider to be unable to code are not the ones who freeze up or go incoherent but can pass some other assessment, but the ones that are "confidently wrong" from a fairly high level, and you drill down until they can't explain a simple if statement, while often still talking with apparent confidence.
Maybe I'm sometimes wrong and they're just too anxious to admit to finding it challenging. But if so, that's a much bigger problem to me than if they're struggling with the interview setting more generally.
I might be able to pass the bar / medical exams with ChatGPT 4. And even if not, I will be able to with ChatGPT 5/6 etc. I have no knowledge at all of medicine or law would you want me as your doctor/lawyer?
The other thought is this: interview questions aim to collect evidence for skills needed for the job in a setting where you can’t observe these skills directly. I had a candidate for a senior engineering position once who clearly used an LLM. They had a vague understanding about what latency is, had no idea about SLOs, P99 or circuit breakers. The LLM then helped them to parrot this knowledge to me. But I didn’t want to hire them because of their theoretical knowledge - they need to monitor systems, act on alerting, build and improve existing monitoring and maybe join the 24x7 rotation. These are all scenarios in which it’s insufficient to know where to look for information given a keyword. The work reality doesn’t provide them always with something they can easily put into an LLM. Even if it did, progress would be too slow if for any incident they’d need to consult their llm first.
- if someone not using them is less performant than someone using them, I should definitely hire the later, because mastering the use of these tools is an important skill, like knowing your key bindings or using a proper IDE and debugger.
- it forces you to design the interview in a way that isn't trivially solved by an AI, and challenges the actual skills of the interviewees. Of course it means the interview cannot be given to brainless hiring managers or HR, but must be handled by software team leaders/managers themselves, or at least senior developers.
I have to admit I have no idea what you're complaining about. I've never treated the interviewees like they were experiment subjects and I don't see where you are reading that in “point 1”…
All I'm saying in the first bullet point is that weather you like them or not, these tools are here to stay if they improve developers productivity and it would make no sense not to take the mastery of a tool as an asset when evaluating a candidate (like if someone is showing that they are proficient with git or a debugger during the interview question, that's a good thing, if they show they are proficient with Copilot or whatever AI-based tool then it's also a valuable skill that you should have in mind when making your evaluation. That's it. There's nothing here about making people lose their time, experimenting with them or whatever.
Personal opinion.
> if they show they are proficient with Copilot or whatever AI-based tool then it's also a valuable skill that you should have in mind when making your evaluation.
Again, this is your personal opinion. I don't even consider this being a skill. Even less than "knowing IDE bindings". What is important during interviews is verifying that someone is at least able to deliver, troubleshoot and will not be a nuisance to the rest of the team (loosely speaking). Can they use AI tools effectively to speed up development fixing undesirable artifacts? Good, but not something that matter to evaluate if someone should be hired or not. Also, if you want to evaluate how they solve problems, using AI during interview will introduce a huge amount of noise.
> Personal opinion.
If those tools improve your productivity (it's highly dependent on what kind of job you're doing tbh) then it's not a matter of personal opinion: all things equal your employer will always value a more productive employee over a least productive one. Refusing for personal reasons to use a particular kind of tools that would increase your productivity is fine, but then don't be shocked if employers favor other candidates!
> What is important during interviews is verifying that someone is at least able to deliver, troubleshoot
Yes, and if his use of such a tool improves his ability to do so, then it is actually an asset!
> and will not be a nuisance to the rest of the team (loosely speaking)
Indeed, and I discussed that [0].
> Can they use AI tools effectively to speed up development fixing undesirable artifacts? Good, but not something that matter to evaluate if someone should be hired or not.
It's not the use of AI itself that's worth being hired, but the overall productivity: if he is more productive with the AI tool than you without it, I'll hire him every time (assuming his non-tech skills are at least equivalent to yours, indeed).
> Also, if you want to evaluate how they solve problems, using AI during interview will introduce a huge amount of noise.
In your job you either have problems that are out of reach of current LLM tech (very likely today, as they aren't that smart) in which case exhibiting such a problem to ask in interview should not be hard[1], or you don't and then you don't need to have employees capable of solving such problems.
Keep in mind that for most part your company and the job you're offering are very likely boring anyway, and you don't need to have a team of genius onboard. You just need decent human being that are able to cooperate, with just good enough technical abilities to solve the problem they are facing. And it's actually good if an IA can lower the technical skill cap, so that you can focus on the human part (we're clearly not there yet, as IMHO the current tech mostly improve the productivity of people solid enough to catch all the bullshit they spit out every other answer, hidden among helpful stuff).
[0] here https://news.ycombinator.com/item?id=40364095
[1] My interview was built around a fairly basic concurrent program problem, which is something we were using all the time and at the same time something ChatGPT 4 really struggles at without being heavily nudged, and when you make it correct its mistake it introduces bugs that where there before.
A surprising number of candidates would fail even the "I can use my own IDE" part of this exercise. It is still the same thing after all - show me what you have in your toolbox and how well you can use those tools.
And if you manage to solve realistic problems using ChatGPT or equivalent, then you certainly are a good enough candidate for the job, which consists of solving real world problems no matter how you do it.
But at this point it's very clear that most people using ChatGPT cannot, so it's a good sign that the candidate in themselves have the added value the company is looking for.
Then again there are other aspects to being a good candidate besides the ability to solve problems, like how you behave in a group and how you communicate, but these are also things you can see in an interview and for which an LLM isn't going to help (And god I wish it could help them on that too, because the number of antisocial people or sociopath having harmful effect on organizations is too damn high).
You should be using AI for interviews, AI for cover letters, bots to mass spam every remote job on LinkedIn (most of the jobs I have “applied” for in the past few weeks aren’t even dev jobs, but an application costs nothing so better safe than sorry), and all manner of other tools to play this game.
And that’s the problem. My action, your collective bill. Tragedy of the commons is very much an unsolved problem and the winning play has consistently been to destroy the commons through max exploitation.
Those can be rotated. I specifically use a different email every job search wave just due to the tremendous volume so I would just need a new phone number and to use my middle name or something.
But you are right, this is a risk.
Like sure, you might make it through the interview with your fifth rotated name and deepfaked voice, but then someone from accounting is going to ask for your social security number so they can add you to the payroll. And if your SS# isn’t assigned to the name you applied with… well, best case you’re fired, worst case you become very interesting to the FBI.
Genuine question.
I'm using the word "morality" to mean what is beneficial to society, within reasonable definitions. Please allow me to just hand-wave that away for now.
It's thinking about what are the ramifications of your actions on others? Why should you benefit and not others? Because you thought of it first? Because you're better at using this technique? Is this the kind of behavior that society wants to promote to achieve its goals?
We tend to use "morality" as a shortcut for meaning "not actively destructive to others." ... or something like that. I know we have to agree on which goals does society have, does society actually have goals, etc.
Or can we just let individuals pursue what they think are their own goals and hope for the best? And what best are we hoping for? Are we hoping that the system stays the same enough so that you personally will move toward your own goals? What if this pursuits prevents others from achieving theirs? What mechanisms do we have for changing things if we detect pathological behavior that will lead too far toward a place that everyone would consider to be "bad" (e.g., no food being produced).
Dog eat dog OK with everyone? What if this behavior ends up being so destructive that it affects even those who were initially excelling? What obligation do we have to others to keep the system working for them? Do we only think about that in terms of the eventual benefit to ourselves?
Morality's a big topic. I've probably mucked it up, but I'll leave it there.
I think the tools he is using is available to everyone. So it’s not like the others are at a disadvantage inherently but are choosing to be at a disadvantage. Should there be some sort of award for not using the tools? and if so, why?
I think your example about the system collapsing is dead on. But that’s proof of a bad system and not the questionable morals of the ones contributing to the collapse.
Also I would agree there is some social destructive dilema that occurs when you take shortcuts to get ahead of others. I spent my childhood growing up in low income areas of nyc, and the saying was as long as you get your piece, meaning whatever you need to do to survive is ok because at the end of the day you gotta feed yourself and your family. Think like selling illegal drugs for income. Sure you can feed your family with that money but the damage you are doing to your community will come around to affect you or your future family at some point. Where do you draw the line of bearing a responsibility to others?
I would argue that if they feel the need to sell drugs or are allowed to is a failure of the system as a whole instead of putting the blame on the person for working around the constructs of their community.
If it makes you feel any better though I'm unconvinced that mass spamming applications is actually very effective though it does carry collateral damage.
And I of course think they're an ass hole, I just think I'm general there's a pattern of scammy behavior in software that I don't really perceive in other industries. My education was not in computer science originally and one of the books I read was To Engineer is Human by Petroski. The professional standards are just higher in fields like aerospace and civil engineering. You might say that software doesn't kill people when it fails but I think that's underselling the powerful role software plays in our society. I think it's completely reasonable to push for higher ethical standards in the field even if it means some undergrads are annoyed they have to take a humanities course.
1. People who stand up for what’s right get fired and blacklisted. Perhaps killed.
2. People who get caught get second chances or never even lose their ill gotten gains. Enron fraudsters have rebuilt their wealth becoming new executives and public speakers on ethics. This is in addition to any wealth they hid away. Boeing has killed many people before the 737 max for similar sloppiness. It didn’t matter then either.
3. Low level fraud virtually never gets prosecuted, yet alone unethical behaviour. We had plenty of cases where we studied some embezzler who got fired. The penalty for ill gotten millions is being fired? That’s just the day in the life of an employee in the world of regular layoffs.
Here’s the problem with every ethics class I have taken. Who won in the end except in the most extreme cases (Madoff)? Usually the crook.