Have you had someone pass an interview and then later they can barely perform?
Instead the way to get past this foolishness is to ask open-ended questions expecting precise answers where the questions are themselves not precise. This presents too much variance. For example: Talk to me about the methods of your favorite Node code library. In that case the candidate has to pick, on the fly, from any of the libraries that ship with Node and immediately start talking about what like about certain methods and how they would use them.
Another example: Tell me about full duplex communication on the web. AI will immediately point you to WebSockets, but it won't explain what full duplex means in 3 words or less or why WebSockets are full duplex and other solutions aren't.
Another example: Given a single page application what things would you recommend to get full state restoration within half a second of page request? AI barfs on that one. It starts explain what state restoration is, which doesn't answer the question.
In other words AI is not a problem so long as you don't suck as an interviewer.
I used to ask a simpler (for AI) question. The candidate reads out the first sentence. By this time I'd have already established that the candidate is not genuine. Our interview process let's ride out the interview as a courtesy and to also try to extract something out of the candidate that the company could use.
Anyway once they read out the first sentence from the AI with utmost sincerity, I'd follow-up with deeper questions into the topic at hand. 99% failed to answer the second question well. The ones who let me ask the third and fourth questions are devs who still have their original thinking hats in place but just use AI out of nervousness or who generally don't interview well. Those we'd explore further and suggest for lesser roles/alternate streams etc. This all my experience and others MMV.
More to the point: Those are all things that you could go from being unequipped to answer, to answering well quickly from research. So, it sounds like these questions would select for people who have used these technologies and techniques, vice for good developers.
Maybe that is why AI barfs on it. But, yes, performance is something users universally care very much about even if many businesses aren't willing to accept that, so the question remains highly relevant. Another way a good interviewer, who has a background in performance, to consider this is to start asking performance questions that a candidate could answer from AI and the asking the candidate why some performance techniques work better than others and by how much. Then you can really see if they have a high confidence AI answer that they cannot quality.
At any point, AI answers aren't a challenge for people who are confidence enough in the craft to ask more qualified questions and determine honesty from the resulting nonverbal signals.
Deepseek gave me a good answer to that.
I think they're good enough at this point to fool interviewers up to at least the senior level. Especially for these sorts of questions that encourage vaguish rambling answers about general knowledge.
If the job is cutting trees, you can't measure them by how long they take to cut a tree, but whether they have stamina to cut through multiple trees.
Take home assignments work, and the good news is they can be shorter now. 1 day or 4 hours of work is enough of a benchmark. Something like a Wordle clone is about the right level of complexity.
Things we look for:
1. Do they use a library? Some people are a bit egoistic about doing things the easy way. GenAI will make a list of words, which is both wasteful and incomplete when they can find a dictionary of words. Do they cut down the dictionary to the right size? It should only be the words not definitions.
2. Architecture? What do they normally use? How do the parts link to one another? How do they handle errors?
3. Do they bring in something new? AI will usually use a 5 year old tech stack unless you give it a specific one, because that's around the average of code it's trained on. If they're experienced enough to tell AI to use the new tech, they're probably experienced enough.
4. Require a starting commit (probably gitignore) and ask them to add reasonable sized commits. Classic coding should look a bit like painting. Vibe coding looks like sculpting, where you chip bits off. This will also catch more critical cheating, like someone else doing the work on their behalf - the commits may be from the wrong emails or you'll see massive commits where nothing gets chipped off.
5. There are going to be people who think AI is a nuisance. Tests like this will help you benchmark the different factions. But don't give them so much toil that it puts the AI users at a large advantage and don't give overly complex "solved" questions that the AI can just pull out from training.
It's similar across all mobile platforms. People call it MVVM, Bloc, MVP, etc. But we want to see the pattern of UI-repo-service and unidirectional data flow. This is three layers and it can be as little as 3 files. If someone can grasp that, it's good enough.
There's not a lot of ways you can screw up hiring mobile devs; most skills are trainable. But this is what costs months. One guy once looped the viewmodel inside another viewmodel because he was using them as the data store and observable. He was let go but it took 2 years to remove everything he did from production even though he worked there for 3 months and most of it was via rewriting entire blocks of code.
We watch for overengineering and perfectionism. Some people insist on more blocks or splitting it into modules. But can they do that in 4 hours?
AI can write all of it easy enough, but it's just a 2-3x multiplier. Some assume it's an infinite multiplier and they can just subscribe to Cursor on the spot and it'll be done in a blink. But do the parts connect in the order given above?
Also Wordle is not ideal for architecture, but it's good enough. It requires a solid understanding of the top level of things, and connection to the data layer. What if we switched the word source from the dictionary to the API? That's a bit more of an advanced test.
An even more advanced test would be structuring it for TDD, which would need understanding of DI, factories, tests and gotchas, etc. I don't think this is doable even with AI yet though.
As a UI test, Wordle is pretty good because you have all your logic and variables on one layer and display it via a different, more complex layer.
Before AI, we'd give template code with the tech stack we're using and expect them to just modify the code. With AI, we let people be more creative and push for things they want to use, as long as they can keep the train on the tracks.
Not everyone gets everything done, but it's a good way to benchmark candidates against each other - someone who gets a third of it done might be worth it if they're taking 1/4 the salary compared to someone who does everything on time.
This is the same type of test as the takehome tests that a lot of companies give, where you can use google or stack overflow previously to do research.
If I was hiring for my own project and needed people that can problem solve, I would be asking more involved questions that LLMs could not solve, because LLMS will give you just the standard most commonly used solution.
For example an analogy to another industry, LLMs can't tell you how to design a mountain bike correctly at a level of detail that matters, because there is no guide online that tells you how to do this.
After working for 15 years at FAANG, startups and no-name companies, I have conducted more than a 100 interviews for all levels, mostly in engineering.
In my experience with technical interviews specifically, there are 4 types of candidates:
- the memorizer - the mathematical brain - the project builder - the coding enthusiast that knows a language well but can’t do algorithms
Most of the time, I have encountered candidates in buckets 3 and 4. Many showed debugging skills, communication skills, but lacked the right answer to trapping rainwater.
I was told to reject candidates who couldn’t pass these problems mainly on the grounds of solved or not solved, even if they clearly communicated.
The only reason I got into FAANG companies myself is because I was the good memorizer. I couldn’t solve most of these problems today without months of prep.
At the end, I left my FAANG job recently because I realized it’s going to be the same or worse at any other company, because internally it’s all the same B.S. once you make it in. Sure, you’ll get a fat salary, but it’s a slow grind.
Instead, I chose to build things.
Is the interview process today bad? I think so.
What can candidates do about it? Take it in their own hands and play the game or get played.
What is the alternative? Depends. There are many companies that don’t do crazy algorithm interviews and pay really well. In my opinion, if you are hiring hundreds of engineers per year, take-home assessments and reviewing them is literally a job on its own. Making candidates build apps from scratch and deploy it, test it, present it? Maybe, but certainly opinionated, not binary.
New grads don’t have experience, but experienced engineers do. You shouldn’t need to ask people with 10 years of experience about trapping rainwater from LC. There are so many other things to ask, discuss, and gauge experience, scope and depth.
If someone asked during such a session (where it's cameras, screenshare) that they wanted to do something like google some documentation, I wouldn't see that as a problem. Obviously it's a problem if someone just googles for a solution and pastes it in.
I see LLMs in the same way. No issue with them using it do something like take the pseudo code they wrote in front of me and turned it into an implementation. Especially if they could talk through this code and make suggestions about further changes and so on, clearly showing they understand what's going on.
The real concern is going to be when sophisticated agents can impersonate (clone voice and video) in a convincing way, as well as the capabilities to see the screen and type away as if it was a real person, and they're responding to you in real time.
If the software is based on the models made by large companies, they'll be happy to give you recipes. They would refuse if the coding request mentioned something about cracking passwords or dumping credit cards. And all of them will have a meltdown if you try to ask them to say something politically incorrect (what a bizarre world that would be if that became the new captcha system for humans trying to figure out if they're wasting their time with a fake human).
That said, this is going to be a cat and mouse game. There will be nothing to stop people from fine tuning models to get around being "jailbreaked" to reveal themselves as LLMs. Perhaps the best means is taking the time to research problems that causes "vibe coding" to completely fall down. And that is likely going to be things that are novel and haven't been littered all over the internet. That has a knock on effect of making such interviews a bit more interesting for the people doing them too.
Oh well. Maybe I just don’t understand the game.
But because of the quality of this message board I've always assumed that Y Combinator made wiser investments and didn't just throw money at whoever said the magic buzzwords of the day. Guess not.
It's a fun experiment how large you can grow an AI system before it falls apart because nobody knows how it works and it's too complex for the AI to grasp, but I imagine this breaking point will increase as AI gets better.
As a potential customer, this is like buying from a private equity owned business. You know you are buying a heap of shit, but hopefully it's cheap (for a while)
Your onboarding will be making collection calls.If somebody can figure out a way to pass the interview with AI they can probably figure out how to do the job. If they can’t, they get fired. Some people who pass the interview without AI end up getting fired, too.
I don’t think there’s anything unethical about using AI to pass an interview.
Like I used to do small writing samples from most candidates. That’s gone now and for client facing roles, I replaced it with mock meetings. I send over prep material + meeting goal and then let the candidate lead a 20 minute meeting with me playing the client. At the end we talk about how it went.
I see candidates who clearly used AI to prep and kill it and then candidates who did little prep (AI or otherwise) and try to wing it.
There are mock interviews and mock meeting services that would really help you prepare.
What you are suggesting is using ai to learn to drive when you should be getting in the car.
No it isn’t.
> For someone who has done 100+ meetings I'm not sure I see the advantages
When you’ve tried using AI to assist in your meeting prep what has your experience been like?
Assuming a job description / interviewer explicitly prohibits the use of AI, then using it "in stealth" represents a basic lack of integrity. And I guess it needs to be explicitly stated, but this is orthogonal to how well they would actually perform in the actual job.
How would this be ethically any different than a student who sneaks in an LLM using a tiny camera and wireless BT headset during an exam?
It’s a risk using AI in a job interview situation like you describe and if I thought the risk was worth the reward I might do it.
I mean maybe there’s some very small number of classified/protected data situations (not just “these are our sales numbers and are private”) where I think maybe the integrity component actually matters.
But if it’s just some random job that doesn’t like people using AI for some bogus reason, give me a break. They can make the rules as easily as people can break them.
The school thing isn’t that different except for the risk and the reward. The risk is higher the closer you are to graduation and the reward isn’t that great since colleges are so easy these days that you almost certainly don’t need to do anything against the rules to graduate.
People are going to break rules.
There are rules you would break that I wouldn’t break. There are rules I would break that you wouldn’t break. There are rules other people would break that you and I wouldn’t break.
I decide the rules that I’m willing to break and don’t particularly care what other people are willing to break. I can’t control them. I can control me.
A lot of people breaking a rule might make me more willing to break it myself or it might not. Depends on the situation and the rule.
I’m not sure what point you’re trying to make?
your classic whiteboard - manual way, using physical from or the whiteboarding web tools.
you can discuss design patterns, how they would solve issues you've ran into in production and compare approaches.
it seems as if software 'engineers' are the only snowflakes that have to go through this ritual.