Applicants are using AI too. I've heard from people who hire/post jobs that they gets hundreds to low thousands of applications, and maybe 5% of them have any relevant experience. The problem is the breakdown of trust is costing all of us.
This happened before "AI" too. When all it takes is clicking an "apply now" button on LinkedIn some desperate people will spam any job they see.
Magicly, the spamming stopped, and we only had applications from good genuine candidates with a real interest in the role.
The job of any technology (like email and "apply now" buttons) is to make life easier and better. If it doesn't do that, then don't use it!
I recall seeing one where you had to send a specific payload to an https endpoint to apply (or it might have been an automated screen immediately after the application was submitted). Forcing potential candidates to briefly open the curl manpage seemed like a similarly elegant solution to me. I doubt it works as well in the era of LLMs though.
At this point, we think using AI and being able to use AI effectively is a skill in and of itself. When you're hired, you'll have access to AI. You'd be expected to be able to use said AI effectively.
So, we still give you a FizzBuzz. You can use AI. Even if we told you not to use AI, we know almost everyone would use AI. But you have to understand the FizzBuzz and be able to explain it to us and make changes to it "live". The amount of people that get weeded out just by having to explain the code they "coded themselves" is staggering (even pre-AI, even on a take home where you had no "OMG I suck at live coding" pressure).
You can likely control for that, if you either interview in person or via screen sharing. (Yes, it could be faked, but that's harder.)
The amount of people that can't even navigate "their own" code is astonishing. Never mind explaining what it does or making changes.
[0] The most reliable strategy I've found for that is choosing questions where the wrong answer is the right answer for some much more common question. Actually spending a few seconds and solving the problem easily lets a human pass, but an LLM with insufficient weights or training data (all of them) doesn't stand a chance.
I’ve mostly given up on all of the standard techniques for interviewing sadly, just because “using ai” makes a lot of them trivial, and have resorted to the good old fashioned interview, where I screen for drive, values and root cause seeking, and let people learn tech/frameworks/etc themselves.
But I was wondering, isn’t a take home question still good, if you give a more open ended and ambitious task, and let people vibe code the solution, review the result but ask for the prompt/session as well?
People will be doing that during normal work anyway, so why not test that directly?
One such question (obviously tailored to the role I'm hiring for) is asking whether SoA or AoS inputs will yield a faster dot-product implementation and whether the answer changes for small vs large inputs, also asking why that would be the case.
I typically offer a test with a small number of such questions since each one individually is noisy, but overall the take-home has good signal.
> why not test that directly?
The big thing is that you don't have enough time to probe everything about a candidate, especially if you're being respectful of their time and not burning too much of yours. Your goal is to maximize information gain with respect to the things you care about while minimizing any negative feelings the candidate has about your company.
I could be wrong, but vibe coding feels like another skill which is more efficient to probe indirectly. In your example, I would care about the prompt/session, mostly wouldn't care about the resulting code, and still don't think I would have enough information to judge whether they were any good. There are things I would want to test beyond the vibe coding itself.
In particular, one thing I think is important is being able to reason about code and deeply understand the tradeoffs being made. Even if vibe coding is your job and you're usually able to go straight from Claude to prod, it's detrimental (for the roles I'm looking at) to not be able to easily spot memory leaks, counter-productive OO abstractions, a lack of productive OO abstractions, a host of concurrency issues LLMs are kind of just bad at right now, and so on. My opinion is that the understanding needed to use LLMs effectively (for the code I work on) is much more expensive to develop than any prompt engineering, so I'd rather test those other things directly.
I get tons of spam that could be generated by even a basic LLM based on public information about me, but for positions that are not a reasonable fit.
Apparently, it is common for such cold calls to come from “recruiters” that are not affiliated with the hiring firm, but are trying to collect some sort of referral bounty.
I have no idea why an HR department would be dumb enough to set up such a pipeline (by actually paying for the third party “service”), but I guess once they have the program in place, they also need an LLM to screen spam applications.
"We saw your profile on github and thought you might be a suitable candidate for our open position at $CRYPTO startup.
PS you must be a US-citizen, and the job is 100% on-site"
Those things seem to be blasted out with no regard for my location - I'm not looking for a developer job anyway - but certainly not one in another country.
Spamming github users seems to be the latest growth hack, and it drives me nuts. I made all my repositories archived when I started getting hit with AI-PRs to review, but I'm reaching a point where I think my life would be easier if I just closed the account.