An invisible desktop application that will help you pass technical interviews | Hacker News Reader

An invisible desktop application that will help you pass technical interviews

github.com

211 pointschurlee12249 comments

Open articleView on HN

Great job on further eroding the trust from a prospective employer.

Require a formal degree in CS? That's gatekeeping.

Need to pass a whiteboard exam? Not representative of the actual work.

Live coding session? Biased against people who don't perform well under pressure.

Take home project? It's too much work to do for free.

Showcase a personal portfolio? Not fair to people with families or other obligations.

Either you enforce a minimal level of competency upfront in the form of an academic degree, industry-standard exams such as a PE Exam, etc. OR you push the entire responsibility of vetting prospective applicants downstream to the employer—which is exactly why interviews are multiple week long gauntlets.

The tech world likes to complain about all of this but other occupations 100% DO have high standards - it's just that it's paid up-front.

Want to become a lawyer? - You've got to pass the LSAT, get into law school, and pass the bar.

Want to become a doctor? - You've got to pass the MCAT, get into medical school, and do residency.

Want to become a pilot? - You've got to get your PPL, pass your check ride, do your IFR, multi-engine, commercial rating, ATP

God there are some days that I ABSOLUTELY HATE THIS INDUSTRY.

If your hiring process is broken by people using AI then the process is either not evaluating the right things, or maybe it's fine because realistically speaking you're hiring a dev who's going to be using AI anyway.

I work in infra. I don’t want an incident to be the time that I find out my coworkers don’t actually know how anything works, and are frantically posting questions to ChatGPT.

That said, I also don’t know how to adequately test, other than to carefully watch people’s eyes during questions; even that isn’t foolproof.

"tell me about potential issues while doing X", then dig into answers? We may get there one day with LLMs, but currently it would be extremely hard to keep the conversation context and process the longer answers while talking to someone. The voice recognition quality and generating speed would be an issue too if you wanted to monitor the whole conversation.

Currently the cheating mostly aims for coding tasks because they're pretty simple to do and self-contained. Much less for helping with a good dialog.

Some of the best tests come from taking something mundane and twisting it just a little off the beaten path in a way that LLMs cannot handle. The usual example is the river crossing puzzle which has a standard format and solution but can be modified to make that solution incorrect. For code, an example would be parsing log lines to compile insights but switching it up so that the timestamp means something other than the time the log was collected, etc.

You can completely change the complexity of a bog standard leetcode question with the slightest of modifications, and I’m certain no extant LLM will catch such a nuance if your problem is a very common one.

A fun exercise is to ask chatGPT what the river crossing goat puzzle is and then say “do the same puzzle but with two goats and a cabbage please”. It can’t get it right even with multiple corrections. Might stop working at some point now that I posted this though!

Interview performance used to be correlated to job aptitude, but no longer.

Unfortunately real life problems involve a lot of context and introducing that to interviews is hard. It either requires a long conversation where someone might lose track, or involve writing a large ish code base and getting them to work on that (which is a lot of work, and once again they don’t have context).

In my experience if a problem simple enough to be completed in an hour, it’s simple enough for AI.

IMO if your interview question has a right answer that's like, a basic early screening tool to see if the person is paying the slightest bit of attention. The sorts of questions that really give you useful insight into who someone is as an engineer are the ones that don't have a right answer, and can't be reasonably solved in an interview. If your question has an achievable correct answer you're clipping the gamut of info you can glean from it.

I recently had a question where I was just given a bunch of tuples and told "let me know anything you can glean about this data in two hours". I think you learn much more by seeing how someone fails at a complex nebulous task than whether someone can implement a doubly linked list from memory or whatever.

Real life programming is like 80% failing at complex nebulous tasks, picking yourself up and trying again (by volume). Interviews should simulate that. If AI helps you, so be it.

Interviews seek to estimate long-term job performance. Since they cannot measure this directly, they measure attributes that correlate with performance. If you interfere with these measurements (e.g. through fraud), you break those correlations. This doesn't mean the interviews are measuring the wrong thing, or that people who cheat on the interviews will meet the performance bar in the actual job.

If someone uses AI to defeat a CAPTCHA, does that mean the CAPTCHA was poorly designed because "the bot is just as valid a user as any human"? Of course not - the CAPTCHA's entire purpose was to establish that correlation between "can solve this" and "is human," and breaking that correlation through automated tools defeats its purpose entirely.

First off, yes, actually that's exactly what it means. If bots can solve CAPTCHAS then the CAPTCHAS are poorly designed, that's the whole point.

I don't think this sort of CAPTCHA type step should have a significant place in a good interview process in the first place, if it is important to you to put your applicants through this sort of gauntlet you're not treating them with the dignity they deserve.

If you're just looking to fill cogs in your machine, sure, but in that case you're just as likely to get kids who studied a bunch of interview questions but also don't know shit. I'd just as soon have someone who's good at ChatGPT.

Let me put it a different, perhaps simpler way:

Let's agree on the premise that good human engineers are much better hires than good AI users. If your hiring process fails to distinguish between good human engineers and good AI users that's evidence that your process is not producing good data.

In the exact same way that if your CAPTCHA is getting solved by bots it's a bad CAPTCHA.

Right, so the problem with the interview process isn't that it tests the wrong skills, but that it fails to prevent the secret use of AI such that a human can appear to possess those skills when they do not, in fact, possess them.

It seems there are at least two different perspectives:

A) If a human can use AI to pass a remote interview that they would otherwise fail, then the questions are unsuitable for use in interviews, whether remote or in person.

B) If a human can use AI to pass a remote interview that they would fail if it were face to face, then the questions may be reasonable, but the safeguards for remote interviews are insufficient.

I disagree with A.

The types of questions that an AI gets right are still bad questions to ask in 2024, regardless of being remote or in person.

If you're allowing your programmers to use AI on the job you're not getting particularly useful information if unaided someone fails a question that they would answer correctly with AI assistance. Sure, being able to answer the question in person may still correlate with being a better applicant, but that doesn't mean it's a good question.

Interviews take time, you should make good use of the time by asking questions that tell you a lot. Questions easily answered by AI (this also goes for questions easily answered by studying interview questions by the way) are not good questions. Measure the things that matter.

   (this also goes for questions easily answered by studying interview questions by the way

I agree, but it’s easier said than done.

> they measure attributes that correlate with performance. If you interfere with these measurements (e.g. through fraud), you break those correlations

Not necessarily... You just get the people that are technically competent enough and are self-corruptable enough to pass the test. There is still a correlation with performance but it is a weaker signal and you are more likely to miss out on people with equal ability but higher morals.

I find this all rather intriguing because to me it says more about the employers/interviewers than the applicants.

Over the years I've interviewed many people for technical positions and there are many ways of getting to the core of a prospective employee's competence.

Of course there are questions about training, past experience, qualifications and those about the work you expect them to perform but other matters also enter the equation and they're almost as important (the person's demeanor, confidence, etc.). By letting the person do most of the talking—explaining things about themselves and their work—you can learn a great deal.

Stating the obvious, interviewers ought to be on top of the job they're hiring for but from this story and from other anecdotal info I know that's not always so. In my case as head of the department I was pretty much up to speed on most aspects of the work, so it was clear to me when an applicant was spinning me BS. If they knew more than I did then they almost certainly got the job (life's easier when you've smart employees, for starters one can ask them questions instead of vice versa).

What's that leading up to? Well, when interviewing it was obvious to me (and to the other interviewers) within just the first few minutes whether the person was worth considering. It's uncanny how good one can become at doing this (and my record speaks for itself, I was pretty pleased with those employees I hired).

Let me illustrate with an example: some decades ago I had an applicant apply for an electronics job. He spoke only Chinese and a few broken words in English and none of us on the interview panel spoke Chinese. BTW, he turned up with a Chinese/English dictionary in hand.

I dispelled with many of the questions I usually asked and showed him a somewhat complex circuit diagram with many subsections that performed very different functions (if I showed this at all then it was usually towards the end of an interview).

Within seconds of looking at the diagram he progressed logically through the processes from beginning to the end (note the processes/stages were not drawn in linear progression and consisted of many branches). He did this mainly by pointing.

It was immediately obvious to both my independent technical assessor and me that this guy knew his subject very well. The only point was a minor objection from the nontechnical employment officer who questioned if we'd manage to communicate adequately with someone who didn't speak English. I didn't see that as a significant issue.

He turned out to be an excellent employee who knew his work very well. And within six months his English was good enough not to be an issue.

In summary, if AI is a problem now then it will be more so in the future and that needs to be tackled. That said, I find it hard to accept that an experienced interviewer can't quickly cut to the core even when an interviewee puts up smoke screens and distracting noise.

Like, Popper's falsifiability it doesn't take much to knock a BS or inconsistent argument off its pedestal when one has enough data, refutability usually works well.

That said, one must always keep in mind that just because an applicant can get some things wrong it doesn't mean that he/she isn't the best person for the job. All that tells me is that I have to probe a little deeper.

Edit: asking awkward or trick questions usually ought to be avoided. An interviewee is usually already under considerable stress and can easily be flummoxed when asked them (even if they know the answers they can bind up and not spell them out).

Usually, it's immediately obvious when an interviewee doesn't know an answer to a question or is struggling with it because of stress—body language alone usually makes that abundantly clear. It's then best to gently segue to another question. If the question is really relevant to the job then one can return to it at the end of the interview when the person is more at ease. By then, it's pretty clear to the panel whether the applicant is in the running or not. If not, then do not return to the question.

There’s an argument to be made that by the time your business runs out of vetted candidates to hire you’re already staffed at some of the highest levels by corporate parasites who got there by gaming the system, and you can tolerate adding more elsewhere.

The pushback is precisely because paying the high standards up front pulls up the ladder that made tech such an attractive option to people without traditional academic skills who managed to be computer whizzes.

Take home projects and personal portfolios are the options that are the best. Those are the ones where candidates are given the time to show their best work and don't require the debt of a college degree in a specific major (whose knowledge is probably largely irrelevant to your day-to-day coding).

I prefer our current system over the industry standards exam system you seem to be advocating for. It’s more flexible. Different companies care about different things. Different employees have different strengths. It all works out.

I'm a little confused about your reasoning that take home projects are too much work to do for free. Aren't lawyers and doctors expected to spend a ton of time studying and taking the licensing exams without being paid (I assume even having to pay for it)? My gut feeling is that's more work than a take home project.

Licensing exam scores are usable for applying generally to all places. A take home I do for company A isn't useful in applying at company B.

You're complaining about a process you've implemented yourself. Have you tried hiring in a completely different manner?

At what point do you question an approach so easily gamed, or any derived metric?

> Biased against people who don't perform well under pressure.

I think performance under pressure is a virtue. An important job is never going to be entirely free from stress - you want somebody reliable who will take that in their stride, you don't want someone who will buckle when things get difficult.

I'm all for in-person live collaborative problem solving, be it on a whiteboard or just a discussion. You see first-hand exactly what the person has in their memory, their problem solving techniques, how quickly they get things, their communication skills, and yes, their ability to work under pressure. If done correctly and collaboratively, you get a dry-run of a real design meeting and have some feeling of what it would be like to work with the person.

> I think performance under pressure is a virtue. An important job is never going to be entirely free from stress - you want somebody reliable who will take that in their stride, you don't want someone who will buckle when things get difficult.

Some people experience extraordinary high levels of stress in an interview setting. From your comment, I suspect that you are not one of them and might have trouble imagining what these levels of stress feel like.

Fair enough. I don't want to be mean, but I wonder, if they are the type of person to experience this in interviews, what other situations would set them off in a similar way? It won't be everything, but I'll bet there's some comorbidity; there is a combination of stressors in an interview setting that triggers them, where else can these occur?

From my experience, there is more overlap with people who generally let stress get to them, than people who are solely stressed out by interviews specifically and nothing else.

I have a relative like this. They struggle in interviews due to stress, they also struggle in exams due to stress. That's two things, I don't want to be surprised by the third at a critical and stressful time in my team.

> Require a formal degree in CS? That's gatekeeping.

Selecting people for a job is literally gatekeeping. The company has the equivalent of a gate (or perhaps an actual one), and most people are kept out of it by the fact that they are not employees or authorized visitors.

No part of would-be employee screening can be faulted for being gatekeeping; i.e. being that which it is.

Shut the fuck up. We'll stop cheating when these corps stop making us solve binary search algorithms to interview for front end React roles. Probably 80% of all technical interviews I've had were completely irrelevant to the actual daily work. They engineered this shitty game, we're just playing the game. No one's making you play this game.

Live coding session? Biased against people who don't perform well under pressure.

No, it's biased against people who perform perfectly well (sometimes exceptionally well) under real, actual pressure. But who do not take kindly to the bullshit / theatrical pressure typical of the standard interview context, these days. And to companies that just can't tell the difference (or are under the delusion that performance in the latter category is in some way predictive of performance in the former).

Take home project? It's too much work to do for free.

It's more that the companies simply abuse the process in various callous and careless ways -- either by assigning projects with inadequate specification and/or unknown goalposts (or plainly unrealistic expectations of the actual time required, given these ambiguities); or simply assigning scores of these assignments (often automatically to every candidate who applies) when in fact they have no intention of even looking at the vast majority of the submissions.

Easily ameliorated by simply paying people for their time.

Feels like in some sections of the US like big tech, the talent vetting process has evolved into some adversarial battle of attrition where the idea is to grind down the pool of candidates until only a few are left rather than find some potential new colleagues. Tools like this have emerged as a direct response to that process.

What a bizarre way to begin a working relationship.

Sadly, these sorts of “practical interviews” aren’t just limited to Big Tech. There’s been an arms race for thirty years now between applicants and employers, the latter creating new hurdles to overcome and the former finding new ways to bypass unnecessary hurdles.

Employers demanded applications instead of resumes, so candidates used copy machines. Then they wanted block lettering, so candidates used type writers. ATS systems could search for keywords, so candidates flooded their resumes with them. Then employers allowed online applications, and the floodgates opened proper - applications from all over the world, desperate for good employment. So they made you create an account, and applicants wrote or used bots to automate it - which became site features like “Quick Apply” on LinkedIn and Indeed. Practical interviews used to be a single round for only the most senior roles, but then employers wanted everyone to be as qualified as those roles but not pay them that much, which started the cheating arms race.

Ultimately, the problem remains a glut of qualified talent demanding salaries to pay the bills, and employers who want to pay as little as possible. The two sides are adversaries by default, and few companies do the work of mending the relationship into something more cooperative.

this does NOT match my experience as an interviewer. Our company went through piles of candidates who were not qualified and it took us months to find a good fit after going through hundreds of resumes and scores of interviews.

I hope the folks we hired are paid enough to retain because there were was absolutely no glut of talent. It was hard to find the folks we wound up hiring. There was a glut of applications, yes, but the vast majority were totally unqualified.

So you're confusing your specific requirements with my generalization of "talent". Depending on your specifics, there might not be a glut of talent in that specific field or expertise, but there's still a surplus of domestic college grads unable to find work in their chosen fields, and companies who refuse to pay domestic graduates what they're worth when they can outsource to contractors, MSPs, or offshore work entirely for less.

My perspective on this issue is that the reason there's an abundance of candidates in any given application pool is because of a mismatch of expectations between employees and employers, even today. Employers have cultivated an environment where candidates must job hop frequently to grow their earnings, because internal advancement has dried up and employers haven't respected loyalty in decades. This results in a plurality of the workforce "always looking for work", because they see no potential in their current role for growth or advancement; in fact, the whole reason they applied to your role is because they wanted that growth or advancement, and now they have it, so it's time to apply elsewhere for the next step up.

I get that this environment totally sucks for both parties, but it's important to note that employers are the only ones who can change the system, since they hold the bulk of the power in individual negotiations. Unfortunately, the current attitude seems to be more of the status quo: more technology to "make it easier to find the best candidates" (thus removing humans from even more of the hiring process), forcing existing workers out (thus increasing the applicant pool), and then bemoaning about their problems on Business News ("too many unqualified applicants") or forums.

Applicants didn't create this mess, and we can't fix it.

> we can’t fix it

Applicants don’t have to apply to Google or Facebook for jobs they’re not qualified for, and if they stopped, the mess you’re referring to would disappear. At least at Google and Facebook’s doors, anyway. It might move somewhere else simply because there are a lot of people (literally everyone) looking for jobs that pay well.

It’s funny to tell someone on the other side of the interview table that they’re confused, when they’re giving you information that can help you. Knowing that there is an abundance of low quality applications is useful information. Mismatch of expectations might be an accurate generalization, but arguing the expectations of applicants isn’t part of the problem is not accurate at all. Applicants definitely aim high and spam companies with resumes for jobs they’re not qualified for. I even tell new grads to aim a little higher than the exact job description, but within reason. To be fair, a lot of people don’t know what’s reasonable and don’t know how competitive their cohort is and they have little to lose by taking their chances.

The lore about having to job-hop has always existed, it was true 30 years ago, and it’s still a story people tell today. Sometimes it works, and sometimes it doesn’t. Don’t get confused by hearing stories. Neither the number of stories you hear, nor the story-teller’s confidence in their take, proves anything about how the business world works. You need more first-hand knowledge of how many people don’t have to job hop. Companies do promote people, and some companies do respect loyalty. Instead of insisting it’s a fight and wallowing, go find the good jobs at good companies working for good managers and figure out how to stand out from the applicants who are less qualified than you.

> Applicants don’t have to apply to Google or Facebook for jobs they’re not qualified for, and if they stopped, the mess you’re referring to would disappear.

Counter-point: employers don't have to pay garbage wages while demanding FAANG-quality candidates, thus focusing the talent pool on the few companies that do pay wages that might help candidates reach into the bottom of the Middle Class.

> It might move somewhere else simply because there are a lot of people (literally everyone) looking for jobs that pay well.

That is the (unspoken) point I've been trying to make, without being so overt about it. People need more money. Prices keep going up, productivity has skyrocketed, but wages have stubbornly refused to keep historical pace in the long view. The need for ever-growing sums of money just to survive is a key driver in the constant glut of applicants for jobs, one that is exacerbated by the staunch refusal of businesses to pay appropriate wages outside of the tech sector.

> It’s funny to tell someone on the other side of the interview table that they’re confused, when they’re giving you information that can help you.

They're really not, though. Employers whine about a lack of qualified domestic candidates as justification for outsourcing jobs or importing migrants on precarious visas, but then also refuse to engage with local Higher Education institutions to "properly train" the candidates they need or contribute to on-the-job training. As the age-old HR koan goes: The CEO asks HR, "What if we pay for job training and they leave?", while HR replies, "What if we don't pay for training and they stay?" There is no zero-sum game being played, here, and the actions of employers do not align with their verbal complaints about the state of job applications and the labor market. They're complaining the basement is flooded while leaving the firehose turned on.

> The lore about having to job-hop has always existed, it was true 30 years ago, and it’s still a story people tell today.

Yeah, nah. This is a recent trend, and a very recent narrative. In the 2000s and early 2010s, I was still being actively penalized for the appearance of job hopping, in the form of lower salaries and limited upward mobility. That really only began changing in the 2014-2016 window, when suddenly it became more acceptable as most millennials and Xers hadn't exactly had stable careers like our predecessors. In less than a decade, that shift from "job hopping=bad" to "job hopping=acceptable" accelerated to "job hopping=required if you want to survive and thrive". That attitude only came about because the cost of everything kept going up, and up, and up, while our wages and upward mobility stagnated if we stayed put. Loyalty hasn't been rewarded since the 80s, but especially in a post-2008 world, it's been a hindrance to individual success and is solely the fault of employers, not employees.

> You need more first-hand knowledge of how many people don’t have to job hop.

I have plenty of first-hand knowledge from my own career, and have second-hand knowledge from the careers of my parents' circles compared to the careers of my senior mentors (Xers), colleagues, and juniors. Add in the data curated by all the other groups trying to sell their own narrative, data gathered by governments, and just casual observations of both domestic and global employment trends, and this isn't exactly a difficult conclusion to reach by anyone reasonably well-informed and suitably objective about it.

> Instead of insisting it’s a fight and wallowing, go find the good jobs at good companies working for good managers and figure out how to stand out from the applicants who are less qualified than you.

It is a fight, though. A fight for everyone to be able to make rent, pay bills, and have a decent quality of life on a full-time job. That's the fight. Such a lifestyle has increasingly been constrained solely to those in Big Tech, especially in urban areas, and it's why there's such a fierce resistance to arbitrary RTO mandates and a willfulness to cheat on exams. Your retorts to me essentially boil down to the age-old propaganda of, "Workers should know their place and follow the rules for their own good," which only works when the rules ensure their basic needs are met. That is not the case, as can be evidenced from even a casual glance at current global events. Populism doesn't arise from peace, love, and good times for all, it arises from a broken system that doesn't satisfy the needs of the masses. That is why hiring is broken, in a nutshell.

A good leader understands the system they operate within, its flaws as well as its strengths, and what is or isn't within their ability to fix or improve upon. Individual managers and employers simply cannot fix what amounts to a broken system rooted in short-termism. Sure, working with recruiters can ease the burden, but that also comes at a substantial cost. The point is that the usual refrains in self-help books or "visionary thought leaders" don't work when the system itself is at fault, and that's very much the case here. It's not possible to eliminate cheating solely through discipline; rather, the system must be amended such that there's little to no incentive to cheat in the first place. Right now, there's every incentive to cheat the system when it could lift you out of poverty or give you a substantially improved quality of life, and that's why it's on the rise professionally today, just as it was in academia not a decade ago.

Desperation is the root cause of a multitude of negative outcomes. To reduce those outcomes, you must reduce the amount of desperation in a society.

Trying to excuse cheating by pretending it's some kind of economic equality issue and badmouthing hiring of immigrant tech workers is a bad look. Software developers themselves don't want to stuck be working alongside any of the -10x coders who cheat their way past interviews and generally are grateful when they are let go so this can't be passed off as some kind of "sticking it to the man" / "power to the workers" thing.

It's often not clear what qualifications for a role really are - I think most of us have accepted roles where we had to learn "required" things on the job. Or where one of the requirements was a joke, liking knowing how to use some random tool or software (internal wiki, expensing system, etc.) that anyone could pick up in 15 minutes. Especially in tech. I've personally gotten jobs in languages I didn't know yet a couple times.

I do agree that for any given job a large chunk of applicants are people who have no business applying (IE barely capable coding themselves out of a paper bag or didn't read the job description), but there's always a long tail of plausible applicants if the job isn't a trash posting.

Our company went through piles of candidates who were not qualified and it took us months to find a good fit after going through hundreds of resumes and scores of interviews.

Except you don't always simply know they weren't qualified. Only that they didn't pass your filter.

I know, I know: In many cases, it's reasonably clear the candidate simply isn't even ballpark qualified. But when we get to "maybe" territory, the recall-precision gap, and hence room for false negatives, is also quite large.

As an employer, I know candidates use LinkedIn’s “Apply in one click” in bulk. When we contact them, half of them notice they don’t even fit the criteria in our ad (be a Java developer, for starters).

And while I sympathize with your frustration, workers are caught between companies who can't even be bothered to send a rejection form letter anymore, companies posting "ghost jobs", and companies who post such broad salary ranges (if any at all) that they're unusable. Then you roll in the "resume farms" abroad (shady recruiters submitting candidates without approval and who don't match the requirements, so they can net a payday), and it's a nightmare for both sides (workers and employers).

Where my sympathy dissipates, though, is the knowledge that workers are merely responding to the ever-increasing demands of employers. It's ultimately up to the employers to fix this mess, because they're the ones who created it in the first place. It might mean an in-person career event to weed out remote candidates, if you're adamant about a local hire, with humans reviewing resumes in-hand, in real time, absent ATS and AI systems getting in the way. It might mean forgoing practical interviews entirely, in favor of better verbal interviews that assess skills and competencies, with hypothetical questions AI can't quickly or reliably answer.

And it also might mean rebuilding company loyalty and incentivizing people to stay instead of leave. It might mean reducing the amount of jobs you're posting externally by resuming promoting internally again. It might mean dangling carrots that keep workers from jumping ship every two years to grow wages in a meaningful way.

Point is, there's a lot of ways to address this, but the current attitude is "MOAR TECHNOLOGY", which doesn't seem to address either side's core grievances around the hiring and employment processes.

How do employers fix being blasted with resumes from unqualified morons?

A lot use recruiters to reach out to talent, that is how I got my current job. I applied directly and got ghosted but a month later a 3rd party recruiter reached out to me and I got the same job I applied for. Lots of other jobs have funnels through referrals and networks, people can make recommendations of past colleagues.

The only way to stop people from blasting resumes is to make it cost them something, for example charge $5 — which goes to charity — to apply for the job.

For coding interviews, the company could make an application form that gives you three simple coding exercises, the submitted code is automatically tested, and when the tests pass you are asked to upload your CV. When you come to the company, you get two more coding exercises of the same difficulty, if you pass at least one of them, you can talk to a human. (This last step is to turn away the candidates who asked someone else to do the exercises for them.)

> The only way to stop people from blasting resumes is to make it cost them something, for example charge $5 — which goes to charity — to apply for the job.

That's just more short-termism. It ignores the desperation of the masses for the Middle Class lifestyle they've been promised, in favor of yet-another-hurdle to be jumped over for a chance at a job they need to survive. All you'll end up doing is encouraging further fraud with such a proposal.

So what can we try instead? Well for starters, we can focus on reducing the need to seek external applicants in the first place. While labor mobility is critical to a healthy workforce, it's also worth noting that most workers would rather have stability over mobility, all else being equal. People want to work to live, not live to work - and having a reliable job, with a reliable wage that keeps pace with inflation, that enables them to buy a home and pay the bills and maybe take a vacation every year, will keep them from seeking other opportunities elsewhere, thus reducing the overall applicant pool for new positions. Workers are surprisingly fine with their excess value going to Capital, provided they're not having to decide between paying for insulin or rent this month because they haven't had a raise in three years.

In addition to decreasing applicant pools by improving job stability and longevity, we can also stop placing arbitrary requirements irrelevant to the position at hand. This sounds counter-intuitive - if you remove the degree requirement from a role, you'll have more applicants - but the reality is that oftentimes workers will pass up roles not requiring a degree because they'll perceive it as being more "junior". This isn't a great idea for every role or business, obviously, and YMMV, but it can be a different way of thinning the herd.

Then there's the tried, the true, the oft-reliable:

> A lot use recruiters to reach out to talent, that is how I got my current job.

Look, in case my comment history doesn't make this glaringly obvious, I'm not great at the whole social networking thing. Recruiters connect highly-effective introverts like myself, with employers who don't want to rely solely on word-of-mouth referrals (especially in an era where said referrals are openly traded or sold on forums like Blind). All of my big career moves came through external recruiters, of whom I'm profoundly grateful to for opportunities I wouldn't have had otherwise. While I'm generally against outsourcing as a practice, the fact is that a good recruiting firm will save your teams time they can use on meaningful work, and only present candidates to you they feel will meet your needs. Whether or not the employer will trust the recruiter is a topic for another day (I've lost count of the number of times my recruiter, myself, and the customer's PoC all thought I was perfect, but a Director somewhere didn't want to trust outsiders), but it's still arguably one of the better investments a company can make in talent acquisition.

> > A lot use recruiters to reach out to talent, that is how I got my current job.

As an employer, I’m flummoxed that a recruiter with a human fibre is required. It means instead of paying higher salary, 20% goes to the pocket of the intermediary the first year.

> All you'll end up doing is encouraging further fraud with such a proposal

How do you plan to defraud the banking system when you need to pay $5 to send in your application?

The issue is that it takes almost zero effort to apply for a job so there needs to be proof of work on the applicants side. The easiest proof of work in through our banking system because there is massive amounts of effort to prevent fraud and ensure interoperability.

Alternatively that proof of work could be that you need to physically drop off your resume like in the pre-internet days.

The same need to go for the company side, they need to have skin in the game too when interviewing a candidate. This is why I am against automated tests which are used at places like Amazon.

So if an employer posts a ghost job, does that become wire fraud?

And if the employer posts a real job but the candidate doesn’t get a response, or is immediately rejected, will the employer get accused of wire fraud?

Sounds like a nightmare for employers.

> It ignores the desperation of the masses for the Middle Class lifestyle they’ve been promised

Oh are we talking about minimum wage jobs, labor & the proletariat now? I thought we were talking about software engineering, where the average salary is very comfortably in the 6 figure range in the US. If you’d rather be a plumber or a teacher, go for it, I hear the interviews are shorter. ;)

Why would charging a small fee for applications encourage fraud? I don’t see the logic. The problem you complained about is the low-effort spam applications, like LinkedIn’s 1-click apply-all button. If that was no longer free, it would certainly reduce the spam and make applicants think more strategically about their budgets.

Companies already use recruiters. Recruiters are expensive and many provide low quality leads, partly because of the way recruiting is incentivized, and partly because good software engineers don’t have to move around that much.

Recruiters are the tried-and-true method of thinning the herd effectively. A good recruiter relationship might cost your company some money, but it's likely to be an order of magnitude less than the cost of ATS systems and AI resume scanners. Recruiters have always been the best path forward, as they serve as valuable "social grease" between high quality applicants who aren't extroverted enough for the network game, and employers who need help digging through reams of documents.

That said, I suppose I should amend my conclusion: no single employer can fix this either, as this problem is the collective result of businesses forcing shorter cycles of tenure in general; short-termism, in other words. Your company can make it easier on yourselves by retaining a great recruiting firm or retaining good hires for longer, but you'll still be inundated with a glut of paperwork until and unless other firms do the same thing in large enough quantities to affect real change.

If everyone is struggling to keep their heads above water, those dwindling few manning the lifeboats are undoubtedly going to get swarmed.

> Recruiters are the tried-and-true method of thinning the herd effectively. A good recruiter relationship might cost your company some money

I know the price: Instead of an ad on LinkedIn (100€ per recruit in 2019), we now pay recruiters 20% of the yearly salary.

In France the “salaire brut” is 60k€ → We pay 12k€ before an employee starts working, excluding the time WE spend in interviews. That’s a 12.000% increase over a simple ad.

> no single employer can fix this either

Yes, thank you for diagnosing the problem! For me, it’s due to LinkedIn. They know perfectly well that bulk-apply-in-1-click is hell. They could fix it in one day.

Employers used to promote internally, train and up-skill their people, and strive to keep them around for longer than the two to five year stint that seems to be the average in tech

Hiring low-level employees requires less vetting. Upskilling for more senior roles within your own company means you get to skip things like cultural fit (because they presumably already are), work history (it's in the employers own records), and references (that would be the coworkers and managers they already have).

Pay enough to keep the ones worth keeping (and increase it with time, to match inflation at minimum), upskill employees, and promote from within, and this problem should pretty well fix itself

This is all fully explained by salaries going up, and the pool of young applicants getting bigger faster than the number of jobs available, while the pool of highly qualified applicants remains small. Jobs at Facebook, Google, Microsoft, Intel, Nvidia, are all legitimately competitive, and because more people are applying, those companies have the freedom to be choosier and the incentive to compete for the good ones. Because salaries have gone up (or even simply the perception of salaries), and because salaries are high compared to other jobs, more and more bright-eyed kids are hoping to punch above their weight in interviews, which is why companies are flooded with low-quality applications and don’t have a choice about needing to sift them efficiently.

This is the natural state of a competitive environment, and interviews will get “better” for candidates when there are more jobs than good candidates. But be careful what you wish for; if the job market gets less competitive for candidates, salaries and benefits and working conditions go with it.

While there might be some truth it to, framing it as an arms race is a mentality that will not help the average candidate, and might do them harm. Employers always have and always will want to pay as little as possible, obviously, that’s the definition of a company. But they do actually compete with each other for good talent, and they do pay a lot relative to other jobs, and some companies’ definitions of pay as little as possible is still to pay enough that employees are well compensated and don’t leave.

I have a theory that job search sights have an incentive to make the application/recruiting process so much of a slog due to (1) the hundreds of applications many people have to send out and (2) the hundreds of applications that companies have to sort through, as a way to justify their own existence.

key insight. the factors influencing this are a couple I can think of. the first is just the sheer volume of applicants made possible by the internet and the mobility of workers, where you get hundreds of profiles to sort through for any job and use a coarse technical funnel to sort them. the second is we can't just pick good colleagues because the regulatory and litigation environment imposes selection criteria where we have to evaluate all these applicants equally.

these aren't justifications for the confrontational interview filter, but some upstream factors that are worth picking at to see if something better could become a solution to them.

I've recently been studying leetcode (for reasons) and went through an interview process.

It was, by far, the most boring interview I've ever had to participate in. The leetcode problem was easy (detect if two strings are anagrams; solved with O(n) solution in < 10 LOC; then aggregate sets of anagrams). System design was easy because it is to the point that it is largely "formulaic". It feels like those rhythm games where you need to know when the trigger is coming and the pattern to hit those triggers (talk about functional requirements -> talk about non-functional requirements -> write some API routes -> draw a load balancer (because of course, there's always a load balancer) -> draw a service -> draw Redis/Memcached -> draw a database). It's comically laughable how it's just like a rhythm game at this point.

What I don't understand is why teams don't just "get to the point". A lot of times, the assessment is looking for specific notes. Why not just ask about those notes directly? A friend recently went through an interview where he was docked for not using C# `yield` when implementing an `IEnumerable`. But had the interviewer asked "What is `yield` and why would you use it?", the interviewer would have gained the same exact insight in 15 seconds with one question.

I think what's also striking is how slow orgs are to adapt their interview process to account for AI. There's a distinct lack of focus on code reviews as part of the assessment. If the future of coding is AIs for productivity, then teams should be adding code reviews into their process (check for correctness, check for defects, check for performance, check for security, etc).

| distinct lack of focus on code reviews as part of the assessment

"Writing code is easy, reading code is hard"

An interview based on reading, explaining and critiquing complicated code would give a better window into a candidate's skills than having them solve the kind of coding challenge that can be completed in a short interview.

Also, being able to identify which algorithms would be useful in solving a unique problem and explain what the pros & cons are would go further than implementing an algorithm abstractly.

That was my favorite interview ever.

The guy gave me 30 lines of code, and asked questions about it. Including finding the questionable constructs in it.

It was also my favorite interview format as well.

Interviewer had a small API. Found missing error handling, missing validations, poor choice of ID column type, missing index for a high volume read query, possible XSS vector.

Loved the format so much I ended up making a small FOSS tool for code review as interview: https://coderev.app

This looks great, I might use it next time I'm interviewing.

I tried this and found that there are a set of potentially good candidates who simply cannot bring themselves to criticize anything in front of an interviewer due to the power dynamics of the interviewer/interviewee relationship.

i would argue that isn’t a good trait in an employee either

Those are the same people who will know the project is heading off a cliff and not tell you before it is too late.

You have to set them up for it.

    "Imagine that this code was written by a junior dev; review it and tell me how it can be improved, where the problems are, and other feedback".

    "We just hired a new intern and he pushed this PR.  I need you to help me review this code for issues and make sure it's production ready".

It’s my personal interview approach, I have a some 60 lines of code example I will ask questions about. What does it do, if I wanted to change x, where would I need to do this, etc. Seniors will take 30 seconds for the first two questions. I‘ve had junior applicants where I had to stop after 15 minutes of trying.

I think this format is very conducive to leveling because there are some tells that are really obvious. Like input validation, error handling, database indices, database column types, race conditions, etc. You can't fake these easily, but with experience, they surface quickly.

sounds like you interviewed for too easy a job, the anagram question you described would be significantly easier than interviews i have or give

It was exceedingly easy but it was for probably one of the highest level IC positions at a large, public BNPL. The entire technical part of the interview could only be described as "boring" and un-engaging.

from collections import Counter

if not Counter(str1) - Counter(str2): # success, they are anagrams

  str1 = "a"
  str2 = "ab"

Good catch.

Counter(str1) - Counter(str2) | Counter(str2) - Counter(str1)

That ought to work. I'd hope Python's caching doesn't recompute the Counter for long strings.

As somebody that gives systems design interviews most weeks, you’d be surprised how many candidates don’t put the api gateway but magically go from the TCP load balancer to the services inside the cluster, or how basically nobody is ever able to explain how a CDN works (as in, put the CDN behind the API gateway, note I have seen this in a company in production too). Or how some candidates will have redis and Postgres and do magical cross storage transactions.

And these are all candidates with 5+ years experience, sometimes 10+. I hate the leetcode gauntlet, but I also am really surprised how much worse candidates are now on average that they used to be (I’ve been around for about 25 years now), and how the minute you try to dig a little deeper to see if they know how things really work, the more it seems that they have no understanding whatsoever of the basics and somehow treat computers as magical black boxes.

I do understand I am privileged by having grown with the technology, starting in the 8-bit era, through an electronic engineering degree, and now in k8s etc. but with so much educational material available for free you’d think that even people in their 20s/30s now should be able to have a better idea of how things work.

I think a big part of the problem is that if you search for most tech subjects, you're more likely to find terrible medium and geeksforgeeks articles written by amateurs than something real written by someone knowledgeable.

The move away from on-the-job training has played a role. If a company doesn't provide any early guidance and mentorship (which everyone could use regardless of industry experience), then that has to be replaced by finding the perfect candidate who can immediately be productive.

> then that has to be replaced by finding the perfect candidate who can immediately be productive.

We're called freelance consultants. And we charge a high price for that service.

yes and companies basically are just paying that high price with a salary

The ideal purpose of hiring is to find a skilled candidate who will be a good fit and maybe even bring talent and energy to the role. The hiring entity will ideally provide mentoring and training to ensure a successful investment and relationship.

But the catch is... the people doing the hiring must themselves ideally be talented, energetic, skillful mentors who envisage building a positive culture.

Literally all the engineers at the company went through the same process. For all the bitching and complaining an interviewee does, there’s probably a hundred people who just went through the process and got hired.

Suck it up, it’s part of the job. If you can’t do it, you don’t belong. Simple as that.

> who just went through the process and got hired.

You say that as if there's The One True Interview Routine™ and it doesn't change from team to team, person to person, month to month. I actually would bet that no single organization interviews with the consistency you're describing, not even FAANG with their holier-than-thou grinders

If it wasn't such a monster drain on human productivity, I think it'd be funny if there was "rolling interviews" where each new hire got to send prior hires though "the process" on the yearly anniversary of those prior hires

This is called “hazing”.

Do you want a job or do you want to live unhazed?

That's actually a good point. I believe that BigTech these days look for conformists, not superstar SWEs. If you agree to waste your time on preparation to meaningless LeetCode style interview then you are one of them and ready to be a part of the company. I do believe that is actually one of the things that being tested by such an interview, it's not a proxy for IQ, it's a proxy for candidate's readiness to eat dull corporate BS and not complain about it.

People would rather get paid $80k a year and be a non-conformist than get paid $200k a year to conform to the requirements of a job. Simple as that. Problem is those non-conformists keep applying to the $200k jobs expecting the $80k treatment.

On one hand, that’s what it always was, by definition - a company is looking for the best candidates it can get with a given budget, and the best-case scenario for the company is there’s a large pool of candidates to choose from.

On the other hand, from my perspective and experience on both sides of the interview table, to me it seems like the perception that this is adversarial is coming primarily from young candidates whinging on the internet in large numbers about interviews being unfair or “broken” or doing it the wrong way. Every time this comes up on HN, there are hordes of comments demanding we stop whiteboarding in interviews, or claiming anything not exactly in the job description is out of bounds, or decrying the abuse that coding live inflicts. There are blog posts every week about how to fix the problem, where the problem is usually the interviewee’s comfort, and not the company’s efficiency or effectiveness.

The truth is that companies need people, and the employees there do want new colleagues. If you truly understand that, and also that the selection process is unavoidable and working as designed, then it will help you navigate the selection process and appeal to the company, it will help you get the job.

I’m well aware of the fears young candidates have, since I was one. Getting a job can be hard, and it might even be getting harder over time. But to me it’s also funny that software engineers looking for 6-figure salary jobs complain about interviews and competition and having to demo their skills for an hour or two. Don’t forget to look around at other industries and ask whether it’s really better over there. People in music and films and advertising and art have to grind for years, suffer low-wage or historically unpaid internships. Doctors have to do residency, grad school, and get licensed. Most people make less money, many people have to do labor. Most jobs have much bigger downsides than what we have to do in our interviews as software engineers.

We have opportunity and privilege and with the right practice and attitude and with reasonable expectations, it’s really not that hard to do well in interviews and land a good job. Many kids are hearing stories of FAANG and million dollar salaries and expecting to somehow land their first job out of school in one of the top 5 most coveted and competitive companies. I know this because I just spent a day at a university career fair a few weeks ago talking to hundreds of graduating seniors, most of whom are all expecting to shoot the moon for the same 3 jobs, and most of whom are not very eager to consider the idea of starting at a smaller company and working their way up for a few years. FAANG feels adversarial only because it’s actually legitimately super competitive, but as a candidate, figuring out how to have a non-adversarial approach is the key to getting the job. Tools like this are the opposite of what they need.

I’m reasonably certain that I interviewed someone using this or something like it in the last few days.

Lots of eye scanning while looking above the window they’d been typing into. Pauses. Big pastes. One of my excellent colleagues noticed that the candidate made use of exciting C++ casts before ever defining the variable’s types. Complete inability to explain or debug the code just written.

So. Frustrating in two dimensions. First a waste of time for everyone. Second, occasional signs of real ability make me think the candidate might’ve made it work honestly. The fool.

This, and this alone, makes me pine for in-person interviews. But I suspect those won’t be back for some time. (For good reasons that are out of scope here.)

It sounds like you want sympathy but it just shows that some people just don't understand the power dynamics of an interview.

I believe live coding is a scourge and we should get rid of it altogether.

Get a tiny open ended project, which requires more design/architecture skills than actually finishing the solution. You do it at your own time and shouldn't take more than 1-2h. You submit it and provide your mental process in writing.

We then hop on a call and we can dicuss it further, talking about possible issues and what to do to get it released if a requirement changed last minute.

If you can do that, come across amicable, knowledgeable and experienced enough given the seniority of the role, and feel like you are not reading from an hidden ai window then you pass.

"Exciting casts" are absolutely worthless in real life, everyone has to balance shipping vs code artistry.

100%

Another one I like was having to code review a repo which had specific issues in it.

Then add some new features.

Then in the final stage of the interview you go through it.

The core issue here is the sheer volume of applicants. Microsoft opened 30 new-grad software engineering positions. Care to guess how many applications they got within 24 hours? 1,000? 10,000?

Nope.

100,000 applications. In under a single day.

With that kind of applicant pool, I’m honestly not sure what the best approach is—even though, in a perfect world, your suggestion would be the more appropriate route. The reality, however, is that these numbers are just absurd.

I think at that point it's a lottery anyway, pick 50 truly randomly and try those?

“We don’t want to hire unlucky people.”

> With that kind of applicant pool, I’m honestly not sure what the best approach is

You should see the number of people that apply to college.

At least in the USA one major piece of the college application is a standardized test - a concept very similar to a Leetcode interview.

A standardized test is considerably more standardized than a leetcode interview though. Not to mention you take it once and then provide the scores to everyone. My process with taking the cpa exam a single time was a much better experience than random leetcode problems every time.

The problem is attempting to optimize a noisy process. Instinctively people think they need to look at all the applicants.

Start interviewing people, and once you have enough people who pass your bar, you tell the rest tough luck.

Secretary problem.

That, to some degree, is guaranteed to happen naturally. But why not use a sieve approach and narrow the pool exponentially using a range of tools that goes from automatic and instantaneous at the start to manual and time-intensive interviews at the end? The potential upside for a company is relatively large - with a small amount of work they have the opportunity to get the approximately best 30 people out of 100k graduating students. From the company’s perspective, this isn’t a problem at all, it’s a massive windfall of an opportunity. They can afford to optimize it, and they’re motivated to. As a hiring manager or company founder, I’d love to have this “problem”.

That said, your comment reminds me of a Monte Carlo algorithm I think I’ve heard about. There is a way to have some statistical confidence in getting the top K out of a sample of N without examining all N samples. I’m blanking on what it’s called, and I think it’s related to Reservoir Sampling. I don’t know if I read this or am making it up, but my instinct is that you can get to high levels of confidence after looking at sqrt(N) samples.

The problem is noise.

Say you want to take 10% at each round. You have to do this three times to get 100 people, and then another round to get the top 30.

If you tighten the criterion, noise matters more, so you will drop your actual best candidates, since people need a high skill+noise. But if you try to keep the unlucky "good" candidates by having a wider band, you pay more.

Also, if you are successful at finding the best candidates in a batch, noise matters even more the next round.

I've been thinking about how to vizualize this. I have it in my mind and I'll try to describe it.

You plot a bunch of points with S, N for each candidate. They are independent, so a scatter plot looks like a 2D gaussian. Lets say skill is vertical, and noise is horizontal.

You want to find the 30 highest points, but you aren't allowed to just look on the scatter plot and select the top points. You have to draw a line S + N = c, and choose the person with the highest c. Basically sliding a ruler out that crosses the S and N axes as far away from the origin as possible.

Observation: if N is high (wider distribution) compared to S, you just get the luckiest people. If S is high compared to N, you just get the most skillful.

Noise is a reality and a potential problem in ranking candidates for sure, 100%. Of course it’s worse than just measurement noise; the noise in hiring is subjective and situational and human. The sampling method doesn’t fix it, using the Secretary Problem approach doesn’t give you less noise, it gives you more noise in the result. The benefit of stopping early is that it reduces the cost of sampling, but if the company doesn’t care about the cost of sampling, there is no “problem” there to solve.

If you want to reduce noise, the way to do it is to have more independent measurements (interviewers), not to stop interviewing early.

The good news is that there’s no such actual thing as “best” in this situation and people have many dimensions, they can’t be ranked perfectly, but they can still be ranked approximately. We also don’t need to get the exact 30 people out of 100k people with infinite precision, we will get an amazing set if we can take a random sample of the top 1000 candidates out of 100k candidates. Having 100k candidates gives us the opportunity to end up with a selection from the top 1%, say, whereas if the number of applications was 40 people for 30 jobs, you might be stuck accepting people who are below average.

I don't see how your method works better than a sieve with noise.

Instead of a "noisy" sieve, you're just using the time of application as your sieve. Which, unless your job opening values the skill of submitting lightning fast job applications, is pretty much 100% random noise.

You say a sieve approach drops the best candidates. Your approach also drops the best candidates.

And if the bar is actually so high that only the best candidate(s) qualify for the job, then your approach would imply you interview an expected half of the applicants before you fill the positions.

It's pretty obvious a "noisy" sieve (with some signal) is better than sieving by application time or sorting by arbitrary order and taking the top N applications. You don't have a magical solution. The people handling those 100000 applications are not that stupid.

The difference is that you don't need to wait for 100k people to show up.

If you're already reasonably confident from the first batch of people what the potential in the pool is, why would you wait? People have work to do and they want those people in as early as possible.

I'm not saying you will do much better sieving by time, in terms of quality. But you will save all the effort of looking at the people at the back of the queue, for little loss.

Sure, half the best people will be at the back, but what is your loss on taking the second best people earlier? Chances are you won't even be able to tell.

If the problem is having to wait for applicants, then you’re right; stopping early will help. In my experience, companies already do what you’re suggesting simply by having a deadline or time window for filling a job. Even if a lot of companies would love it, most job posts don’t get over three thousand applicants per seat available.

In this particular case, MS didn’t have to wait for people to show up, they got 100k candidates before they could have even made a decision on a small subset. And in general, the problem being discussed in this thread isn’t having to wait, it’s how to deal the flood of applications coming in too fast.

* huge edit to this comment after this rattled around my head a little more: actually, duh, sieve and early stopping are completely orthogonal, this is not an either-or situation, and you are probably assuming a sieve in your proposed approach. You can sieve 1 candidate at a time, since it’s a series of criteria ranked in order of how much time each one takes to evaluate. If someone doesn’t meet minimum requirements, they’re rejected quickly and not invited to interview. If you’re going to stop early but you have a lot of candidates to evaluate, then you still have to sieve. Like, I’m pretty sure you’re not proposing to conduct 37,000 interviews for 100k candidates, right? Even if we stop before looking at all candidates, each candidate will have some early criteria used to cull them, and only the ones that pass all the early criteria are invited for an interview. That’s true whether or not we stop early. The sieve is a given and unavoidable. The only question is whether to stop early, which we would only do if it solves a problem. We can, if we want, interview the same number of candidates either way. The sieve allows you to look at all candidates efficiently, but does not require it. If you do look at all candidates, then the more people in the top of the sieve, the statistically better the final interview pool will be. Stopping early is valuable only if the applications process is slow (which is not the case here) or if the early criteria take significant time to evaluate.

Wow, 30 for all of Microsoft... they probably lose 30 people a month just from attrition!

> Get a tiny open ended project, which requires more design/architecture skills than actually finishing the solution. You do it at your own time and shouldn't take more than 1-2h. You submit it and provide your mental process in writing.

I used to really like the idea of take homes, then I had kids and discovered just how hard it is to find time to do these (particularly for multiple applications while working).

Don't get me wrong, I think that they're generally a better test for the actual role, but they're not perfect, and they're game-able by spending way more time on them.

Interviewing is hard. Like, ideally, we'd just centralise the administration of these tests and you'd do a few every year (so formal CPD rather than an expectation that people do it in their off time) but that is never gonna happen in the tech industry for various good and bad reasons.

> I believe live coding is a scourge and we should get rid of it altogether.

I would take an hour of live coding any time over a take-home task that is one hour only on paper and takes you three in reality.

Engineering is collaboration, and it's fair to expect candidates to talk, solve problems, and explain their solutions.

Where we could do better as an industry is the type of problems we give to people. I'm not a fan of LeetCode-style questions, especially when multiple ones are asked. Something closer to Earth would be better. But even if you have to ask something algorithmical, I'd prefer the style of "Advent of Code" exercises where the same problem has variations and multiple levels of complexity.

> But even if you have to ask something algorithmical, I'd prefer the style of "Advent of Code" exercises where the same problem has variations and multiple levels of complexity.

That's how Amazon does their interviews.

It only takes you three hours if you don't pay someone else to write it for you and provide you with a written brief about how it works, which is what I suspect some candidates must do.

Your comment is excessively dismissive. The irony is that parent actually did what you’re suggesting, tried to talk about the interviewee’s solution and found that they didn’t come across as knowledgeable and experienced enough even though their code suggested otherwise. Whether it’s live coding or take-home is mostly irrelevant.

Your proposed alternative is absolutely subject to the same problem, AI-written code and LLM-written mental process. Think more carefully about whether it solves anything and exactly what problem it solves, because it does not solve the problem of people trying to game interviews with AI, your suggestion is potentially worse on that axis. The whole point of doing it live is to get to the talking part that you suggested more quickly.

Wouldn’t you rather be discussing your thinking immediately and get a sense for what the real questions are and how you’re doing, and have the opportunity to talk about it, than waste several hours doing it at home trying to write something up without getting clues along the way or being able to ask questions, only to have the next guy get hired because they used AI, or because they’re fine with live coding, or because they’re better at writing than coding, or because they got lucky?

You have to live code on the job. Do take a moment to consider what the hiring manager needs and whether take-home problems are either more effective or more efficient for the company. Managers want to find the best candidates, and best involves more than coding, it involves communication and collaboration skills too. Live coding can give some indicators of those things, and take-home problems cannot.

I did that for my previous job and I loved it. Had plenty of time to get over my nerves and then build something cool. And then brag about my choices in writing.

Some people want to be hired based on 25 minutes of easy live coding. It’s ok that you’re not one of them but don’t piss in someone else’s cereal.

> made use of exciting C++ casts before ever defining the variable’s types

an example of this, please?

> Lots of eye scanning while looking above the window they’d been typing into.

Are you expecting people to not reference external material, such as documentation or syntax, and only code from memory?

Absolutely not. In my standard intro material I explicitly say otherwise, tell them that actual work involves checking external references, and that I want the interview to be as near to a collaborative experience as it can be in the odd circumstances.

I also tell them that they should tell me when they're checking external resources (the CPP reference is what I mostly have in mind, but StackOverflow or whatever is fine with me) so that I understand how they go about solving problems. Because that's the actual important bit. All the coding and the questioning is a proxy for that.

Why bother with this style of interview where the candidate is under extreme time pressure and isn’t allowed to use any of the research tools they would normally use in their job? What is this controlling for?

It barely made sense when these were conducted in person, and it’s completely inane over video meeting.

Instead of grilling people, talk to them. Give them meaningful take-home exercises that you can discuss together during the interview. If they can explain the decisions they made and discuss alternatives and trade-offs, that’s a much better indicator of job performance than pretending to invent a leetcode party trick algorithm in 10 minutes.

I agree but a big problem remains. How do you make it scale for 100k applications?

Filter people out by requiring them to count the number of s's in sensational.

Employing someone is renting their labor.

How do you find an apartment rental in a large city with thousands of options?

You can’t expect to visit them all. And you certainly can’t come up with increasingly bizarre torture tests for landlords to pass (“I might rent from you if you can answer enough trick questions about tenant law within 15 minutes”).

To find the apartment, you probably sample enough of the options to form a picture of what’s available, and then make a decision knowing that you didn’t have perfect information but you had to move ahead. The same applies to hiring.

Except landlords have almost all the power, just like the job gatekeepers. A landlord gets to choose who rents their property so they aren’t under any obligation to answer your questions, they’ll just find a renter who doesn’t ask questions. When you have all the power it inevitably gets abused which is exactly what we’ve been seeing in rental with things like real page.

That’s kind of my implicit point. Employers don’t want to think of themselves as renters. It’s offensive to their unquestioned self-image as being the ones who hold the cards and deign to interview candidates.

But if you can give up on those worthless power games, you can have a hiring process that’s better for both parties.

How many jobs have you got? 1000?

How about you hire the first 1000 people who seem competent, and then tell everyone else you've filled the roles?

If you have literally hundreds of applicants and don’t have a process you are confidant provides a strong scoring function you should just randomly hire 1000.

I let candidates use the Internet (except AI tools) during coding interviews. All I care is if given a problem can they do the necessary research to solve it. It usually involves making them use a library which they are not already familiar with. This is more than enough for most of the programming jobs.

I think this is a continuum:

I let candidates use X during coding interviews.

I've personally seen this evolve over the past 20 years

Whiteboard --> Text editor --> IDE --> ...?

Maybe "AI Coding Assistant" is next.

Well yeah, it is really bad, which is why I have no qualms with this project.

If a company wants me to do some leetcode crap I decline the job.

Live interviews control for wasting the candidate time in a way that take home exercises don't.

Both the candidate and the interviewer need to take time out of their day for the interview. If it's an in-person interview, the candidate also takes travel time etc, which could be more disruptive.

But it takes almost no time to give a take home problem, so there's potential for a significant assymetry in time spent between candidate and employer.

Huh? I've done take-home problems and they've been fine. I'll take it 100% of the time over live coding.

Sure, if *you* did them they might have been fine. One really important thing the interview needs to screen for is the qualifications of the person actually trying to be hired, and not their buddy doing the work instead.

I am really glad I don't live in your world where interviewing is such a profoundly low-trust activity. I feel sorry for interviewers where that's a legitimate concern.

I mean, dude, take a look at what we were originally discussing -- an app on github that helps you cheat on live interviews in Zoom.

FWIW, I've interviewed almost a hundred candidates in the past couple years and I think the number of candidates that I thought might be cheating are in the 3-5% range. It's fairly low, but if the take-home was the main thing that determined whether the candidate was hired, I wouldn't take that 3% chance.

Just in case this repo isn't a joke and some job seekers are seriously thinking of this tactic...

Companies already know about cheat methods like this and candidates still have to demonstrate skills in-person on a whiteboard in a 2nd round of tech interviews.

A high-paying FAANG job isn't going to hire candidates based on just one remote Zoom tech interview.

That still may not deter some folks and they'll try to continue to find some cheating technique to use in front of the whiteboard. In that case, some creativity is going to be required. E.g. : https://www.google.com/search?q=chess+cheating+anal+beads

The issue is that "chess board moves" is very low bandwidth and can be efficiently encoded into pseudo-quasi- Morse Code vibration schemes. However, if you're asked to "invert a binary tree" or any other open-ended random puzzles, it's more difficult to compress a secret 2-way high-bandwidth communication with ChatGPT into a hidden butt plug. (But then again, if a candidate is actually able to fool people with hidden ChatGPT hacks in a face-to-face interview, maybe FAANG should hire them based on that alone.)

Keep on the lookout for a github repo with ChatGPT butt-plug communication.

looking at the despair at https://www.reddit.com/r/cscareerquestions i am completely certain people are not above cheating

even if its not an app, everything is so cheap and small now, 1mm camera and microphone, and very tiny ear piece (invisible with longer hair), or even earless headphones, realtime whisper turbo and groq llama3.3 can absolutely pass most interviews and answer most questions

even without the tech, i am sure people are now cheating on face to face interviews as well

i worked at fairly big company, and did maybe 500 interviews, i never knew how the candidate would look in advance, so i dont see why cant someone else shows up to the interview, pass it and then the real person shows up for the job

The problem today is not that to get into FAANG you've got to jump through hoops and loops like a well-trained cage animal (aka fresh graduate) but that this is getting applied more and more to literally any startup under the sun, even some with 0 funding.

As a first engineer for a tiny startup, show me some past projects and let's grab a (virtual) coffee talking tech, that's all.

Now it's like McDonald's requesting to perform Michelin level dishes under 45 minutes with Gordon Ramsey in the room while still ending up flipping burgers minimum wage at the end of it all.

If this keeps going and getting worse, I wouldn't mind a butt-plug communication, as long as I can pick my favourite AI (I personally kind of prefer Claude right now over ChatGPT.)

> As a first engineer for a tiny startup, show me some

How do you expect startups to make money if you think they aren’t even legitimate to see you program during an interview?

Oh, right, you don’t care about your employer’s finances.

They're probably just confident in their ability to mentor. A lot of jobs just aren't that hard. Many people struggle to accept this about their own line of work because they think it undermines their intelligence. Find smart people (or middling with a great work ethic) that are easy to work with. Iterate.

“Flipping burgers…”. This made me laugh because it is so true. My last involvement in interviewing candidates was for a position for an ETL developer. One of the other team members who was part of the interviewer schedule proudly announced that he had discovered a great source of tricky SQL puzzles that he intended to use. My comments were along the lines of “you do you, but I’m more interested in how someone would generally approach the 3 or 4 categories of tasks they will be routinely doing day-to-day”.

The in-person whiteboard interviews isn’t always a thing for all SDE rounds, but def there for SDE-2 and above I guess. Still, it’s just a matter of time before this is also solved.

> A high-paying FAANG job isn't going to hire candidates based on just one remote Zoom tech interview.

What do you think they do? One round of interviews was enough for my Google recruiter to congratulate me on getting a job there.

Did I get the job? No. But the process didn't include any other interviews. There's only one round.

>What do you think they do? One round of interviews was enough for my Google recruiter to congratulate me on getting a job there.

Your Google interview process might be different because you've previously said you were ex-FAANG from 2000s so your work history and street credibility puts you in the "experienced senior hire" bucket. The interview process may be different for that type of candidate.

My recent college grad friend's process went like this (2022 so post-COVID):

- an offsite remote interview over Google Meet using Google Docs as coding canvas. 45-minutes long

- 5 onsite interviews on same day: 4 technical and 1 behavioral

Pardon me, but I don't understand. Was the recruiter mistaken about you getting the job?

Yes. Here's the full process I went through:

1. The recruiter sent me email suggesting I reapply to Google. She asked for my resume and put me on track to apply for an entry-level position.

2. I had a phone screen. (1 interview, over the phone, graded to a lower standard.)

3. The recruiter informed me I passed the phone screen and scheduled the round of interviews. (5 interviews, back to back over one day, over some kind of internet platform.)

4. I had the round of interviews.

5. The recruiter informed me that I had passed, excitedly wished me congratulations, told me to expect a job offer within six weeks, and told me that I'd be having some "team fit" interviews over the intervening period.

6. Weeks of radio silence.

7. When I inquired what was going on, she asked me if I could provide an altered resume explaining my employment gaps.

8. More weeks of silence.

9. She called me and informed me that although I had passed the interview process, my interview scores were not high enough to qualify for a job.

As to the question of whether there might be more than one round of interviews, I complained about this treatment to several people who worked at Google, and I've been told by multiple of them that the recruiter was correct about the process and there are no further rounds.

FAANG jobs absolutely hire engineers over zoom. It’s not just one interview, but the whole process is done remotely over video. It typically includes somewhere between 5 to 10 sessions covering coding, architecture and behavioral.

Source - work at a FANG which engineers exclusively via remote video call.

Is the option of in-person available to the candidate?, zoom interviews are exhausting

I haven't worked FAANG, but a Fortune 100 tech company who's been hiring a ton of ex-Amazon management. Interviews 100% remote, even though there's an office in Chicago. No option.

I received an offer from Apple in July after performing six remote interviews. Did this change since July?

> A high-paying FAANG job isn't going to hire candidates based on just one remote Zoom tech interview.

Have FAANGs gone back to real onsites? I've only see virtual onsites since the pandemic.

I empathize with new grads. It can be hard to stand out especially at the beginning of your career.

But I predict this won’t do what the authors claim: it won’t fix anything, and it won’t change technical interviews that much. At most it might increase the number of companies that require remote candidates to fly in and interview in person to do the same practical coding problems they used to do online. Which would be the standard from just a few years ago, anyway.

Practical coding problems (also known as work simulation interviews) are the most effective and least biased tools available for evaluating candidates. Not every hiring manager does a good job using these tools, but if you see misuse, take it as a sign that the manager might not be good to work for.

I have trouble understanding the authors thinking here that anything would be worth the cost of this: it’s a trust-busting tool where people are deceiving their potential employer. Trust is foundational to everything, in work and in life. We need more trust in our society, not less.

Someone beat my lazy ass to it XD

Had something in the works when I first got to know about multimodal GPT and electron made it super simple to prototype something. Refined it a bit to even do the invisible window thing after feedback from my peers. Didn’t have the ahem courage to do it.

Also recently noticed there are actual companies around this space who are marketing this off as “interview assistance”. Two of them I’m aware of:

1. https://www.senseicopilot.com/

2. https://www.finalroundai.com/

——

I’m sure the knee-jerk reaction from the tech industry to these tools would be:

1. Return to in-person interviews

2. Force install spyware/vanguard like crap to scan your whole personal computer before doing the interview

If you are a tech leader/senior engineer with influence on these matters, I *HOPE*, that this is not the path you take. It’s high time FAANG and others stop gatekeeping with these useless rounds that are equivalent to forcing people to hand-solve complex math equations instead of using calculators. I also partly blame the YT influencers who keep peddling this BS but they will stop if the companies themselves take a stance.

You can rather use the same 1 hour slots to have a conversation, pair program on a realistic problem required for the role you’re hiring for where you and the interviewee can both benefit from the knowledge transfer. I’ve tried this in many of the interviews I’ve conducted and always went away learning something from an excellent candidate or a candidate who didn’t meet the expectations, going out knowing where they fall short.

> If you are a tech leader/senior engineer with influence on these matters, I HOPE, that this is not the path you take.

On-site interviews should be the standard path to take to detect the frauds and cheaters before they become potential bad hires.

Online technical assessments now have little added value given that AI tools like finalround.ai, etc just make it easy for frauds to cheat the technical interview.

This is actually a strong argument to a return to on-site interviews and should just draw the line on this and detect the frauds altogether.

The fair compensation to fly someone internationally for a face-to-face might make sense if they stay on for 15 years.

> 2. Force install spyware/vanguard like crap to scan your whole personal computer before doing the interview

People who professionally cheat on online exams just use an HDMI splitter/capture card to inject whatever image they need using a second computer. It’s a lost game.

For offline interviews there are hidden earpieces etc.

I might be calling obvious but if candidate uses something like this, is this not lying to potential employer?

I'd rather be rejected than got job I'm not qualified to do, that's why I never prepare for interview above studying the actual company- and that is more to give me proper signal if I want to work for them, I want to be motivated to make move on more than just money.

I also have never done leetcode, though I like competitive programming problems, especially bot programming contests on codingame.com

> that's why I never prepare for interview above studying the actual company

You and me both. I just read a bit about the company before applying for the job. In the interview, I ask the interviewer only two questions: what’s the management style? Is it a servant leadership style with autonomous work, or is it a hierarchical task-based one? The second question is, what does an average day-to-day job look like? Sure, there are few stressful days and others are relaxed, but on average you can tell. Is it chaotic, always running against deadlines and having new tasks thrown at you, or is everything sorted out properly? Sometimes, I also ask about the meetings. I personally don’t like useless meetings. A lot of middle management uses meetings as part of the power dynamics and exerting power rather than actually discussing and solving issues.

Of course I would also want the information from your second question, but it never occurred to me that asking directly would get me that information. I guess if it's a leaf node they might just tell you the truth, but in general you'd expect the interviewer to say what they think you want to hear wouldn't you?

> I'd rather be rejected than got job I'm not qualified to do

As many people have pointed out, the interview process has diverged from real-world, day-to-day tasks you would be expected to accomplish once you get the job. Not actually being able to do a leetcode test (either due to lacking knowledge or interview stress) might not have any reflection on someones ability to succeed in the role they are interviewing for. How many people are dealing with sorting algorithms or traversing binary trees without access to the internet as part of their job?

It's a 100% unethical and flagrant cheating.

It's no different than if you had a much more technically competent friend off-screen feeding you the answers.

The thing I see people getting wrong with live coding interviews (speaking as an interviewer) is that their code rarely matters. If you write some code, I might ask "So what kinds of failure modes will this code need to account for?" Whether you named your variables well or wrote comments didn't matter. If you can't talk about your work or explain your thinking or discuss tradeoffs or question constraints or show your thought process, that's what's going to fail the interview for you. Getting the exact right solution is bonus points, but the job isn't to consistently get things 100% right writing code by hand.

What you said is true, but it's much easier to do if you (as the interviewer) have full control over the questions and evaluation of the candidate in your interview.

For larger orgs at scale, often the interviewer needs to stick to a script, ostensibly to avoid personal bias (which probably does happen to a degree), and to make scores between interviewers comparable. In such cases it might be difficult for the interviewer to argue that the candidate has done well despite "not getting the answer correct".

A lot of the grievances people have for the interview process are artifacts of scaling the interview process for the largest tech companies. As for the smaller ones, it's basically good ol' cargo-culting. Small startups that follow FAANG processes probably don't realize they can customize their hiring process to their advantage.

How often do doctors have to prove they can still recall every obscure medical fact when changing hospitals? When a lawyer moves to a new firm, do they have to retake the bar or answer weird hypothetical cases all over again? Aren't their credentials, past cases, and reputation usually enough? Why do we in tech keep forcing experienced professionals to jump through these puzzle-like hoops instead of trusting their track record?

Because it's much easier to get-by faking it as a software engineer compared to a lawyer or doctor. We've all have had these colleagues - and I've seen them jumping to other companies easily.

I don't know if that's true. I'm not saying software engineers don't fake it, but I think many doctors and lawyers are faking it, too, if we're defining faking it as looking up information, or even assisting with your core job responsibilities. I don't consider that faking it, but I think that's what we're talking about, isn't it?

One time I asked my optometrist what he was doing when he left me in the examination room for 15 minutes. He said he was looking up the details of the medicine he was going to prescribe me. I don't consider that incompetence, and it sounds a lot like what I do when I need to know about some technical issue. Everybody in my industry knows that Googling stuff is a huge part of the job, but you're not allowed to acknowledge that in an interview.

I think it is actually an interesting and relevant question: why don't past performance or job references play more of a role in hiring, compared to technical screens many people hate and think are pointless? Checking references and former calling employers—if it's done at all—is often a pro forma final step, rather than the first step, which might plausibly be a more effective way to screen candidates.

I don't think that's faking it at all. Being able to research is part of any knowledge based profession.

Faking it is shirking your duties. Most bad medicine doesn't result in obvious harm/cost to a patient or a board complaint. A poorly managed medical practice that runs off support staff and makes patients wait three hours isn't going to threaten a license, and that's faking practice management. A primary care doctor who punts anything beyond the very, very basics to third party vendors or specialists is faking it.

In my major metropolitan area, general practice veterinarians are often absolute crap at doing their (required) medical notes. If you ask them for copies, they'll send you invoices. If you push for real notes, they'll use their 48 hour response window to make something up. They almost always get away with it because during future visits, the vet (whether it's the same person or not) will ask a bunch of questions to (necessarily) fake knowing what was done previously. It's obviously going to affect patient care and cost, and if you look at board complaints that aren't thrown out, the "punishment" is usually because the board saw that the vet has shit for notes. And don't even get me started on vets using AI, which results in hallucinations in the notes -- which I think are notably worse for patient care than just empty notes.

I think that's a good example of even good doctors faking it.

Being able to do the work with reference materials is fine. Maybe if you spend all day on stack overflow asking questions and waiting for answers, not that great. But it depends on the level of work; for me, a lot of my real work problems either have no search results or lead to only reports of questions with no answers. Formal documentation is typically absent, although sometimes there's insufficient or misleading documentation instead. Someone who can't figure things out in these conditions wouldn't be able to do my job.

In my mind, faking it is learning what to say in interviews to get the job(s) without having the skills and ability to do the work.

That said, when I interview, I'm really looking for can they describe the output of a program before the write the program, and then can they write the program that works as they said it would. Also, the program should have a loop inside a loop, because nested loops happen all the time and there are too many candidates that can't deal with them.

Candidates that can't manage a nested loop or can't write a program that does what they said it would immediately prior to starting to write the program but somehow managed to get into my interview are faking it in my book. Some candidates clearly are just having a bad day, and some are probably having a bad day but not as clearly.

References are hard because a) some people fake those b) fear of litigation means many companies prefer to only provide start and end dates and maybe eligibility for rehire. Also, the modern world means nobody answers their phone so this process becomes very asynchronous.

Doctors and lawyers build up immense amounts of "open-source" work that proves their skill. Almost everything a lawyer writes becomes public at some point. Doctors have patient results and billing records that can be checked on a reference call. Software engineers comparatively work in the dark and rely on softly-defined metrics.

I don't think patient results can legallay be shared (in the US) on a reference call with any specifics, and vague statements are no better or worse than a manager describing a dev's performance on projects.

And I'd be kind of surprised if a practice shared billing performance info. That'd be like sharing a tech support person's ticket close rate, but even more sensitive. Maybe at a very high level if HR isn't concerned with defamation claims?

Patients cannot, but other doctors see these outcomes and can comment on them in vague terms. Administrators can also comment in vague terms.

It's all anonymizable.

New doctors, conventional engineers, and lawyers only come with analytical status, though. That hiring process focuses on what you’re like to work with as a person. I recommend that formula because those domains know how to layout a firm which can avoid malpractice. Those firms compete on problem solving, where 10x skill or availability might eventually mean 10x salary.

Firms which associate disruption with profit are altogether different. No such thing as malpractice; and if there was, they’d prefer to hire the “risk taker”.

There’s a lot of doctors in my family. They all carry malpractice insurance. I’m told that malpractice lawsuits have a lot more to do with bedside manner than actual performance.

Doctors and lawyers are licensed, software engineers are not. If a doctor or lawyer screws up they will often face sanctions and lawsuits, whereas a software engineer can just jump ship to a different company and let their resume do the talking. In general it is way easier to get 3 years of experience as a software engineer than it is to go to law school and pass the bar.

I am sympathetic to the argument that software engineers should be licensed, which would reduce the need for dumb technical interviews. But I imagine HN wouldn't like that very much.

> I am sympathetic to the argument that software engineers should be licensed, which would reduce the need for dumb technical interviews. But I imagine HN wouldn't like that very much.

Oh yea, HN hates this idea, but I'm sure it would cut down on all the screening, FizzBuzz, hazing that happens during interviews. If you had a - basic, barebones - certification or exam, that all software engineers must pass, that at least says "This candidate can function at a very minimal level" it would at the very least filter out the 50-75% of candidates who literally cannot code or even speak coherently about the basics of programming.

I think people here are really in love with the romantic ideal that someone with no college, no credentials, no formal-this or certified-that can (in theory) jump right into a senior FAANG job. Yea, that's great I guess, but in practice does it really work? It seems to me that in practice, it just means every company gets thousands of totally unqualified candidates every time they post a job offer, and this helps nobody.

Isn't there a business opportunity somewhere, providing exams and certificates for software developers? I find it surprising that many rich companies have this problem, and there is no standard solution yet.

How much would it cost e.g. Google to start an independent certification company, with branches in 20 different cities across USA. The top candidates could be also offered a job at Google, but every candidate would get a certificate confirming their skill level that many other companies would probably be happy to accept.

At first, the testing could be done for a symbolic cost, because Google already spends some money to interview its candidates anyway, and this would also serve the purpose. But later, when many companies start accepting the certificate, you could let the candidates pay the full cost.

The first rounds of testing could be done at a computer (provided by the testing company), you would just make sure the candidates are not cheating. Only those who achieve a sufficiently high score would be later interviewed by humans. This would be way more efficient than interviewing everyone individually.

What's curious is that in the US there are many IT trade certifications (e.g. A+) and many engineering certifications (Professional Engineer) but software engineering falls into a weird gap. I think IEEE offers some certificates but they don't seem especially valuable.

They kind of offer this service for free already. Who's going to pay them to do it? :)

Oracle has Java certs. That's something MS could also do with CSharp, AdaCore with Ada,...

How useful that stuff is/would be in practise, I don't know.

> "How often do doctors have to prove they can still recall every obscure medical fact when changing hospitals?"

Doctors go through 5-7 years of residency with long hours, low pay, and a full fledged physician watching them who can fail them. And that's after an undergraduate degree and medical school.

If you want to endure a 5 year miserable grind just to avoid interviews, go for it.

Doctors, lawyers, teachers and other licensed professionals do continuing education every two or three years. Software engineers are not licensed professionals, so there is no legal standard of quality that all software engineers are guaranteed to have met (and continue to meet). Hence, the interview is an assessment along with all other parts of the application.

“Continuing education” for doctors could mean someone lectured in the university lecture hall for 45 minutes while you nodded off, then you got your free slice of pizza while signing your name on the continuing education form circulating around the room. It’s a farcry from school.

Here is the bit that makes it invisible:

https://github.com/ibttf/interview-coder/blob/main/electron%...

I wonder: if a LLM can answer typical interview question better than a human, what kind of questions should we ask so that using a LLM becomes worthless? Something that LLMs are forbidden to talk about? Something that LLM doesn't know well?

In my experience, LLMs are pretty bad with low-level languages like Rust/C when you need to deal with pointers and ownership, avoid extra copies.

Another option is to ask for something impossible, as LLMs usually do not admit inability to solve the problem and prefer to hallucinate a fictional solution.

“Forbidden to talk about”. That’s thought provoking and hilarious. I’m just imagining politically incorrect leetcode descriptions.

So cheat to get a job then get fired when they see you are useless? Genius..

I do agree, however from my experience some folk smart enough to build this kind of system will struggle in an interview situation. Maybe rather than outright cheating, this could be more of a leg-up for those who find interviews difficult. Still wouldn't use it myself.

Technical interviews are useless and its normal that their victims fight back

I heavily disagree with this. Technical interviews are great if done correctly.

Or do you think that technical interviews hold such trait that is always present, that you can label all of them useless?

I think they’re useless unless they touch some part of the domain you’re howling to work in.

Why do they need to touch the domain? The point of generic interviews is to find smart people who can learn any domain.

I’m using the term domain very broadly here, I apologise.

What I mean is, if the job your applying for is mostly about working with implementing 3rd party APIs IMO it makes sense that your tech test represents that to some degree.

Can you understand basic documentation.

Can you use data structures.

Can you write tests and understand what actually needs to be tested.

Can you handle exceptions.

Do you try and get the entire resource?

Believe it or not I’ve joined a team in the past who owned a service where no one knew what pagination was.

Bonus points if you demonstrate that you understand what a transaction is.

And I forgot to add, can you design a sensible database schema.

Can a candidate talk about those simple concepts?

Now, would I be able to do a tech test based around domains in machine learning? I would have to do a lot of research first!

I think a better option is to have a system in place where companies hire them for a month or so then after that period you choose whoever made a better impact. And before you say it will be costly, most companies can afford it, especially if it will lead to a better outcome. That way they get tested on the actual systems and not just some useless questions from the internet.

As someone with a job I'm not giving up my job security to take a job that might go away after 1 month. Doesn't matter how much more it pays - I need income every month to pay rent!

That’s a fair point, maybe the proposed system can be adjusted for a 2hours a day then. The point is to test that person on the systems rather than just QA style.

Sure, that's costly for employers, but it's also costly for the candidates.

If you're planning to have N candidates do this when you only have 1 position, you're disrupting the others' lifes so much. It's like one of those chef competition shows. You better pay these people at least 6 months salary to show up for this.

Plus, do you really have the supervisory system in place to have N new hires and evaluate their impact after 1 month. Do you have the work for them to do? Or are they all going to independently work on the same project?

I ask a useless question in interviews, but asking every candidate the same question lets me calibrate the candidates as well as improve how I present the question so we can get to the problem solving and programming skills I want to see. Some questions are worse than others, and I hope mine is less worse; I've had several people tell me they enjoyed working on my question, although I also had a candidate refuse to work on it and ask for another question (which I didn't have, and I was the last interview of the day, so I returned the balance of his time and walked him out)

"Victims"

This just makes it much worse for the industry that is already weeding out frauds or those that claim / lie about their achievements without backing them up when asked very technical tests.

I would say the online technical interview solutions like leetcode, hackerrank, etc are losing their worth given they can be easily cheated but you are just cheating yourself in the on-site interview.

If you don't like technical interviews then what the industry needs is proper and formal certification process akin to the medical, legal, and engineering fields. In the absence of some outside resource to validate that a person has the basic fundamental skills, then you have to ask questions to determine if the applicant knows any of those skills.

The recruitment process is broken.

In tech? Unless you really are useless you'll be more likely to coast.

Obligatory "I don't agree with this and think it seems like a pretty depressing way to live", but:

A colleague yesterday was telling me about how much he and all his friends lied to get their jobs, and he suggested I do the same, because if I don't, I'm competing against people who do, and said that's unfair (for me).

It wasn't even an office job, but I've seen the same thing there too. (Also at university...)

I'm not under pressure so it's easy to say this, but I'd rather not get a job than lie during interview.

I encourage anyone to stand their ground on this. Competing with liars is an enormous soft-skill challenge. But it doesn’t exercise critical thinking. Structuring a personal plan that builds those soft skills is really annoying. Especially if you just want to solve complex problems all day.

That said,

- the interviewer likely embellished or lied to get his job. Be careful.

- a lot of this thread is about avoiding a bad hire. In engineering, someone technical who can detect deception is too valuable to put in the interviewer role. Send them to conferences to “poach”.

Just like the soft skills of competing with liars, you can also improve detecting deception.

In engineering, someone technical who can detect deception is too valuable to put in the interviewer role. Send them to conferences to “poach”.

Those are wildly different skill sets and/or personality traits.

I’d say chances of getting busted are high as well. The invisibility of the window to common interview software won’t do anything mask candidates’ own suspicious behaviour.

I was doing a 45 minute screener interview with a guy the other day who I could see kept glancing over at another window or screen as he was typing in code. Every couple of characters or so he was pausing looking over, then he’d type a handful more characters, then pause and glance again. He was typing incredibly slowly as well, like a character or two per second, even when he was typing. It was honestly a bit of a weird experience to watch unfolding. His solutions were spot on though.

I didn’t make an issue of it, partly because I already had other concerns as well. I can’t think of a situation where there’s anything to gain by getting into an argument with a candidate: I simply didn’t invite him to the next interview stage.

Gong back about 20 years we had to introduce a "pre-screening" interview for infosec roles because of the calbire of candidates we were getting through.

They'd made it through the HR "sift" by having the right words on their CVs but we found that a huge majority of these candidates didn't know even entry-level basics.

There was initially a "quick fire" round where we did some real basic definitions - Confidentiality, Integrity, Availability, risk, etc, etc.

Even back then candidates would hit us with the:

"Hmm, great question"

/tap tap tap tap

[Wikipedia definition of confidentiality]

RAID 45 strikes again

> /tap tap tap tap

Another fine reason no one should be using mechanical keyboards :-D

That is completely ridiculous because the candidate could be looking at a relevant documentation page, which is fair game. Next time, try a voice interview.

Given how poor and assumptive your logical reasoning is, the candidate is lucky for not working with you.

If it's so poor, it should be pretty easy to explain why.

Yes, but it should not be necessary to explain why. Just because a candidate is looking somewhere is very weak data to suggest that the candidate is cheating. I shouldn't have to say that the candidate could be looking at necessary documentation, or even at his own resume, or at the job description.

Since interview process is also flawed in many ways luck based, if you cheat and moderately capable, you can totally stay there and even go up in ranks.

People rarely get fired solely due to technical incompetence, in my experience.

Technical competence happens to be much easier to test for than most of the other factors, and it can compensate to some degree for e.g. bad interpersonal skills. This is indeed part of why tech is such an autism-friendly field. But I've only seen a few people get let go for being great team players, excellent requirements analysts, etc., but just totally unable to code - for better or worse.

(More common is they get shifted to a role which better aligns with their "true" skillset. Why didn't they get this role in the first place? Again: Hard to test for.)

It's starting to feel more like "cheat the job interview because the cat-and-mouse game has pushed everything to the point where you can't legitimately get an offer for a job you're actually fully qualified for anymore"

That's not quite where things are now, but the initial phases are beginning to show.

Perception on the degree of severity seems to depend largely upon the size and structure of your social networks. In the post-GPT4 tech economy, your work friends are also the only worthwhile recruiters.

There's a positive association between 1337code interview hurdles and lack of management competence.

Someone without the confidence to hire someone without going through a bunch of hoops to justify themselves is even less likely to admit humility and say they were wrong. Which makes places that employ these interview techniques terribly passive aggressive places to be.

This kind of tool is why companies will push back towards on-site interviews, and consequently RTO, given that the people who interviewed are the ones who could easily go to the office to be interviewed.

Honestly, it's hard to blame the company. If the employee is willing to lie and cheat even before they were hired, it's expected that companies will take away their tool to do it: remote calls and WFH. As always, the honest worker gets shafted.

Yeah tech has always been a meritocracy built on ethics and we need to remember that.

Trust.

Some people think that trust just means a someone blindly trusting another, but of course that's not how it works. People will be trusted to an extent that they can be trusted, and this goes for company-employee relations too. If people prove they cannot be trusted with remote interviews, companies will take remote interviews away. This was a luxury given to us during the pandemic, and it's now being rolled back because of tools like these.

I remember people here on HN complaining about CEOs mandating RTO without data to support it. Well, this certainly doesn't help.

This is still not data.

For a while my company did aptitude tests that basically are measuring certain types of intelligence. I personally found a very good correlation between somebody doing well on those and them fitting into the team.

For some reason (legal?) we stopped and now we are doing interviews that have a way worse correlation.

One of the few things that I hate more than popular 'technical interview' methods is... lying.

In my recent experience, CS courses should incorporate acting classes to help future job seekers. Right now more interviewees than I expected are doing a really bad job at trying to fool interviewers with LLMs and tools like this one. It's blatant obvious and too easy to catch cheaters (and then they are banned from any future positions on ethical grounds).

> Never fail an online technical interview again

All these online assessment solutions such as leetcode, hackerrank, etc are essentially close to useless given all these AI based cheaters and LLM products being used by candidates.

On-site interviews it is then, and this time you cannot use those LLMs to save you.

If you wanted a completely invisible system you could do it via audio. Speech to text and back again and use a two pairs of same model wireless headphone with say the left ear connected to a hidden device and the right ear to the interview machine.

Other than a bit of practice it would be completely undetectable visually. It’s extra nice that we encourage thinking out loud.

We should assume people are doing this. The solution is interviews that don’t require information that can be looked up or in-person interviews or permitting the use of AI.

The long term "solution" here for candidates will devolve to a second device that uses a camera and microphone to monitor the device from which they are being tested and presents feedback on a second monitor visible to the applicant but not interviewer.

For interviewers, the solution will be proctored testing centers with metal detectors and all devices required to be in a locker outside the test area. (This is how the FAA conducts its written exams.)

This whole thing is getting ridiculous and is a symptom of wealth inequality and our cutthroat society that demands we pay for the privilege of being able to work.

Maybe?

It's also true that pretty much the most powerful AIs in existence are available to anyone willing to fork over twenty bucks a month and you'd be hard pressed to get better algorithmic outputs as a nation state with $100m to blow.

So there's an angle that suggests that this is not primarily a story about wealth inequality but rather savvy in wielding tools.

Tech companies are so tall,

they've become scared of their own shadows.

what interviews get wrong is they typically assess a candidate's lower level thinking skills rather than their higher level thinking skills

cf. https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/

Typical interview question (lower-level thinking, remember): List the primitives in JavaScript. (remember)

Better interview question (lower-level thinking, apply): When would you use an object instead of an array in JavaScript?

Really good interview question (higher-level thinking, evaluate): Here's some raw data that we'd like to put in a database, what data structures would you recommend and why?

I simply give option of candidates to use AI helpers they want. Still a lot come up short. All you need is a baseline that is the same for every candidate and let them differentiate themselves with the same tools they'd use in real live setting anyway.

I expect version 2 to be an AI clone of myself so I don't even have to take the interview.

As much as I hate cheating, I wonder if these apps will lead to invisible and ubiquitous AI helping us navigate life.

As a teacher, it is my job to make sure my students control that AI instead if being controlled by the AI.

I'm thinking: could you just not use second (or third ...) monitor, while only the primary desktop is being recorded?

What options are available for the on-site?

I'd rather just refuse to have a LeetCode interview style interviews all together.

When LeetCode and HackerRacnk shut down, the industry will be a much better place.

Honestly I worry more about hiring an oppositional / defiant / insubordinate “asshole” more than the AI use itself because what does it say about an applicant if their first instinct when challenged is to say “your process is invalid, I’m going to cheat instead”?

:) this took a bit of thought to get the right words..

I am ALL for this, for a number of reasons: - will help people 'find the words' - it's pretty easy to spot someone who doesn't actually know what they are talking about, even if what they are saying is correct and valid - actual technical capability is only a small part of the job.

:D so, 'find the words', interviews can be stressful as all hell for both sides, as someone who assumes the power of the interviewer that's a heavy stick to wield over someone who is trying to explain why they very much want to be paid by you. Communication is _hard_, even if you gel with someone, actually explaining something to them in a way that both people completely grok can be really hard. If someone joined and front and centre said 'I'm using AI to help me with my answers because I find interviews hard' I'd sit up straight and be all 'cool! lets do this!'

'Are you sure about that? what did it feel like?', ok so this one is VERY subjective but it's a pretty solid baseline to go on. Interviews should be a verification that they actually are interested in working with you and that you are interested in working with them. So to be able to make that judgement you and they have to get to a point where the interview can be concluded, and whilst there's a fair chunk of other stuff to go through, technical understanding is important. To be clear, lack of it is fine, that's why different levels exist, you want to be a software engineer but you have no experience or skill? Well ok but you will be starting at the bottom. Now to that, there will sometimes be people who want to 'show a higher level of capability than they have current attained' and that's what my job is, to try and see that when it happens. It's pretty easy to do to be fair, humans experience deeply, computers do not. So if I want to talk about Entity Framework Linq helpers then I'll be looking for the spark of excitement when it suddenly pinged to them that this is WAY easier! And pretty! It makes the queries dance! ..and yes, that's my experience, but it doesn't take much to guide people on sharing their excitement in this stuff, and frankly if you get to the end of the interview and _haven't_ found that excitement or deep interest then skill level aside, you've probably already got your answer.

Finally is around how relevant is a specific level of technical capability? As I said, that's what we have graded levels for (junior, senior etc), and besides this whole bag (for me at least) is much more about how you solve problems than what you specifically know. I've seen people wildly switch careers into software and sure, there's a lot of hand-holding and stuff early on cos it's a proper throw into the deep end, but it's the mind, not the current capability.

I don't get why companies waste time with this sort of interview that just turns into this stupid cat and mouse game. Even ignoring this sort of software, someone cramming leet code exercises is the same thing in my view (they won't remember or use in their work 99% of that stuff).

Just talk to the person, talk about the sorts of problems you expect them to solve. Can they understand the problem? Do they have ideas on how to solve it? Maybe give them a small take home exercise related to the work they'll be doing. Discuss it with them to see if they were the ones that actually wrote it.

Sometimes you don't even need the take home exercise. If they have niche knowledge on the subject you care about they're probably good enough by default (since they are clearly invested and studious in that case).

I only hired a few people in my career but every single one was a huge boon to the team, still around even after I quit as a manager. I didn't even give them a take home exercise. Didn't need to.

Same! This topic gets me soo irritated. If someone is supposed an expert in a field (hence them being an interviewer) how can an average joe fool them into thinking they know more than they actually do during a conversation?

Even now with LLMs generating almost real time, there is always a slight delay and we all know that LLMs are far from perfect.

It seems the tech community on one hand complains at how bad LLMs are for doing deep tech while at the same time fears them as they can make any candidate pass interviews. So which one is it?

Also, the simplest trick even old-timers professor already know:

1) Prepare an open ended question about X

2) Ask it to ChatGPT, Claude, etc.

3) Take note of the titles, keywords used, etc.

4) Ask it to the candidate

5) Compare responses

Not all interviewers are experts. If you are building out your business you will be hiring roles that don't exist yet. Of course that brings different requirements for that first hire, but judging their skill level might prove very hard.

I think it's worthwhile doing simple coding tests to check that the interviewee can actually code. It seems crazy but I've interviewed plenty of people that couldn't write a simple for loop.

I agree asking difficult leetcode questions is less useful because it depends so much on whether people have practiced (or now whether they cheat with AI). Often these questions are so difficult there's no way you could come up with a solution in an interview situation if you hadn't already seen it.

Getting rid of all coding interviews is stupid though. Other industries would kill to have such an easy to test skill! We shouldn't throw away such a nice thing just because some people take it too far.

There are people who can talk their way out of trouble for sure. That is why I suggested the take home exercise followed by a discussion.

It's only when there's niche knowledge involved that I think it isn't necessary, because you won't be able to answer any hard questions properly without having actually worked on code for that niche at one point or another.

Most applicants fake the take home problem or have someone else or ai do it for them.

You have to watch someone produce a compiling for loop to prove they’ve ever built software before. Fraud is rampant.

I've been lucky to never have to deal with that. Sucks for everyone that has.

On the other hand, I personally cannot code properly while someone is conceptually "breathing down my neck". A for loop sure, but I'd probably fail at writing a hash table or whatever despite nearing 10 years of experience on many different projects, including compiler frontends and a simple garbage collector I wrote for the fun of it.

Today, I guarantee you that candidates will eventually take that risk to cheat in the online assessment stage in the hopes of getting an offer from the company and will likely turn into a bad hire.

Not hiring them is better than a bad hire.

If it were me, I would not use Hackerrank, Leetcode, or other online puzzles sites. Instead it would be just a conversation in to just show me your open source contributions in large repositories or you building projects (open / closed) that others are using.

That is it.

For contributions, they must be under public code review or you doing the review and for open source projects they shouldn't be hello world / test projects but useful to others which are used as dependencies in other larger open-source projects.

A basic easy conversation that rids of 95% of cheaters and frauds and those who cannot code.

Not everyone has open source projects to show. Also, for the ones that do, extremely few would be used as dependencies in larger projects, so that requirement is just ridiculous.

Also, how do you even know if it's the candidate that has developed the code? You simply don't know if the candidate copy-pasted the code.

Secondly, just because someone cheated at an extremely difficult Leetcode problem doesn't in any way mean that they can't code on the job. Jobs hardly require such problems.

Instead, ask the candidate to explain specific parts of your code that you show them. It can also be cheated, but it has both a screen and a voice input component, so cheating would take more work.

When I say for loop, I mean for loop. No tricks.

Declare a variable with any number type initialized to the value 0. I’ve seen candidates fail that, spend 30 minutes attempting to declare a variable in the LANGUAGE OF THEIR CHOICE.

That is really grim. I work on rather niche systems engineering stuff so that already serves as a pretty major filter, your average codemonkey won't even bother applying to this sort of stuff.

Exactly. "Yes to fizzbuzz, no to leetcode" seems like a reasonable position that few companies take; usually it's either both or neither.

In my entire career, I only once had a situation where they gave me a trivial technical problem, I wrote the first line, and they were like "okay, you passed the test, no need to finish this, we just use it to filter out the people who can't even start". And then we talked.

But usually it's either just talking, or some puzzle that requires you to remember some extremely specialized optimization technique which you will never in your life use again if you pass the test.

Yep, it’s why I stopped looking for software development jobs long ago. It's this die-hard criterion that’s based on very isolated problem-solving issues rather than the big picture or the concept at least. This has resulted in the current outcome of software development nowadays, where you need a ‘wrangler’ (aka product manager) and all sorts of useless methodologies to deliver a product that’s literally identical to hundreds of products out there.

It gets even worse than that, with hiring based on arbitrary values. One of the companies I worked for before had an engineering manager who chose one candidate over another simply because he had a repo with thousands of stars.

Creating and maintaining popular open source software seems like a reasonable criterion for deciding between two otherwise-similar candidates for a software development role.

I agree. My whole interview technique is just to have a long conversation with the person.

My team was hiring recently. Guy comes on, says he's never used Rust, but has some thoughts on it. We talked about all the pros and cons of it, what the system requirements would be, and so on. He talks us through everything to do with the domain, just two seniors talking to each other at a high level. Very pleasant.

We hire him, and a month later he's making idiomatic Rust.

No need to put people through a ridiculous gauntlet.

You got very lucky. Congrats!

This industry is unfortunately full of very charming, polished, smooth talkers who would ace that interview and then 3 months later it would dawn on you that you made a big mistake. There is this growing breed of "LinkedIn thought leader" types who, during character creation, put all their skill points into Charisma. And they're really good at sounding technical, using the right buzzwords, overwhelming the interviewer with word-salad, subtly steering the conversation towards areas where they have crisp, prepared answers, and so on. They look great, speak eloquently, and have those ivy-leaguer mannerisms that allow them to breeze through these kinds of informal chit chat interviews. Doubly so if that kind of interview is conducted by an exec or someone less-technical. Beware!

> conducted by an exec or someone less-technical

It only works if you are technical. Yes, your sales guy will not know a diamond from a fugazi.

But as a senior tech guy, I don't think it's ever gone wrong. I've never hired a guy who I thought could code and it turns out he couldn't.

You have to be willing to have the conversation with only a few scripted questions, just a few essentials. Everything else an open field. You let them sculpt it into a garden or dig their own grave, either way you give them a shovel.

Every single big corpo knows about it. We just haven’t been able to come up with a better filter for the amount of candidates that are going through the interviews that isn’t extremely subjective.

Small companies that I interviews with had a very simple process (2x talks with upper management, mostly technical talks, also a semi-coding related talk about systems). There are a lot of very well paying (maybe 15-20% less than FAANG) jobs out there, but they’re just that fancy looking.

I guess FAANGs in particular have rather messy recruitment due to the amount of applicants, but my experience is from a giant multinational, talking hundreds of thousands of employees globally here, not "small" by any stretch.

Reply on news.ycombinator.com