Hacker News Reader

Comment by perlgeek | Hacker News Reader

Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead?

There's very likely a real answer to that question, and that answer should shape the way that engineer should be assessed and hired.

For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request.

It seems to me the heart of the problem is that companies aren't very clear about what value the engineers add, and so they have trouble deciding whether a candidate could provide that value.

View on HN

The even bigger challenge is that hiring experts in any domain requires domain knowledge, but hiring has been shifted to HR. They aren't experts in anything, and for some years they made do with formulaic approaches, but that doesn't cut it anymore. So now if your group wants to get it done, and done well, you have to get involved yourself, and it's a lot of work on top of your regular tasks. Maybe more work because HR is deeply involved.

I saw this at the big corporate (not faang/tech) place I work at. Engineers run and score interviews, but we don't make the final decision. That goes to HR and the hiring manager who usually has no technically background.

Yup, I have seen some really poor decisions as a result of this. I'm also curious - what will be the effect of AI assistance during behavior interviews, etc.

>hiring has been shifted to HR

Well, unless you know sufficiently senior people. But I suspect that is a deeply unsatisfactory answer to many people in this forum.

My long term last, only technically-adjacent, job came through a combination of knowing execs, having gone to the same school as my ultimate manager, and knowing various other people involved. (And having a portfolio of public work.)

Personal networks only disadvantage those who have none.

I suspect many people who don't have strong networks for whatever reason resent that. To which you could probably tack on not having gone to the "right" schools or having a public portfolio.

also hard on introverts who already get punished in workplaces that promote ppl based on proximity and visiblity.

Well, they also promote people based on impact and, with rare exceptions, if you're holed up in a corner someplace you're probably not having a huge amount of impact.

Reality is that the ones quietly holed up in the corner are usually doing all the unsexy maintenance-type work that the extroverts don't want to do (because it's not sexy).

Nobody cares about that work... until it doesn't get done. And so, nobody doing it gets promoted.

Or it gets outsourced which is what often happens.

Communicating the importance of your work is a professional skill.

And if it actually isn't very important, you should probably find something else to do or move on in some other way.

If it was all about impact then ppl wouldn't be paying thousands of dollars to learn to play golf with their bosses. But you knew that already.

Is closing deals on the golf course even still a thing these days? I suppose it probably is in some circles but I haven't seen it in a couple of decades of tech industry life when it was more likely to be fun runs or skiing.

closing deals with your boss? what does that even mean.

Getting a promotion?

But my broader point was that golf course socializing seems like mostly a different world today, at least in my tech circles, relative to other venues.

Dealing with people and communication can be learned.

I get it. By nature I was very much an introvert except for certain scenarios when I was in my comfort zone until at least my mid 30s. I was an only child, the stereotypical short, fat kid with a computer growing up in the 80s (still short, became a gym rat, part time fitness instructor and only stopped the latter as my other obligations became greater). Horrible dating life and a bad first marriage before turning 35 (happily remarried since then).

It became apparent that to get ahead in my career, “codez real gud” was going to limit my career. I slowly learned how to “act like I like people”.

But you can only add so much value to an organization typing on a keyboard. There is a reason that every single tech company promotes based on “impact”, “scope”, “dealing with ambiguity”. Those all require soft skills.

Well put. I'm an introvert, I can't do math, I won't travel, etc. are all things that some people claim as if it's the unchangeable nature of things. If that's their chosen path, so be it. But they should understand it will probably be pretty limiting because the world they live in.

Yes they are unchangeable. I've tried many many times to break out of it; it works for a while but i revert back to my base behavior.

We know what kinds of temperament a dog has within few months of it being a puppy ( and who the puppy's parents are). Why would it be different for humans.

Claim that Our temperaments ( and our likes/dislikes for travel) are all learnt is a bizzare blank slate claim that doesn't track with my life experience and what i've seen in the world.

I hated small talk traditionally. It was very much a learned trait. One of the best conversation openers is “what keeps you busy?” and then ask open ended questions.

Ask about their favorite travel destinations or even what are some interesting things about where they live.

On the other hand, step outside your comfort zone and try something different so you have something to talk about interesting.

https://tynan.com/letstalk/

You didn’t become a software developer overnight. You won’t become a great conversationalist over night either.

“How to Talk to Anyone”

https://www.amazon.com/How-Talk-Anyone-Success-Relationships...

(not an affiliate link)

People can certainly decide that certain activities aren't their thing.

I don't want to push to give presentations at international events is certainly a valid decision for any of a number of reasons.

So is preferring to spend more time on coding than managing/mentoring/etc.

But it all has consequences and some branches will lead to more promotions/money/etc. than others. And you may be perfectly fine with that. But go into with eyes wide open.

> Why would it be different for humans.

Because animals typically live in a way more static and uneventful environments, and they have much more limited mental capabilities?

Humans (and other animals) aren't a completely black slate - but unlike most animals, humans have very complex societies that affect their behaviors throughout their entire lives. A few years in a different environment start to change people. Kids (with their still-growing brains) adapt faster, adults - not so much, but the traces will be evident. Move a not-too-fucked-up Russian to the Pacific Northwest, and they will eventually start to smile now and then.

Also, thanks to the language, humans can think things up even when alone, drive themselves crazy in all the weird ways, then overcome all that self-inflicted stress and possibly develop some behaviors as a result.

base temperament is irrelevant then?

My shy cat could've been a party animal like her sister :)

Of course it's relevant, by definition of "temperament". The question is how much of our (very complex) behavior is biologically based and independent of learning.

For a cat, it probably plays a significant role. Cat behaviors are complex but still much more simpler than humans. And changes are rare. Although I've heard of a "lazy" apathetic cat moving into a house with giant outdoor catio and becoming drastically more active, almost like a different kitty.

I'm not sure about humans - how much of our behavior is a true temperament and how much isn't despite tending to not change throughout one's life. I've seen introverts becoming eager activists after they went through some bad things, like war and prison. I've seen people who were jumpy and always nervous becoming relaxed after many years in safety.

> But you can only add so much value to an organization typing on a keyboard.

In my understanding, non-junior software development jobs never were about typing on the keyboard. Senior software engineer is a fancy name for a problem solver, and code is just a specialized tool they can build to possibly achieve the goal. It always was about talking to stakeholders, figuring out what the heck they actually want today, how it fits with what they think they want tomorrow, learning more about those stakeholders so you can guess what they will think they want next week. Only then it's thinking about it all it for a while, and only after that it's getting to press the actual buttons.

But I'm not sure those things require "soft skills" aka - in my understanding - being a people person. For me, it was a very simple learning process - I (as a junior) coded something, a manager came next month and said I have to rewrite everything again because things have changed. I hated it, so I started to think how to possibly avoid or minimize it and optimize my own processes.

And in my mental model, it's not about people (save for tiny companies where a whole department/role is a single person, so I have to account for their mental chaos monkeys), it's all about business. That's why I wrote "stakeholders", intentionally dehumanizing (with no negative connotations) the model.

Thinking about engineering leads I know who aren't about leading huge teams, it's still about mentoring, talking to people, making connections, often talking to external audiences, etc.

I think a lot of that is "soft skills." Maybe not becoming a stereotypical sales person. But it's also not don't ever bug me and let me code.

The way that most tech companies define levels - yes I’m simplifying slightly. I’ll provide citations:

Junior - you are told what to do (business objective) and how to do it (technical).

Mid - you are told what to do (business objective). But you are expected to use your experience to figure out the “how”. You should be able to lead a decently complicated feature/epic/work stream either by yourself or with others and mentoring other juniors.

Senior - You are expected to lead major projects that involve multiple epics with multiple developers, talk to “the business”, disambiguate, deal with XYProblems, communicate trade offs between time, cost, meeting requirements, etc. Now you also start having to deal with cross team coordination.

Staff - cross team impact, dealing much more with business strategy and setting technical direction.

As an IC, the only things you have at your disposal are your relationships and reputation. Both require soft skills.

Leveling guidelines:

https://www.levels.fyi/blog/swe-level-framework.html

https://dropbox.github.io/dbx-career-framework/

Believe it or not, extroverts also have to develop professional skills, sometimes even things that don't come naturally to them.

No question. Yet, social connection seems worth 10x or 100x competence in any particular circumstance and the effects compound. There are some real benefits from and needs to be prosocial and socially competent but I've regularly seen social competent but technically incompetent people advance far over technically competent but less socially agenda driving people (that are nonetheless socially competent). This only gets worse at scale and as you progress.

I love coding and do it reliably well with joy but as my career has progressed I've struggled more and more with getting a company to let me work at a "low level" or to navigating what seem like sociopathic behaviors to really contribute at my capacity.

What do you mean by 'low level'?

Are you talking about a traditional Staff / Principal engineering role or something different?

Sorry for my lack of clarity, yes. Pure code and technical contribution, up to mentoring, as opposed to holding architecture summits, politicking, and the like. I've been pushed into management and socializing without regard to my willingness.

As one, I have to say there's really nothing about being an introvert that prevents one from being affable and available. The idea is that human interaction does not boost the introvert's energy the way it does the extrovert's, not that it's impossible.

Self advocacy is part of the job.

And just because you’re an introvert doesn’t mean you are incapable of building soft skills. Talking to people is absolutely exhausting for me, but I force myself to do it and practice at it because I know it is important for my career.

HR are experts in HR, which is to say they have a broader view of the institutional needs and legal requirements of hiring staffing than you do. It's always annoying when that clashes with your vision, but dismissing their entire domain is unlikely to help you avoid running into that dynamic again and again

I've never seen hiring completely in the domain of HR. HR filters incoming candidates and checks for culture fit etc, but technical competency is checked by engineers/ML folks. I can't imagine an HR person checking if someone understands neural networks.

HR involvement is unavoidable at big companies; and basics like "years of experience for payband" can cause issues. They fundamentally do not understand the job, but somehow have to ensure its not a biased hiring process.

Yes, and it is kind of necessary when hiring people from outside trusted networks. HR makes sure if people are who they say they are, background checks, and so on. Years of experience and so on are crude filters and should be bypassable by the team/hiring manager if the candidate meets the requirements. I know in large companies this can become political.

The big issue with the implementations I have seen is because HR aims to minimise the number of checks they do, so they can appear efficient.

So most aim to do as much resume filtering as they can first, then short calls, then background checks.

And the filtering and the call are areas where domain knowledge is relevant.

> hiring has been shifted to HR

Not everywhere. At my company, HR owns the process but we -- the hiring tech team -- own the content of interviews and the outcomes.

> Company A wants to hire an engineer, an AI could solve all their tech interview questions, so why not hire that AI instead?

Interview coding questions aren't like the day-to-day job, because of the nature of an interview.

In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.

It also has to be a problem that's solvable within about 40 minutes.

The problem needs to test the candidate meets the company's hiring bar - while also having enough nuance that there's an opportunity for absolutely great candidates to impress me.

And the problem has to be possible to state unambiguously. Can't have a candidate solving the problem, but failing the interview because there was a secret requirement and they failed to read my mind.

And of course, if we're doing it in person on a whiteboard (do people do that these days?) it has to be solvable without any reference to documentation.

>Interview coding questions aren't like the day-to-day job, because of the nature of an interview.

You have missed his point. If the interview questions are such that AI can solve them, they are the wrong questions being asked, by definition. Unless that company is trying to hire a robot, of course.

That’s silly. It’s like saying that a car can do a 100m dash faster than a human so it’s not a useful test for selecting players for an NFL team.

> In an hour-long interview, I have to be able to state the problem in a way the candidate can understand, within 10 minutes or so. We don't have time for a lecture on the intricacies of voucher calculation and global sales tax law.

If you send me a rubric I can pre-load whatever you want to talk about. If you tell me what you're trying to build and what you need help with, I can show up with a game plan.

You need to make time for a conversation on the intricacies of voucher calculation and global sales tax law if you want to find people jazzed about the problem space.

Proving if they are technically capable of a job seems rather silly. Look at their resume, look at their online works, ask them questions about it. Use probing questions to understand the depths of their knowledge. I don't get why we are over-engineering interviews. If I have 10+ years of experience with some proof through chatting that I am, in fact, a professional software engineer, isn't that enough?

Have you ever hired?

No, it's not enough. There are people out there who can talk great talk, and have great resume, but cannot do their actual job for some reason. Maybe they cannot read the code, maybe they cannot write the code, maybe they can write the code but not in the manner that keeps the rest of codebase working... I've had people like that on my team, it was miserable for all of us.

It is essential to see candidate actually write and debug code. It would be even better if we could see how the candidate deals with existing huge codebase, but sadly this kind of thing can't be easily done in a quick interview, and good candidates don't want trial periods.

I have seen people passing leetcodes and other types of intensive interviews with high ranks but then failed miserably at their jobs. I have seen hard working/learning people, having zero idea on specific programming language but know some basic fundamentals become the best engineers. I know lots of people who are great engineers and have become VPs of engineering, staff engineers, architects, etc. who didn't complete the technical assessment successfully. This means that they weren't perfect at the time of the interview but had other skills to succeed in their jobs in various companies. They were hard workers, great learners, versatile, innovative, etc. But that was a past era of interviews when people were not searching for perfection but mostly evaluating soft skills. Nowadays people just search for the perfect candidate. Any error in the technical assessment or leetcode means rejection, any imperfect/doesn't meet my own solution response also means rejection. Nobody checks for or evaluates soft skills, nobody checks whether I have solved similar problems in some past experience. I guess that's the issue with bad fits, not just the lack of technical skills (which I agree that needs to be evaluated as well but not in an extremely hard and strict yes/no format unless you are google and co).

One of the best interviews I've encountered as a candidate wasn't exactly a pair programming session but it was similar. The interviewer pulled up a webpage of theirs and showed me a problem with it, and then asked how I would approach fixing it. We worked our way through many parts of their stack and while it was me driving most of the way we ended up having a number of interesting conversations that cropped up organically at various points. It was scheduled for an hour and the time actually flew by.

I felt like I got a good sense of what he would be like to work with and he got to see how I approached various problems. It avoided the live coding problems of needing to remember a bunch of syntax trivia on the spot and having to focus on a quick small solution, rather than a large scalable one that you need more often for actual work problems.

Problem is, company A doesn't need an engineer to solve those interview questions but real problems.

This is the answer.

Let's not pretend 95% of companies are asking asinine interview questions (though I understand the reasons why) that LLMs can easily solve.

Let's go one step further: LLMs can't solve anything, but most interview questions are covered so much online that they'll parrot a passable answer.

Yeah, what you want is a General Intelligence that has learned the topic you care about. Google search returning an algorithm when you ask it doesn't mean that you shouldn't test candidates on that algorithm, since you still need a General Intelligence that knows it and not just the algorithm itself.

“Real problems” aren’t something that can be effectively discussed in the time span of an interview, so companies concoct unreal problems that are meant to be good indicators.

Really? How short are your interviews, and how big are these Real Problems such that you can't get a sense of how your candidate would start to tackle them?

The “real problems” most companies want people to help solve involve the evolution of products that last for years, involve repeated design discussions, in depth research, and applying retrospective learning. I don’t need someone that can just glue a Rails API together. If I did, I can literally just download that from the internet for free.

If my problems could be solved in the time span of an interview, why would I waste my time doing that interview instead of just solving it?

I don't see the issue here. Nobody expects candidates to build actual product during the interview. Having a (targeted, scope and time-limited) design discussion or giving your candidate some made-up context around an engineering cycle and then doing a retrospective with them are practical and useful ways to interview a candidate.

I'm also not sure what the alternative is? Just not hiring?

> Having a (targeted, scope and time-limited) design discussion or giving your candidate some made-up context around an engineering cycle and then doing a retrospective with them

You just described a contrived, “unreal” problem.

> I'm also not sure what the alternative is? Just not hiring?

The alternative is to come up with questions that are representative of skills related to “real problems”, as you just did, and use those instead. Unfortunately candidates consistently complain that such questions aren’t realistic.

OK I think we're on the same page, and just had a semantic issue.

I've had some success with just describing what we're doing and seeing what the candidates ask.

Mind, I work in very small companies and never had to give input for filling 10 positions at once... just one at a time.

I've tried this, but it becomes very hard to justify, with clarity, why it's a yes or no in the feedback, in a way that can be understood well as it passes through all of those up the chain that are involved with hiring.

And, I've also had people speak very well, doing great with the verbal explanation and questions, even good pseudo code, and then be unable to write a simple for loop, of any kind, in any language. These people also often have a resume full of short runs.

So, I structure mine around a, fixed, work related problem that lets me clearly justify the yes/no in a way that upper management can stomach, but then just bias my feedback a bit based on the "personal interpretation" things like what you describe (which I think are usually better indicators).

Also, resumes are 90% fiction, from what I've seen, especially from certain demographics (not allowed to perceive that though). I don't bother believing them or talking about them, unless there's time after.

> well as it passes through all of those up the chain that are involved with hiring

Yes, this mostly works in small organizations. I'm mostly in positions where I have to pass the feedback once, or at most twice up the chain.

On that, these unreal questions/problems are decent proxies for general knowledge for humans, but not for AI. Humans don't have encyclopedic knowledge, so questions on a topic can do a decent job of indicating a person has the broader depth of knowledge in that topic and could bring that to bear in a job. An AI can answer all the questions but can't bring that to bear in a job.

WE saw this last year with all the "AI can now pass the bar exam" articles, but that doesn't lead to them being able to do anything approaching practicing law, because AI failure modes are not the same as humans and can't be tested the same way.

Tech interviews in general need to be overhauled, and if they were it’d be less likely that AI would be as helpful in the process to begin with (at least for LLMs in their current state).

Current LLMs can do some basic coding and stitch it together to form cool programs, but it struggles at good design work that scales. Design-focused interviews paired with soft-skill-focus is a better measure of how a dev will be in the workplace in general. Yet, most interviews are just “if you can solve this esoteric problem we don’t use at all at work, you are hired”. I’d take a bad solution with a good design over a good solution with a bad design any day, because the former is always easier to refactor and iterate on.

AI is not really good at that yet; it’s trained on a lot of public data that skews towards worse designs. It’s also not all that great at behaving like a human during code reviews; it agrees too much, is overly verbose, it hallucinates, etc.

I want to hire people who can be given some problem and will go off and work on it and come to me with questions when specs are unclear or there's some weird thing that cropped up. AI is 100% not that. You have to watch it like a 15 year old driver.

It's because coding interview questions aren't so much assessing job skills as much as they are thinly veiled IQ tests.

I think if it was socially acceptable they'd just do the latter.

A lot of companies have IQ like tests, in particular big consulting companies like McKinsey and so on.

McK's case interview is just as game-able as HackerRank style interviews. There are entire consulting clubs at many colleges that teach this exact interview style. It's true that it's harder (but not impossible) to use AI to help, but calling it an IQ-like test is true only as much as any other technical interview.

That being said, McK did create an entire game that they claim can't be studied for ahead of time. If the intention is to test true problem solving skills, then maybe that's roughly equivalent to a systems interview, which is hard(er) to cheat .

> That being said, McK did create an entire game that they claim can't be studied for ahead of time. If the intention is to test true problem solving skills, then maybe that's roughly equivalent to a systems interview, which is hard(er) to cheat .

Sure, right up until someone leaks it

Maybe, but they have hundreds! And new ones every quarter! Having done both systems and case interviews I can say that leaking is not as big a problem as with coding interviews. Sure you can memorize how to build Netflix/Zoom, or how to analyze P&L for a sandwich shop, but those interviews depend on interviewers throwing in complications or asking clarifying questions.

IQ tests are just as gamable.

And they're losing all but the worst candidates because of it, which explains a lot.

Plenty of companies administer IQ tests. The reason everyone doesn't is that it doesn't work well.

Nothing works well but IQ tests predict job performance better than anything else.

Do you have a citation?

Also, an explanation for why more companies aren't doing them given their effectiveness? They're extremely easy to administer.

Possibly legal and reputational risks, considering some groups do badly on IQ tests.

No, this is a message board canard. IQ tests are used at a variety of large companies with deep pockets for discrimination settlements. If there were real legal risks, that wouldn't be the case.

There are real risks for companies without deep pockets (for settlements or public relations). People I know, responsible for hiring, have told me they won't use IQ tests because of how it would come across, so the concern at least exists but how widespread is the question.

Your citation addresses it. Less than 1% of employment lawsuits are about selection criteria and employers win over 90% of them. They suggest GMA tests are _more_ defensible than other approaches.

The most interesting thing in that paper is that years of experience performed so poorly. It’s in the lowest cohort. Worse than “interests” or more general “biographical data”.

There is still the reputational risk of using selection methods with widely known disparate outcomes. Other methods also have disparate outcomes, but most of the criticism is directed at IQ tests. I've heard "IQ tests are culturally biased" but never "work sample tests are culturally biased", and I'll guess that's the experience of most hiring managers too.

Somehow these reputational risks accrue to the median hiring company, but not to the global brands like PepsiCo and Proctor & Gamble that do GMA testing already. I maintain: this is a message board trope.

Surely you can understand that the hiring needs and reputational risk of a tech startup and Pepsi are different?

Can you cite Pepsi suffering any reputational risk based on the practice?

Have you ever heard or read a newspaper article, or can you cite an actual example of any company actually suffering reputation harm for administering iq tests? Your citation suggests candidates view GMA tests _favorably_.

Most hiring managers believe experience matters in hiring as well, perhaps that’s the belief that keeps them from using iq tests.

For what it’s worth, IQ tests are biased (see duyme’ adoptive studies for a drastic economic impact). That largely is orthogonal to if they are predictive in the ways your citation outlines.

Schmidt et al., 2016 The Validity and Utility of Selection Methods in Personnel Psychology: Practical and Theoretical Implications of 100 Years of Research Findings

A company wants to hire someone to perform tasks X, Y and Z. It's difficult to cleanly evaluate someone's ability to do these tasks in a short amount of time, so they do their best to construct a task A which is easy to test, and such that most people who can do A can also do X, Y and Z.

Now someone comes along and builds a machine that can do A. It turns out that while for humans, A was a good indicator of X, Y and Z, for the machine it is not. A is easy for the machine, but X, Y and Z are still difficult.

This isn't a sign that the company was wrong to ask A, nor is it a sign that they could just hire the machine.

This is a great point. Though what if the answer is that the company can hire that AI to solve a significant fraction of its actual problems? People who do the assessments and decide what features should look like are often called managers (product, engineering, etc.).

For a while I’ve been skeptical that the rate of hiring of engineers would change significantly because of LLMs, but I’m starting to feel like maybe I’m wrong and it’s already changing and companies are looking toward AI to lower costs and require fewer humans. In that case they are probably still going to want people who are technically exceptional - maybe even more so - but are able and willing to create, integrate, and babysit AI generated code, and also do PM and EM style feature management.

If companies are slowing hiring due to AI, I would expect interviews to get worse before they get better.

> For example, it could be that the company wants the engineer to do some kind of assessment whether a feature should be implemented at all, and if yes, in what way. Then you could, in an interview, give a bit of context and then ask the candidate to think out loud about an example feature request.

So a Product Manager?

Maybe.

Maybe now, or maybe in a year or two, AI coding tools will be good enough that a single semi-technical person can be Product Manager for a small product, and implement all the feature through AI/LLM tools.

Probably not for something of the complexity of Google Maps, but for a simpler website with some interactive elements, that could work.

But then, this was just an example. There can be lots of reasons that companies still need engineers, my point was that they need to think about these reasons, and then use these reasons to decide how to select their engineers.

In most companies every engineer above a junior level is expected to pass features and bugfixes through their common sense filter and provide feedback. Product managers and designers aren't infallible and sometimes lack knowledge about the system or product that an engineer might have.

You can't just take requirements and churn out code without a critical eye at what you're doing.

Reply on news.ycombinator.com