Comment by PaulRobinson | Hacker News Reader

Comment by PaulRobinson | Hacker News Reader

Parent Full thread

The problem is though how do you evaluate for real work without a portfolio of experience? Real work projects take weeks - perhaps months - to design, build, review and release. How do you test for that, really?

I agree leetcode style interviews are artificial, but I think they persist because few people have identified and popularised effective alternatives. At least with leetcode you've shown people in front of you have been prepared to go learn arcane stuff and apply it. It's not good, but it's better than nothing.

I was once asked to do the Gilded Rose kata[1] for one ~200 headcount Series D "startup", and it not only resembled real work - it's a refactoring problem - but it also showed their engineering culture. When I joined, I found smart people who weren't leetcode robots, but thoughtful engineers trying to solve tricky engineering problems. I "stole" Gilded Rose to use in other companies when interviewing until I joined my current employer (a FAANG, heavily prescribed process), with great success. I would like to see more katas that are as good at testing real world skills.

Also, something I've only ever been interviewed for twice in 25+ years, which I think is underplayed: pair programming and handling code review feedback. Do this more, please. If you hire people not knowing how they're going to respond to a principal engineer telling them "we need you to think about 4 other things that weren't in the original scope given to you by the PM, can you please go deal with them", why are you surprised when toys are subsequently thrown out of metaphorical prams?

[1] https://github.com/emilybache/GildedRose-Refactoring-Kata

It seems fairly straightforward to just ask verbal questions?

For example, if the position is mostly coding C then just get them to explain how some X interacts with some Y and how that interacts with some Z. Maybe with a copy of of K&R and get them to point to page so and so.

Do that a few times, maybe use a whiteboard, and I think any experienced C developer will be able to tell who's faking it and who really understands within 2 or 3 iterations.

Even if there are no experienced folks in the company, then it will take longer but, after an hour or two it'd be pretty difficult for anyone to fake a solid understanding of hundreds of pages of K&R.

Not the point you’re making but in a C shop, if I interview and somebody pulls out K&R in 2023, they’re not getting hired.

I get the wider point you’re making, and yes, a chat can help. Elsewhere in a reply to this I talk about how I personally like to do that, but the problem is, a significant number of people can talk about coding but can’t actually code.

I once interviewed somebody who had passed 3 previous calls. Asked them to implement fizzbuzz. Couldn’t.

That is a real problem. It’s not myth.

As such, I’m not hiring somebody without seen them writing code the same way I wouldn’t hire a designer without seeing their actual personal portfolio.

> a significant number of people can talk about coding but can’t actually code.

We removed pseudocode from our pre-screening due to this. Some people are absolutely great with English and pseudocode, but can't write a for loop, in their "preferred" language of their choosing. I'm sure they could be great at programming...eventually.

Asking people face-to-face is a lot different then asking over a phone, it's easily 1/10th the info, or less, so of course the percentage of fakers is much higher.

A Zed Shaw's Learn C the Hard Way fan, I see.

> It seems fairly straightforward to just ask verbal questions?

This is basically my interviewing POV. If you've done the work, you can talk about the work intelligently if I ask some questions about it. If I ask you how you would think about solving problem XYZ, you can probably verbally think through how you would solve the problem, what a solution might take.

I write a lot of C and gave up on reading K&R as it is just too outdated.

This seems like a better way to evaluate real-world coding skills. Did you use it in a standard 45-60 min interview or as a take home problem? It seems like it'd take a decent amount of time to read the requirements and just introduce the task.

Take home and “please time box it to maybe an hour maximum, you’re probably doing a load of interviews and tech tests at the moment, so you don’t need to finish: I just want to see your approach”.

In-person time I used to like to ask a question I got from here: “Tell me about your favourite coding work or side project”. We’d then dig into choices, trade offs, obstacles and how they got around them.

My favourite answer to this was “an OpenGL implementation for the X Window protocol which I wrote in Common Lisp”. Just getting through the first layer of “Why [x]?”, took 15 minutes and we spent an hour talking about all the facets. Hired him, was a superb colleague.

"Please timebox to an hour" is a bit dodgy though. If one candidate really does that, but the rest give it an extra 2 or 3 hours, that 1 hour candidate will probably be at a disadvantage.

I often ask for them to version control and send me a zip of the git repo, so I can look at commit histories. Yes, they can fake that, sure, but I can normally tell a lot about a dev from that, and actually if I see 4-5 commits over an hour perhaps with some wrong turns, that pushes the candidate up in my estimation than somebody with 1-2 commits that don't tell a story, but does show a complete solution - which is most likely what would happen if you edited history a bit.

When I'm actually writing code and hashing out a solution over the course of a single hour in my job, I'm not going to commit the wrong turns.

What you're really asking for is a candidate to spend extra time inventing a convincing git history.

I really think the only way this actually works is if the problem is on a page which you can only view after you start a timer or something.

Or do it on site and watch them.

This is all high pressure but if you really want to time box, there can't be a way around it.

Thank you for mentioning this. I always find myself going further back and forth with employers who allow a take-home exercise but ask that it be time-boxed nonetheless; it's difficult to try to unpack how they're trying to communicate what kind of time pressure a role experiences, and that if I took longer than their expected time box, does that look bad just because I needed some more time to understand the problem?

I don't think most employers are out there looking at the timestamps on your commits to see if you really completed it within the timebox or not -- I certainly wouldn't.

I don't think that candidates who spend less time or turn in incomplete take homes are necessarily at a disadvantage. Sooo much can be discerned from a take home even if it's not finished. I can evaluate a candidate's familiarity with modern syntax, how they organize functions, how readable their code is, whether or not they used ChatGPT/CoPilot or copy/pasted from an online tutorial (surprisingly easy to discern when you're evaluating many submissions), and so on.

All of that tells me a lot about how well someone functions as an engineer, as well as what level they're operating at, and it doesn't require the completion of the take home problem.

Thanks for sharing your take. I wish I came across interviewers like yourself in the last 18 months of job searching.

In a current hiring loop my company is doing for a senior, we have a take home divided into a series of steps, with explicit instructions that say that completion of the entire exercise is not necessary to advance (and we mean it).

The take home serves two purposes: (1) should the candidate progress to the in person interviews? (2) if the candidate stumbles in the in person coding, can I deduce from the take home that they do actually know what they're doing, and the stumbling was a side effect of interview anxiety?

I can tell a lot even from an incomplete take home, about whether someone is likely to do well on the in person interviews, but I mostly find it extremely valuable for (2), as a tiebreaker.

It hasn't happened yet in this cycle, but I can imagine a situation where someone turned in a fantastic take home and then brain farted during the in person coding, and, thanks to the take home, got hired.

Refactoring is the best kind of test! It has a slight bias for programming languages but the bias can be mitigated by having refactorable libraries in many languages.

I'm fairly certain I would not perform well with this, unless the library was trivial.

Refactoring is a task that's mostly about reading, understanding context, and a bit of rumination.

I think having them design the library, or a fraction of it, from scratch, would give a more momentum.

> I think having them design the library, or a fraction of it, from scratch, would give a more momentum.

I have done this as a take home test and I am ok with this as long as it doesn't take more than 5 hours.

I would never allow this. I need to know that they generated the answer. In several video interviews, I've had people copy paste the first google hit, and one used co-programmer, listening and writing the answers in another window. The second was good at coding, but couldn't explain a single line of it. I think it's only safe to assume that anyone with a take home will cheat. But, it's also very hard to fire people, in our group.

>I think they persist because few people have identified and popularised effective alternatives

Astrology persists, but that's not a testament to its efficacy.

Thanks for sharing, I've not come across this kata before. Seems like it'll be a fun one.

Reply on news.ycombinator.com