Interviews in the Age of AI: Ditch Leetcode – Try Code Reviews Instead
chrlschn.dev
chrlschn.dev
1. Post well written job post on major job boards.
2. Immediately reply via text to every applicant (yes, we have opt in on the application)
3. Initial screening including several knock-out questions via text. Immediately let candidate know if they are not selected and why.
4. 30 minute can you write a working function from requirements code test. Allow one re-try. Immediately let candidate know if they are not selected and why.
5. Interview and code review session. Bring a side project, show us how it works.
6. Offers out.
7. Background checks start in parallel with on-boarding. First week is spent with business unit teams to learn what the company actually, really does.
8. End of first week, we occasionally have to let someone go on background issues. Otherwise, week 2 is first week coding, expect a working pull request by end of week. All candidates informed position is filled, ask if they would like to be alerted if similar job opens.
Works like a champ. Helps we write software that does this.
Average developer tenure is two years, company is three years old. We write telecom software that does texting, voice and video interviews.
"Average developer tenure is two years, company is three years old." Got it. I don't think this statement says what you think it does.
- people who have free time to work on side projects (usually young people have more free time than people with families)
- people who don't mind sharing their code with others
- people who work on interesting side projects. If your side project is boring, that will probably bore your interviewers -> no offer
- people who work on side projects on regular basis. If I get to work on one side project every year, well, chances are I may have forgotten the hell I did on that project (depending when the interview takes place). If I work on side projects constantly, I have no trouble picking up a fresh one to talk about
This hasn't been my experience at all. The interviewer doesn't care how "interesting" the side project is. What they want to see is what your actual working code is like, that you demonstrate a good grasp of programming concepts, etc.
That's the whole point. Companies would always prefer someone with no responsibility who can be guilted and pressured into doing overtime over someone who can point to actual needs outside of work that they need to attend to. It's a great way of filtering out people who will have boundaries.
Could you expand why? We allow people to substitute a tech interview round with side project showcase. Don't see why it doesn't make sense in your view.
> "Average developer tenure is two years, company is three years old." Got it. I don't think this statement says what you think it does.
So what does it say?
As an option to replace a different interview segment I think that's quite fair. But both choices have to be clearly presented to the interviewee, and they should understand that either option is treated as equally good. OP's interview format seems to assume that everyone does the side project and the tech interview round isn't an option.
It depends on the definition of tenure... if tenure is defined as a closed interval (i.e. defined only for developers who've joined and then left) then it means that, of the developers who have departed, none had wanted to grow with the company, or the company had let them go. For a startup company this might not be a good sign.
If tenure is defined as the length of time the developer had worked, OR, has worked so far, then it means they have a core group of developers and aren't growing the development team particularly quickly. Again, this might not be a good sign for a startup.
What. Of course that "of the developers who have departed" all have left/been fired, that's a tautology. How is it "not a good sign"? Or is it supposed to be a bad sign that there even is a person who is not working there anymore?
Assuming tenure is defined ONLY for those who have already left, 2 years is a bad sign. For a 3 year old company, if that definition is used, it is indeed pretty bad. It means that _of the people who are leaving_, people stay a couple of years, then bounce. This means people stick around long enough to get past the warmup of new employment, get used to your stack and tech, then bounce for greener pastures.
The very important metric this leaves out though is what percentage of the company actually left. If there's 7 people in this 3 year old company and only one person left last year . . . that says almost nothing. Any single person can leave for a great variety of reasons. You'd need a decent sample for this to matter.
The flip side definition of "tenure" is worth considering though: if the average tenure of 2 years includes people still working at the company, there are a lot more variables to content with before you can know anything. A 3 year old company could have an average employee of it have been working there for 2 years and not a single person who joined the company having ever left (e.g. if at year 1 there was a decent amount of hiring). I think this is probably why parent wanted to restrict the definition (and why it's worth thinking about this angle) - because otherwise the company ramp up and trajectory and hiring patterns become hidden variables and you can't glean anything out of the tenure numbers on their own.
Part of it is like a sibling comment said, if an interviewer's interests lie more in line with devops, then a devops project that solves a problem he's acutely aware of will be given much higher praise in his mind, and allows him to engage with the candidate more. If the interviewer rarely does any devops and isn't totally sure of the problem space, he may be biased towards thinking it's overengineering a problem that doesn't exist, or may not fully grasp the problem in the time allotted. The candidate could articulate the side project exactly the same but have completely different interview outcomes in both outcomes.
And an argument could be made that "it's a test to see if the latter candidate can articulate a problem and peak the interviewers interest" but in this case the former didn't even have to do that, and thus you are judging candidates by different subjective metrics. And it's especially an issue if the side project isn't directly related to the role (e.g. video game vs cli tool vs database vs website, etc...).
It also means that some candidates will be judged on system design (if a side project is wide scoped) while some candidates will be judged based on algorithmic design (if the side project is very narrow scoped with the details in the implementation).
It just adds so many variables to the interview project that you are no longer comparing "which candidate is the best fit for the job I am actively hiring for" and instead "which gave me the best vibes", specifically because you did not actually measure all candidates by the same measure.
> You have to be mentally sick to ask people to code review their side projects.
It would be great, IMO.
Meanwhile, you get to spend more time with your kids.
By which I mean I'm potty training my 3 year old.
Maybe your view would change if you were actively looking for work? Spend a week on some fun project while taking a break from resume writing.
I thought it was bad enough these people trying to get you to start as soon as you clear HR.
I'd never even thought about the idea of starting before getting the greenlight... this is insanity.
But....
>Bring a side project, show us how it works.
What do you do if someone doesn't have a side project (or want their side project to be reviewed by someone they don't know)?
Some people code to live, not live to code.
> Some people code to live, not live to code.
Totally fair. But there are lots of companies who prefer people who code because they love coding over those that do it just for the paycheck, and that's also totally fair.
If you don't have time to code for "love", you won't have time to jump at unreasonable requests and pull tons of crunch time overtime either.
Given the choice, companies would love to hire someone with no responsibilities outside of work, this is just a legal way of filtering out parents, people with dependents etc
As opposed to doing leetcode? when you also have to spend time outside of work to practice and memorize them?
If you are looking for a new job, you have to invest time and effort. One way or another. Unless you are a known quality.
Can you suggest a fair way to evaluate a candidate?
Sometimes a side project catches my eye. But I have more than a singular interest. And kids. And to be honest, when a side project catches my eye, work suffers a bit because those tiny moments I have to fill with thoughts are spent on my side projects rather than work projects.
That doesn't mean not loving coding.
It means that you might have other hobbies that you want to spend time on, or other obligations you have to spend time on, etc.
I like what I do at my job. I would even go as far to say as I love it. I also don't do it for an extra 20 hours a week (beyond the 40 hours I already do for work) because I have other things I love to do too (sometimes more!); quality time with friends, family, my other hobbies, etc.
Is there even a CE credit available for coding up a hobby project (and who from)? Completion of a course, attending a lecture, attending a lunch & learn, etc. are the CE credits I am familiar with (spanning a few different professional bodies).
But anyways, that's not really my point: My point is just that in professions it's not uncommon to be expected to perform some kind of extracurricular activities related to your job. Often software "engineers" aren't members of a professional order, but I'd argue that the idea still applies. Tbh learning by working on a hobby project is way more appealing to me than watching some PowerPoint presentation...
If my employer _expects_ me to do something related to my job, I get paid for it. I really wish we'd all stop normalizing working for free.
If it is a personal interest or hobby, I do it on my own time. If it is something required for work, I do it on company time. If there is a lot of overlap, I do it whenever.
Other than that, I learn and improve like any other person does.
Continuing education credits, which is what started this subthread, is something required by the professional body that my company wants me to be a member of. So they happen on company time and dime.
Software engineering exists in a sort of gray area where you can often be a professional software engineer without having to be a member of any order, which is great in many ways. But I feel like one could argue that the informal expectation of software engineers to care about software outside of their work is similar to what is expected in other professions with more rigid governing bodies.
I didn't say they do it for nifty-ness. At risk of repeating myself again: if it is a requirement of my position, I get my employer to pay for it. Why it is a requirement doesn't matter to me.
If you want to pay for and do continuing education things on your own time and dollar, I'm not going to stop you.
>But I feel like one could argue that the informal expectation of software engineers to care about software
I didn't say I don't care. I just have plenty of other things that I care about that take priority when I am not working (spending time with my family, friends, doing other hobbies, etc.).
Yikes!!! I’ve never heard of a potential employer giving offers before background check. Do you tell candidates this before they quit their current jobs?
I worked with a guy who it later came out that he was convicted of conspiracy to commit murder when he convinced a buddy of his to kill his ex-wife and her new boyfriend. Probably should have had better background checks at that company.
I also decline signing NDA, providing data they should not request. Asking to cross off anything that encroaches intellectual property I might create after hours.
But if I would need that paycheck ASAP all of it goes to trash and I would comply to most of shenanigans, maybe not all but still.
We're pretty big on making sure that our contract allows this as we encourage developers to do side projects - open source, etc. We do have an anti-double dipping clause.
If it’s fully communicated that the background check happens in parallel and the new hire doesn’t expect anything to pop up, it seems like a good option (provided the employer is reasonable about trying to solve unexpected inaccuracies that may arise).
In my last two positions, I’m pretty sure my employment contract included a section indicting that at any time they can do a new check on me. And I believe my most recent one explicitly asked if I expected anything to come up.
That's true, but consider this: by running these checks in parallel you are exposing your employees to whatever risks the new element brings in. Perhaps they were convicted of SA etc. and decide to stalk someone at your co.
It just doesn't seem like the best idea.
If the employer does the bg check before handing the offer, that means the candidate hasn't resigned yet from their current job. So wouldn't the bg check expose the candidate? (E.g., my boss would know I'm thinking about leaving)
>>8. End of first week, we occasionally have to let someone go on background issues
Yeah, that's bonkers.
And in that case you do the check before they start with you, yes?
Here in the EU at most you can ask if they worked there, nothing else.
Aside from things that may lead to accusations of violating equal opportunity employment laws, you can ask just about anything. Many previous employers won't do more than confirm employment though, as a negative review may open them up to liability. As a result, the norm in many industries is to only ask (and receive) confirmation of employment.
But that signals to their current employer that they’re applying for other jobs. Unlike the EU, employers in the US can be retaliatory about that.
Want a job or rent a place? Background check, baby!
Not just prior employment but even civil court cases and criminal history are all a part of it, and it doesn't help that even something as small as a traffic ticket shows up on your records.
Have you ever sued a landlord to get your deposit back? There's a good chance your new landlord doesn't want you.
Were you convicted of petty theft years ago? That might cost you your prospective job.
In most of the EU, much of this information isn't public, and in many countries, criminal records are inaccessible.
Often, in the latter, you can get a declaration of “good behavior” from the government if your prospective job has some sensitive elements. The government will then issue one or decline based on the specific job and its risks with your record in mind.
Were you caught committing a DUI? You won't get one for a job as a cab driver, but you can safely get one for working at a bank.
Have you got caught embezzling money? Then you can't get one working at a bank, but you're welcome to become a cab driver.
In the US? Well, you're SOOL. Enjoy being marked for life as your options to get an income legally have significantly shrunk.
HR at the orgs I've worked for call that a reference check and it is usually concurrent with the other background check processes before the offer. The background check I'm thinking of is more like criminal history, work eligibility status etc., potentially drug screening etc. I know I have no criminal record but I'd really like to know that whatever service HR is using to screen agrees with this before I quit my job.
I would think that people affected by this are probably used to this coming up and are up-front with what happened before the accepting the offer. "Hey, so great to hear about the verbal offer. So about white collar crimes..."
Unless the company is somehow getting expunged records or something like that, it's likely that the person is lying about something they shouldn't have and got caught.
What kind of issue would that be?
Embezzlement or something? Probably a no-go, especially if you are anywhere near money.
There's a whole host of crimes out there that you could have committed and be hired after. But the opportunity to explain yourself looks much better if you do it before accepting the offer.
So you are filtering out candidates that have families or lives outside of work that do not spend their free time coding?
It also doesn't scale up high enough, because the pool of candidates is smaller. It's a big part of why you won't see large companies hiring this way, typically. The other big part is that big companies have lawyers who see potential for lawsuits due to discrimination.
90% of code review is BS - and AI excels at BS.
First of all nobody should be forced to be judged on side projects alone. There are several talented "family folks" out there. And even if not there are enough "recharge on my time " folks which is totally fair.
Secondly just like in leetcoding interview all it takes is an inexperienced interviewer with a script to degrade the experience!
I want to see good engineering practices, I might be curious about your language decision, but I'd never tear you apart for your choice.
"As you can see, it's based on a I-IV-V chord progression..."
"Usually I try to take advantage of the negative space, but it's not cheating to add a little white gouache in spots like this..."
"...and here you can see that I did more than 12 reps, so next time I'll up the weight."
"When she makes that face she might be pooping. If you ask her and she says 'HHHNNNNNO!' she's trying to hold it in, so bring her knees up like so..."
Most all the coding I do is for the firm I work for, occasionally some of it escapes to the open source world if I can convince some manager to let it.
At the end of the day, if I've wrung out all I got, that's all there is. It is best to recharge and get ready for the next day.
I've written some pretty cool stuff. But often the code isn't 1/4 of what is interesting about what got written :)
Somehow the implication is that if you build even one little side project for fun that means that you spend every single hour of your time coding. It's such a reductio ad absurdum argument.
If I was hiring an architect, I'd expect to see samples of their work.
If I was hiring an artist, I'd expect them to have a portfolio on Behance or Artstation.
etc. etc.
I work in a domain that's very much not front end development, I've participated in interview rounds for at least 100 different candidates, I don't think I've seen "side projects" ever be a factor in someone getting hired, not even once.
The one thing that did put an applicant way ahead of the pack was contributing to open source projects relevant to our domain. It didn't have to be anything super duper impressive, we just liked to see a history of submitting meaningful PRs and getting them through code review.
The initial reason why we have Leetcode today is that Google originally determined through their interview research that the smartest candidates were the ones who were best at algorithms. Google wanted to hire the smartest people, not necessarily the best coders, so that’s why their interviews were mostly algorithmic.
Of course everyone started copying Google without doing the same research and the message got lost. They just blindly asked coding algo questions and that’s why we have an entire industry built around coding algos and the signal is entirely lost now.
It’s funny that people have lost that intent and are now creating derivatives around that, ie thinking code reviews are a good replacement.
It might be, but there needs to be data done to research it to see if you get what you want. If you interview using code reviews, you get good code reviewers. If that’s what you want, great.
But remember that Google’s initial goal was to hire smart people, and there’s no research to suggest that good code reviewers are smarter than average.
This extends to every test though. My ACT score jumped massively on my second try. The reason: I didn't realize how much little time they gave test-takers, so I ran out of time on some portions. The second time around, I rushed through as fast as possible.
In a "can you solve this new leetcode problem that just came out and is not in the training data"? Awful.
If you want to see an example of a new problem check out https://news.ycombinator.com/item?id=37910297. You'd think if leetcode makes you good at solving problems then it should be doable any higher ranking leetcode programmer.
The comment I replied to claimed that "LLMs would do pretty well at leetcode interviews". OpenAIs own report [0] shows how GPT-4 fails miserably at medium/hard leetcode problems that are not in the training data.
Or do you do well solving the problems on your own and just have trouble when being interviewed?
I did extremely well in algo classes in school.....but since then I have built actual applications that run businesses and haven't used algorithms at all.
Grinding leetcode feels like running on a hamster wheel for me.
Nobody happy with their pay and job is going to grind leetcode.
They could just ignore leetcode and ask puzzle problems instead to test ability.
However, I agree with the general sentiment that almost nobody is doing the research to ensure their interview process selects for the things they actually want.
2. Its not a hamster wheel because you get a high paying job for your efforts
These companies don't want an employee who starts to question "why" things should be done or why things are important. Demonstrating that you are willing to apply yourself at something objectively useless simply because you are required to is a huge plus.
Grinding is a poor proxy for work ethic. It is a good proxy for submissiveness.
One of the hiring fads before leetcode was to ask puzzle questions, with a heavy emphasis on Fermi questions and trick questions. These had the same failure modes.
It turns out that hiring is hard, involves some risk, and always tends to illustrate Goodhart's Law: whatever process you put in place to avoid that risk will ultimately become worthless when people start optimizing for it.
In fact this is how you become "smart" - some talent and a lot of hard work (grind)
Hmm. I think this is a different definition of "smart" than I'm used to. I would say that some talent and a lot of hard work gets you "skilled", not necessarily "smart".
Skill is a form of "functional intelligence", where skill = talent + grind. The more talent, the less you need to grind, but combined the two are killer. To me, "smart" where the grind is 0 is not "smart", but just "lucky" (born with talent)
I'm not going to grind leetcode regardless of how happy I am with my pay and job. I have very strong objections it. If a company requires it, I take that as a pretty strong signal that I won't get along well at that company.
They really need to hire less geeky people (disclaimer, also geek but with self awareness). We hire a percentage of people whose first degree is not programming and have formed relevant career paths.
It's not clear to me why this would be a result of their engineering hiring practices as opposed to their business strategy and product management culture.
How did they measure "smartness"? The Leetcode algorithms are mostly solved problems and I know people who can "solve" any leetcode you throw at them, but they are very poor at actual real world problem solving.
It's quite a bit like saying people who memorised Wikipedia are smart, because they know things.
To me being smart is to know where to look.
I'll just get the smartest of the smarts and rule the world! If I only get the alphas of the alphas, I'll surely win.
Reminds me of all the NBA players getting schooled in FIBA, because the international teams have better chemistry. At one point they had to do a total re-boot, because it was so embarrassing.
I know of at least one second tier, non-sv shop that goes hard to hire the best developers in terms of intelligence. It actually leads to huge quality problems during recruiting and poor diversity. There's just something about mega-nerds with extremely poor social skills that keeps the women and minorities away.
What I'm talking about isn't just awkwardness and lacking charisma. I'm talking about randomly telling women that they can tell she is on her period and things like that.
> What I'm talking about isn't just awkwardness and lacking charisma. I'm talking about randomly telling women that they can tell she is on her period and things like that.
These don't seem limited to SV and Big Companies™ like Google, all it takes is a cursory scroll either through this website (even this thread!) or a trip to any of the CompSci-focused boards on Reddit. Additionally not limited to women and minorities, but anyone from a non-traditional background as well.
If anyone feels like working on these issues, the problem is extremely local, you don't even have to leave HN!
As a result you get pretty average people. Even the weirdos tend to be just quiet and reclusive. My brother on the other hand, went to work for a smaller B tier to google, and he had multiple co-workers that were just beyond bizarre.
Have you considered that what's actually happened is people are finally realizing that they don't want to optimize for the characteristics that Google is optimizing for?
Setting aside the question of whether "smart"s are something that can be sought out without a more specific definition, the average business doesn't need "smart" programmers. They need programmers who are predictable, who reliably produce good-enough software, who communicate well in speech and in writing and who work well on a team. Algorithmic ability provides essentially zero signal for these capabilities, so it's not so much that people have lost the intent that they're realizing that the process they cargo culted isn't appropriate for their hiring needs.
Also, I wouldn't take Google's (nor anyone's) claimed research about hiring at face value.
For backend web devs I'm hiring I established a modified version of a process that has worked for me well before:
1. Interact with a RESTA api and solve the problem by doing cURL requests and some lightweight string mangling and computation to generate key. Then submit the result with your BASE64 code in the request.
Your code is analyzed with ts-node (I look for TypeScript devs) and we check the validity of your generated key.
On success I get an email with the person info and the code. To follow up.
I give the url of the challenge to anyone whom I get their CV and seem interesting for the position. A lot of people disqualify themselves because they dont know a REST endpoint needs the correct content headers, or dont know how to send Auth Bearer token, etc.
Then I see the code, and if I like it, i send them a link to self book a 90hr interview. 30/50 mins are to talk about their experience, resumé in hand, and the rest is to make a version of the server to process the requests from the client they submitted. The problem is common knowledge for both of us, its basically what tgey will be doing, and it allows me to vouch for how they code. Mind, I dont care if we dont finish it, I just want to see if we can solve a problem together with the candidate driving.
And that's it. At the end of the interview I can tell them if I'm going to make them an offer .
I try to do interviews the way I'd like to be interviewed. I hate tech interviews, its stressful and seems pointless sometimes. Why would I do that to people who will be my peers? Bad taste.
I refuse to interview at places that are Leetcode-heavy. Never prevented me from getting 7 figures offers at FAANG-type companies.
Similarly when I interview candidates, I don't ask them Leetcode questions either. I like to get a sense of whether or not a candidate can reason in semi-unknown territories and has good intuition, not if they can parrot a textbook.
"I interview at FAANG companies and refuse to interview at FAANG companies"
In the "age of AI", I'm not sure we'll ditch the mini projects. I think testing a candidate's ability to _use_ AI to generate code is an important skill. However, it's so new, I'm not sure what the best practices might be.
I'd be curious if anyone has any advice on this front...
I think your approach is good, I would introduce things that work but are terrible design decisions and have the person interviewing figure out what could be improved.
No, but I do encourage them to use the best tools possible. :-)
To be honest, as a CTO (not currently hiring though) I want people that can use AI to speed up their coding but who know how to use it well. If anything I suspect those devs are better than devs who don't use AI as they are practicing using their code reading abilities more often and filtering away any bad code written for them.
If I were capable of looking good during the only experience in my adult life that feels like having a solo in a middle school strings recital, I’d be applying to those places, not yours.
1.) DS and Algo knowledge are way overrated in day-to-day job functions. I have yet to have to implement a single DS or Algo from those classes (they are almost always available to the framework, standard library, or language), runtime complexity has literally never once come up in my life, and the most I've ever needed to know about data structures is "which one should I use here?" and 99% of the time its either a vector (arraylist) or a map (dictionary).
2.) Just because you don't have a proclivity for code-golf does not mean you don't know DS and Algos, it just means that you don't waste your free time writing unreadable code for useless puzzles which serve only as interview questions.
There are definitely different types of programming jobs, but for some contrast: I have never had a job where runtime complexity did not come up as a significant factor somewhat frequently.
It really comes down to what you’re doing, but at a certain scale or size of problem you really do need to know about algorithms, data structures, and runtime complexity to avoid having your runtime explode, server costs become untenable, opening the door to extremely long running requests and other problems.
So... That's less likely where you'll need your dynamic programming or palindromes in constant time or space.
That's what most people on HN forget. The programming world is huge. Even more so, if you include all the stuff we wouldn't traditionally include, such as people using code to optimise business or production processes.
It's always best to assume that you don't know what "most software jobs" are like, because you're very unlikely to have seen more than a tiny slice of it.
But what leetcode and leetcode interviews optimize for isn't all that valuable. In the context of the job there are usually different space/time constraints than what you'll see in leetcode, and there are also integration constraints that can change/invalidate textbook approaches.
Most importantly, though, you always have more than 30-60 minutes to figure things out, and can consult whatever references you want.
I found that the biggest performance increase came from async downloads. By sending multiple http requests at once it went from 10 to 2min. After that, optimizing the queries sent to the API reduced the amount of data that had to be downloaded. Finally, I found the best way to bulk upload data to mysql was to use file based uploads which simply copy themselves into the database. Sending 500 inserts at a time was wayyyyy slower. Managed to get it down to 30s. I also switched from python to Java but I never measured the difference between the two
Most developers tend to ask questions about stuff they found challening or interesting, without consideration for how this related to the day to day job.
Like asking a web developer to perform binary shift operations, idenity magic bytes, work with complex numbers, non-trivial question about geometry from people who have no reason to use geometry in this or previous jobs.
I had two interview stages at AWS, second interviwer asked the exact same questions as the first one, and became angry when I pointed out that they probably had a mix up.
Also you sometimes get poor communication from interviewer, unclear instructions, and interviewers literally making mistakes in recursion but telling the candidate they got it wrong.
But my point was more that if you find leetcode hard-to-impossible on difficulty scale, it probably means you are self taught and didn't bother with data structures and algorithms lecture/book/video/course.
Again probably bad communication on my part.
Our interviews is slightly modified fizz buzz. What we are looking for is basic programming skills and ability to test your code. For take home also basic git usage.
It’s the social environment and performance, often sustained for hours, that’s entirely unlike anything else I do. It’s not like emergencies (great at those), it’s not like being in a meeting with a hostile client (good at that), it’s not like any socially-difficult thing I do in work or life outside of it, and it’s certainly not like normal collaboration. Programming while being watched and judged is horrible and energy-sapping in its own special way.
If it was like a normal exam with own and paper, that woupd be easier
It’s also most similar to the job, where I won’t be standing over their shoulder doing live coding.
The candidate wouldn't have to figure out what the code does, because there would be a PR description and/or walkthrough video explaining the code.
There wouldn't be any bugs to find, because code reviews aren't a great way to find bugs. The bugs are found by automated and manual tests.
There would be no stylistic feedback to give. The linter already enforced everything enforceable, leaving only things that are a waste of time to fight about.
There's no point in giving architectural feedback, as the architecture was already socialized and agreed upon before the code was written.
Now on to what code reviews are good for. The changeset is probably a small piece of a bigger picture, because the code is being shipped incrementally. The candidate has no comprehension of the bigger picture to weigh in on downstream issues.
The candidate has no broader understanding of the codebase in general, to suggest reusing an existing pattern, or avoiding a common pitfall with a private API.
The candidate can be well-versed in the OSS framework, and suggest better ways to write some particular boilerplate, or avoid a framework pitfall.
I fear an interview structured like this says more about the company than the candidate. If your company's development lifecycle is mature, there's not much a candidate can weigh in on.
The most useful case that jumps out to me is gaining an understanding of the candidate's familiarity with a framework. A contrived diff falling into lots of common pitfalls could cover more ground than having them write something in the framework. But having them write something in the framework shows you a lot more than just their awareness of pitfalls, so I'm not convinced this is better still.
Code reviews are way more subjective than writing a piece code (at the end of the interview session, if the code works, then that's a huge thing already).
If I highlight something like "Perhaps we could inject that dependency, instead of harcoding it": sure, that seems probably the most correct assesment, but again, what if we are dealing with a really small project and hardcoding the dependency is the best way? Same goes for variable names ("i" is too short? Depends on the scope, right? Well, probably it depends on the interviewer's mind), what about composition vs inheritance? Again, subjective as hell. And don't let me get started with anything related to microservices vs monolith...
Most coding challenges that non-FAANG companies give aren't terribly hard; they mostly want you to work through your process out loud, ask questions, interact.
These soft skills are as important as technical aptitude in determining whether or not you'd be a good fit with the organization.
> but still think that time and people watching you will add pressure
And yes couple hours of interview is different than two weeks sprint, but you are very naive if you think two week of work days means 80 hours of work. Plus I expect a lot more from you during a sprint than during an interview. During an interview I am fine with pseudo code as long as you can explain what you want to do. During actual sprint I actually expect tested deliverables.
I used to, but even then nobody actively watched me code. Everyone else had their own jobs to do
> During an interview I am fine with pseudo code as long as you can explain what you want to do. During actual sprint I actually expect tested deliverables.
So you're optimizing for the ability to write pseudocode under intense time pressure and social awkwardness. Is that the most important quality you seek in a candidate?
It gives us great signals into the things we care about in designers: communication skills, design and product thinking, their mindsets and approaches to thinking about and evaluating products and design, and even their craft.
Everything about the JIRA/timetracking/project management regime is expressly designed to prohibit that - you're a programmer, program! Faster! Know the answer RIGHT NOW!
Here's what I'm going to try if I ever again have to conduct interviews.
1. I'll present the candidate with a LeetCode problem (or let them pick from a list), after emphasizing that I'm not going to be asking them to actually solve it.
2. After they read it we'll discuss it, starting with the problem statement and examples, until we are both sure we think we know what the problem is actually asking. We'll also talk about any edge cases we anticipate might give us problems.
3. We'll talk about possible approaches to the problem.
4. After we have come up with some approach that with sufficient hand waving could convince someone that we could actually right a solution, we'd go the forum for the problem and start looking at the solutions people have posted.
5. For some of those solutions we'll try to figure out if they would work, if they handle the edge cases, spot bugs, see if they are well organized, talk about how well they appear to be written, and stuff like that.
Most of the solutions will have been posted after submitting to the platform and passing hundreds of test cases, so you'd have to pick a very specific problem and solution to find any bugs/not handling edge cases.
* hiring is a very high-stakes decision
* the normal interviewing process has no hope of directly measuring a candidate’s fitness for a particular position. There simply is not enough time. You’d want to evaluate the candidate over maybe a couple weeks of full time work, which, of course, a large number of candidates aren’t willing to do.
* So you have to use the limited time you have to get as much information as possible and accept that direct knowledge really just cannot be obtained.
The idea of leetcode exercises is that someone capable of the analytical problem solving skills and determination necessary to do well at those puzzles will also be capable enough to handle the harder parts of software development jobs.
It’s an imperfect approximation, but for a problem where there is no closed-form solution, it could be one of the better of a number of poor options. (That’s the idea anyway, I don’t know if it’s true.)
All that said, I think a code review is a good idea. Maybe do a couple leetcode puzzles and a code review (and a design discussion, and a “explain a recent project of yours to me” discussion, etc)
> Intentionally introduce some logical flaw or defect and see if a candidate can spot it. A good idea is to go back and find recent bugs that were solved and pull the source before the fix was applied. Can the candidate identify the root cause? How would the candidate suggest resolving the defect?
I've had this exact idea on the back-burner for years! https://news.ycombinator.com/item?id=18794465
The example bug that gave me the idea initially is this one I reported on the "Satsuma Graph" dotnet package: https://sourceforge.net/p/satsumagraph/tickets/2/
Very keen to hear from anyone that has experience with this, from either side of the interview fence.
We cloned a repo, ran they tests, it failed, and the task was to identify why it failed.
It was a tough process for me, since I've never really used Python debugging tools, so I actually had to learn during the interview, but I was able to reason well and picked things up quickly enough that I found the cause, and we had time to discuss ways it could have been fixed.
It was different. I do well during Leetcode questions, so stepping out of that framework was uncomfortable and stressful. I got an offer, but I felt like it was closer than it should have been.
Were they indicative? shruggie. I mis-typed, I interviewed there, didn't work there. And I haven't used the python debugger since, but I also haven't worked on much python code.
I think the code review method could tell you a lot of the same things about a candidate and show whether or not people can issue criticism without being jerks.
What strategies do people leverage to avoid purely AI answers?
This is remarkably effective at filtering out people who had a little too much help in solving it, either from friends, Googling it, or now using an LLM.
It doesn’t happen often, but from time to time someone will come in with a solution they supposedly wrote in the past week but they are unable to discuss it. “I don’t remember exactly what I did here…” and other excuses. It doesn’t take much discussion to reveal someone who never actually understood their own submission.
Of course, I always continue the technical interview to cover the odd possibility that maybe they were too nervous or something. So far I haven’t had anyone who is unable to discuss their own take-home yet shows well in the rest of the interview process.
The arrogance most software "engineers" have is astounding. 99% of companies aren't splitting the atom. Companies shoot themselves in the foot and waste so much talent this way.
Anyone can train and get good at it, it makes interviews short and not time consuming, it’s a standard accros industry so you can focus on studying it and not have to read one book for each company you apply to.
What would be a better alternative? Hiring based on connections? Take home assignments that takes 15 hours to complete? Domain specific knowledge grilling?
My biggest problems are getting requirements, understanding the problem, working around a legacy system or deciphering old code, not actually writing code when requirements and documentation is on point.
If you really valued your time you would spend a week studying for the interview that gives you a 50pct or more increase in salary so you can work that many fewer years
With that said I agree on your other point, putting in the work and sacrificing time, even months, will pay dividends long-term with a salary that allows someone to retire early.
Hearing some people, you'd think they ask you to memorize chess openings or something.
Most Leetcode Mediums fall into the category where the type of solution is obvious (sliding window/graph search/etc), and the code isn't hard to write, but there's a "trick" embedded in the problem and if you don't already know the trick, you're never going to figure it out in 30 minutes.
So the best strategy is to find a "Top N Leetcodes" list like Blind 75 and just memorize them. Of course you can't expect those exact questions to come up in every interview, but having a few dozen solutions already memorized should let you do some pattern recognition and lower your cognitive load while you're trying to figure out what that problem's trick is.
I am sorry to say, this is just not true.
> but having a few dozen solutions already memorized should let you do some pattern recognition
Chess opening memorization is pure "remember the moves, play the moves" that's it. No thinking or pattern recognition involved.
If this is about being able to solve problems by generalizing from having seen a few dozen solutions, then I don't think "memorizing" is the appropriate term here, at all.
You know you live in a clown world when memorizing nonsense makes one more employable than learning some front-end stuff to be a little more full-stacky.
Why would you not do this? If I’m hiring for a specific role, I want to make sure they are good at that role. Doubly so if their resume states they’re an expert at something - let’s find out.
This doesn’t need to be hostile or one-upping, but to prove someone’s actual depth of knowledge in an area they claim to have expertise in.
Not saying the method works, but I think it's weird to argue that it isn't at least an attempt at having good candidates.
I probably should have taken that job offer. It was hybrid, not remote, and the pay and benefits were a little lacking compared to my other options - but in retrospect, they might have been a better fit than where I ended up.
What tips would you have for doing this?
If you can get access to review work by folks on other teams that do more coding, that would be a good start.
From there, you need to allocate time to looking at changes closely. Don't just accept things at face value. Ask basic questions like "why is that there?" and "what does that really mean?" and hunt the answers down. Code consistency can mean many things, formatting and variable casing are the obvious things, but also things like how method names are constructed (e.g. when do you use "get" vs "fetch" as a prefix), are concepts named consistently (is "business hours" always spelled "business hours" adjusted for casing or is it called things like "hours" or "biz hrs" in some places.
I look for three big things: * technical soundness - does the code do what it says it does? * ontological soundness - are the concepts used well defined with clear boundaries and do they play well with the conceptual framework of the rest of the application? * semantic soundness - is the meaning evident? does the code do what you'd expect? How hard is it to reason about what the code does when it is called?
I view writing code as a primarily communicative activity. So, I evaluate it based on how well the ideas it embodies are structured and how well it communicates those ideas.
In my (limited) interviewing experience the good candidates were the ones who had a good mental model for solving novel problems, were inquisitive and asked good questions. I think these attributes are also a reasonable indicator of general intelligence.
The people who I hired that were less successful were the ones who solved the coding exercises but had a poor mental model of how to solve novel problems and did not ask good questions.
Trying to use ChatGPT could take the pressure off from regurgiating coding minutae. The interviewer would be able to observe how the candidate approaches the problem and whether they can spot where ChatGPT goes wrong. It might also help introverted candidates since it feels less confrontational and there is likely to be less anxiety about unimportant details (after all any syntax mistakes would be ChatGPT's mistakes).
Just curious - why would you not feel comfortable using it in an interview? Interesting to hear a different POV.