An interview code submission that wasn’t even submitted changed our process
stackoverflow.blog
stackoverflow.blog
I think many companies through the hiring process completely lose sight of what they are actually looking for. They forget that the barriers they erected (whether they be whiteboard problems, take-home coding tests, algorithmic dance on your head textbook problems, etc.) are simply signals about the actual person and their capabilities...and usually, they're _exceedingly poor signals_. That's why you end up with situations where extremely accomplished and talented developers don't pass many technical interviews at companies where they would have otherwise gone on to be outstanding employees.
And it keeps happening, and happening, and happening.
Interviewing for a developer role should be far, far, far more holistic than it is today - rather than throwing away my entire history of accomplishments and creations and just assessing me on the spot in a 2hr winner-takes-all high-stakes game of whiteboard/technical trivia.
For example:
* If I have a strong GitHub portfolio of projects I can prove I worked on, that should 100% take priority as a signal as to my capability and how I think about committing regularly, writing clear commit messages, etc.
* If I've built products I can prove I alone built and I can walk you through my code, that should 100% take priority as a signal as to how I write code, my proficiency with a particular language, etc.
* If I communicate clearly over email, have a blog with my writing demonstrating my ability to clearly communicate technical topics, that should 100% take priority as a signal as to how well I communicate.
etc, etc.
Sadly, today, I'd say ~90% of technical interviews completely and utterly ignore all the above and want you to do the technical monkey dance...because clearly, what makes a developer great is grinding leetcode for 3 months and then pretending you haven't seen the problems before during the interview.
I think you're right about that. But I've also seen (from both sides of the table) that many companies also have no idea of what they are actually looking for.
There's real contractors and then there's fake contractors which should always be called staff augmentation.
No real senior software dev contractor would accept $75/hour in the Bay area. You're looking at minimum $100/hour.
Although, "fake contractors" or staff augmentation, those can be paid $75/hour or less.
Absolutely. There was a topic here about a week ago where one commenter in particular personified this line of thought (sorry I don't have a link to the comment).
Paraphrasing, they said that if they very to interview someone they're already working with and know to be a very strong contributor, they'd still subject that person to the technical monkey dance, as you put it, and if they "fail" then they wouldn't be hired.
It blew my mind that anyone can think that way. Clearly the interview hazing culture has become so entrenched that the game of the interview has grown to be an entity to itself. We seem to have forgotten the actual goal is to hire good people.
Personally if I'm hiring someone I've already worked with, I don't do any interview. What would be the point? There is nothing I can learn in 45 minutes that I don't already know from months/years working together. If I called them up from my contact list is because I already know they're capable. All I need to do is describe the role and job and ask if it is a fit for their career goals at the moment.
Someone here said that about John Carmack, which just blows my mind. Exactly how bad would his 8-Queens solution have to be to change your opinion of him?
We would also subject John Carmack to that technical monkey dance because we don't believe in free handouts.
> Paraphrasing, they said that if they very to interview someone they're already working with and know to be a very strong contributor, they'd still subject that person to the technical monkey dance, as you put it, and if they "fail" then they wouldn't be hired.
I think I might have been one of commenters you guys are talking about here: [1]. I took my downvotes without complaint, even though I think people failed to apply the "Please respond to the strongest plausible interpretation of what someone says" HN guideline.
I'm not for hazing or monkey dancing. In fact I also hate these kinds of interviews and think they should stop. I'm against special treatment for celebrities. Why should someone who is an expert at growing their personal brand get to bypass the queue with a "quick chat" interview? What does their superior SEO on their blog, or association with some name-brand popular company have to do with their development skill? Does constantly speaking at meetups and conferences actually make you a better employee? Normal no-name developers always have to go through the standard interview process, even no-names with clear, provable track records. Companies have standard processes because we want to measure candidates fairly. Many go even further to try to remove prejudices and unconscious biases from the process. All to get some kind of objective picture of the candidate's suitability.
Why should someone get special treatment to bypass all that just because their names are well-known? I say fix the "standard process" to make it better, but still send everyone through it so you are giving everyone the same chance and comparing candidates fairly.
Interviews are meant to gather information so that you can make an informed decision about hiring the candidate (and vice versa). If you already have some information, it's silly, wasteful, and a bit rude to ask for it over and over again. Suppose you got Carmack in for an interview and he flubs the programming test (maybe he's exhausted or sick or his dog died that morning). Are you really going to pass on him because he missed an edge case, or didn't see the O(1) solution? Frankly, I don't believe it.
I'm also not entirely convinced that you can have a totally "fair" standardized interview. The trade-off for being unbiased[0] is that you will instead have incredibly high variance. That imposes real and mental costs on applicants too: interviews are stressful, take vacation days, etc.
Instead, I think it's better to tailor the interview to what you need to make your decision. For a new grad, you probably do want to see some evidence that they can code. Assuming you can even get a well-established person to jump through your hoops, I'm still not sure you should. Your 45 minutes with Carmack, for example, is probably better spent figuring out if/how he will fit into the role for which you are hiring.
[0] And even...there's a lot of bias in how people prepare. I had no idea I'd be redoing a DS&A class at my first interview; I think Stanford has a whole interview prep class.
Whenever one of my former coworkers would come in for an interview, I would recuse myself from the loop. I figured it would be a waste of time for everyone and I would have an undue influence on the process, especially since they already knew I approved. I would ask to be put at the very end of the loop and then just ask the person if they had any questions about the people they talked to or anything the learned about the company that day that concerned them.
If they were just a friend who I had never actually worked with, I would interview them like normal but would still ask to be near the end so they could ask me better questions during the Q&A.
The worst filter is choosing at random. If I shuffle a stack of resumes and throw half in the trash, I have eliminated the unlucky candidates. I have certainly rejected a good number of skilled and highly qualified candidates. Regardless the filter did its job, I have fewer applications to review.
If I give candidates whiteboard problems, I filtered out thoughtful people who don’t perform well under pressure. If I look at interviewees github submission and blog posts I have filtered out intelligent people with a dearth of free time. If I combine the two filters with an OR relationship, I now let more people through and have a filter that is less effective at narrowing down candidates.
All filters filter out good candidates. Anyone who finds a way to find exactly the right candidates from a filter in a way that isn’t bespoke and is scalable is going to be very successful.
Not everything has to be measurable to make good decisions.
You trust your gut when you have no other option, not because you believe it to be better than a data driven approach
Do you measure your romantic relationships or marriage or family interactions? Do you measure any human relationship?Because hiring and working with someone with fundamentally a relationship, not a data science or math problem.
I had a manager once who said, "if it can't be measured, it can't be managed." You sound like him. Good luck with that approach, and i hope we don't work together :)
It is a method and it may even lead to great hires but it also throws fairness out the door.
That seems more like a baseline case than the worse one. I feel like I could definitely come up with one that’s worse than random. Negative points for indications of intelligence, knowledge, collaboration, and kindness.
I disagree with the filtering attitude. Everybody wants the best candidate but they are not offering the best job.
If you can, hire the first person who convinced you that it's not going to be tough to work with and build an adult and professional relationship where each side is clear on expectations and demands.
Of course if you need a specialist, then be clear (and feel free to check) that specific ability are required. But why don't you have the time to make somebody grow organically those ability/knowledge internally?
I was asked to write a Roman numeral to decimal converter in an interview for a senior developer role. I hadn't thought about it before, so I came up with a lookup table that handled all the special cases (there are only a few, such as IV and IX).
The recruiter (who was also the owner of the startup) didn't like my solution because he expected me to write a generic parser. Now, if I were him, I would have expected a senior developer to correctly determine if the specs for a 2000 year old problem could change in the near future and write the more performant solution when possible.
That being said, there’s nothing wrong with saying “I’d like to see you write a parser. Let’s make one for Roman numerals so it’s fast and simple.”
You came up with a working solution that fell outside the spectrum of what they were expecting. And they couldn't adapt their thinking/hiring process to that so more fool them.
In a generous interpretation of the situation they weren't a good fit for you anyway.
How did you handle looking up two characters vs one? Take a pass for all the doubles first and consider those negative? Or keep a history as you loop over it?
People can create fake portfolios and you even have systems to make it look like you are active on git hub. Then learn to talk through the code or present that code without understanding any of it. You really really don't want to hire someone like that.
If I ask you on the spot and have discussion about something new you just had to come up with, it is much harder to bs.
It is not just about you, it is not just about company. It is about relationship. When I meet someone for the first time I don't trust them, there is so many scam artists that can tell colorful stories where they are "military paramedic just back from mission". Then this story will fall apart only after they are already months in job. If they are scammers they probably will do other bad stuff as well.
Oh come on. What’s the proportion of people doing that out of the applicants you ever interviewed?
Having a five minutes chat about any of their projects would weed them out right away.
"When I meet someone for the first time I don't trust them, there is so many scam artists that can tell colorful stories"
Sounds like they've been scammed before and are taking it out on candidates.
I think that you should work on these trust issues, the interview process is about hiring people not rejecting them.
This is my biggest gripe with today's technical interviewing. They act like leetcode is real world engineering.
HRs responsibility is to protect the company from liability in hiring, firing, promotion, retention, labor relations, wage/hour rules, etc.
To the extent that part of their public statement of purpose is helping hiring managers find the best employees, that's mostly bureaucratic euphemism for “preventing hiring managers from indulging their own conscious or unconscious racial, sexual, age-related, etc., biases in a way which creates legal problems for the company.”
Cookie cutter approaches where everyone goes through the same checkpoints and they are weighed the same way, including cookie cutter technical interviews rather than a nuanced consideration of other factors that determines the need and coverage of a technical interview, is exactly what having a significant HR department encourages.
There's almost never any science or systematic approach to evaluation.
How can you possibly evaluate a test without predefined criteria for evaluating that makes the result good and what makes it bad?
Whether or not the code is "good" is simply a matter of opinion in almost all cases.
Another down side to coding tests is that if the assessor knows less than you about some area of coding and sees something they don't recognise or understand, then they might well negatively view what in fact is a much better way of doing things.
And, much worse than anything else, you have a very tangible chance of spending 12 hours doing it and hearing nothing back at all.
And as for "Tic Tac Toe" as ANY sort of effective test topic, well, that's where you should pull out and find a different job to apply for, where they can at least formulate a relevant test topic and come up with tangible and defined criteria for measurement as well as a commitment from their end as to who will assess it, against what criteria and within what time frame and what feedback they will give you, if any. Ditch the Tic Tac Toe companies.
I did two coding tests in recent years ... in one, their feedback was "they didn't like it", and the other one I worked on for 12 hours because I believe in doing a good job and meeting the requirements, and I heard nothing at all back - not a single word.
Coding tests are, for the most part, bullshit.
...which is so very frustrating. It's painful to read things like this:
> That was all we asked; we intentionally left it open-ended. Some people tried to impress us with huge, elaborate apps that used different services and engines.
The people hiring viewed this as a trivial, small, meaningless ask. They are proud to have "intentionally left it open-ended."
Some potential candidates did not (through their own example) view this as trivial, they went ahead and poured a ton of time into it and most likely were "replied with well wishes" as the company moved on.
It's irresponsible.
That said, if a candidate spends 12 hours or 10,000 lines coding up tic-tac-toe, the company is not entirely to blame here...
I clearly hear what you're saying, and I understand your stance on it. But I can't agree with it.
The burden is on the folks hiring in my opinion.
This is not a "real life" scenario. It should be a SHORT assesment (for both parties). If you were employed within a company as an engineer and you received a nebulus spec (or no spec at all), the burden is then on you to identify, educate, and communicate.
Not during a pre-interview (for the chance and maybe being blessed enough to get a full interview with this world changing company).
FWIW, many of the FAANGs are famous for asking intentionally underspecified questions, and the candidate preparing properly by figuring out what's in scope and what's not is part of the scoring. But at least in an interview you're timeboxed to 45 or 60 minutes!
Hmmmmm... I'd say that "my point of view" differs from yours, and does not fall in line with common group think. Cognitive biases are typically pushed back on, even when they are in your favor.
> since it's the candidate who's the prospective engineer, wouldn't that imply that it's their job to clarify the nebulous spec
Gosh, no. You have it backwards my friend. It's a real opportunity cost on your part to think like that. It's a tactic for some to offload the work onto others so they don't have to spend the time doing it.
> FWIW, many of the FAANGs are famous for asking intentionally underspecified questions
99% (or more) of the startups/companies out there are not FAANG's. They are not the litmus test on how to effectively interview. They have candidates FIGHTING each other for a position at their company. Not the other way around.
Maybe, except when you try to clarify and their answer is that it's open-ended -- and yet they want something super specific in the end. And the super specific things are just one of many tech opinions and not objectively good design.
> FWIW, many of the FAANGs are famous for asking intentionally underspecified questions, and the candidate preparing properly by figuring out what's in scope and what's not is part of the scoring.
Why is this a good thing though? Why is solving riddles in a professional interview somehow ok?
Even moreso if you only have nights/weekends to work on it.
That’s it. No further instructions.
How happy would you be with the instructions from that teacher? What advice would you give to your child?
While the company isn’t entirely to blame for how much time is spent, they are being irresponsible to a degree.
I personally hate it, where I work we do this to people and you can tell just how long they spent on these things, and god knows how many they have had to knock out as part of a job search
I tried to end the practice but got a lot of resistance
Leaving people to guess at their own evaluation criteria is disrespectful, and implicitly biases in favor of people who have lots of free time/energy.
If you are expecting a candidate to have a certain skill-set, having a simple timed coding test as the first round is the easiest way to filter out a lot of unqualified candidates. Stripe does this quite well for example.
When I hire for my company, I do a live coding round with the candidate after an initial phone screen. Since I get far fewer applicants, this works out for me. The objective with the coding round is to understand more than their ability to write code in a particular language. It's a good way to learn how they communicate, how they approach a new problem, how they seek help, etc
The problem is 99% (or more) of the rand-o startups/companies that do this are not Stripe. They are not companies that people are fighting to become employed at.
> this works out for me > their ability > how they communicate > how they approach > how they seek help
What's ends up happening is you miss out on a lot of truly talented folks by putting the burden of proof 100% on them.
> is the easiest way to filter out a lot of unqualified candidates
It may be the easiest way for YOU. It's an important distinction as you (and this is not a personal attack, sorry if it sounds like it is) may not have the hiring skillset. It is a skill. Just like managing people, supporting software, being a CEO, etc...
I've been asked to write a full-stack app from scratch, and I spent like half of the time debugging the webpack config, which could've easily been solved by having a template prepared in advance if I knew what I was doing.
If anyone's reading these, please have the coders work off of an already existing template that you give them.
As to the coding part, I have interviewed some people who seemed very together, but they could not write a simple program. I do mean simple, not some brain-cracking gauntlet of a test. Something you let them work on, in front of you, for maybe 10-15 minutes max.
A lot of frontend devs don't even work with the DOM directly after having spent years with React. Could they do it? probably. Are they prepared for DOM-related manipulation questions on a whiteboard? nope.
And suppose there are a lot of cheaters out there who don't know the things on their resume, even including earning a degree without ever having to write a working program on their own. How would you screen them out?
Make a loop that sums up numbers in an array.
Write an algorithm that sorts a list of three numbers. I watched students spend an hour making a bunch of if statements to try and cover all cases... ultimately getting it wrong.
Compute the accuracy by computing the number of matches between two lists and dividing between the length.
Split a list into two lists.
Generally the failure mode is to find something vaguely near the task, but overly-sophisticated from google. Then wrestle with trying to hack it into working by frantically scouring stackoverflow etc for their next code adjustment.
At my previous role we were surprised just how many people looked good on paper for a development role but barely even attempted the first question on a 3 question HackerRank test.
Oh and that previous paragraph? That's how I explain our coding test, I'm very clear what our standards are.
>How can you possibly evaluate a test without predefined criteria for evaluating that makes the result good and what makes it bad?
If it's even a slightly reasonable solution it's a pass. It doesn't need to be "good" it just needs to not be terrible.
>Another down side to coding tests is that if the assessor knows less than you about some area of coding and sees something they don't recognise or understand, then they might well negatively view what in fact is a much better way of doing things.
A correct solution is acheivable in between 50-100 lines of code. If you do something I don't understand I can look it up. In fact, people have done things that I don't understand and looked up.
>And, much worse than anything else, you have a very tangible chance of spending 12 hours doing it and hearing nothing back at all.
Every candidate has a phone interview first and every candidate gets a response to the coding test. It's unusual to complete the test without e-mail back and forth with one of us for clarification (which is part of the test).
>And as for "Tic Tac Toe" as ANY sort of effective test topic,
First you wanted something that didn't take 12 hours, now you're complaining the test is too simple. You don't know what the company does, so how do you know the skills you need for a tictactoe solver aren't relevant?
>I did two coding tests in recent years ... in one, their feedback was "they didn't like it", and the other one I worked on for 12 hours because I believe in doing a good job and meeting the requirements, and I heard nothing at all back - not a single word.
Companies being bad at coding tests doesn't mean coding tests are bad. Companies are liable to ghost you at any point in the interview, that's nothing to do with a coding test.
Our interview process is on-site and normally takes 1/2 visits, it will involve probably 8 person-hours of our engineering time + management. So yes, we want to make sure that candidates are atleast vaguely able to solve practical problems before they come in.
- The task was to implement a basic tic tac toe AI (with open book !!!)
the state space is so small that anything would work fast enough in a modern computer.
- Some candidates somehow manage to build a program that loses?? that is, it's buggy and they failed to test it.
- The interviewer uses subjective criteria that somehow results in broken code being accepted unanimously.
- Some people overengineered their solutions and this is called impressive.
- Someone failed to implement the exercise, wrote an answer that can be googled in a couple of seconds. Made an irrelevant comment about "min-max" vs "negmax" ( performance is not a concern (small state space) and negmax's main advantage is being DRY).
Made a comment about "test bloat" despite empirical evidence from the submissions showing that there was under testing.
and the conclusion that I am supposed to have is that interviewing without code is better? a worth-mentioning portion of the candidates couldn't ship a working implementation of a standard algorithm and the conclusion is less code??
To be honest tic tac toe is a bad task to choose. it's trivial, easily googlable, and has little room for skill expression. It's also doesn't test the candidates' ability to choose the correct algorithm for the task. and it fails to probe daily aspects of dev work ( concurrency, networking, databases ..) that traditional onsite interviews are less able to test.
Everyone seems to have very different opinions of what good or bad code looks like, what's overengineered and what isn't.
Why is the problem even asking for AI? are they applying for an AI position?
And why is it ok to ask people to work for free on a coding project? From what I've seen, people generally ask for pretty high rates to do consulting, with good reason.
Technical interviews are very similar, only smaller scoped. The goal is to use the right tests to learn what you need to know about a candidate. A lot of this boils down to having a rubric - what qualities are you looking for in a candidate that makes them a good fit for your team? If you cannot answer this, then your interview process is going to produce poor signals.
This also leads into another pet peeve of mine when dealing with technical interviews - the ghosting and lack of feedback. If you are unable to give generic feedback on why a technical assessment did not pass (not the cultural), that is a strong signal that you don’t know what you are looking for as a baseline in candidates. There is nothing wrong with generic feedback along the lines of “relative to other candidates, your commit messages and code style was lacking.” It is especially frustrating when the company invites you to apply again in the future. That’s like failing a project in school, but the teacher is giving you a second chance with the only feedback being, “better luck next time.”
About two months ago I walked away from a pre-interview coding challenge that looked like it was going to take 8-12 hours. The problem was too many requirements.
The same job was posted in last week's "Who's hiring" thread. In this economy, if you can't hire, the problem isn't the candidates. It's you!
If I was working with a recruiter I would have been a lot more blunt.
Edit: if I see the same job posting in another "Who's hiring" I might reach out again... If I'm still looking for a job!
This gaslighting bullshit has to go. People are given vague and contradictory guidelines for the chance at an interview.
"Well we really like the UI and it passed all unit tests and didn't crash and wouldn't crash and follows typical MVC practices, but we didn't like the system design, and for this particular position we score heavily on system design even though the requirements said not to prematurely optimize and not to spend more than 4 hours on it"
But I thought it was scored by a double blind committee so what exactly are other candidates doing from scratch in 4 hours--- oh..... they're just like me but recycling code from other similar interviews two months ago and adding to it until they get a $500k compensation package
This is still happening.
Affecting the screening process is hard, as an applicant. You really either have to game the system or just be that much better.
Affecting what you apply for to maximize your likelihood to not be screened out is probably much more useful, but also much less likely to be universally applicable. You can apply for in-person local jobs, but there may not be many good ones around you. You can market to one of your less common skills, or learn some new less common skills, but those may not be in demand locally or require a lot of work.
Whenever i used to interview (years ago), 90% of older engineers were polite, more reasonable, and you could talk on and higher level... and in general they respect your time, as many understand that it is wasteful to just do stupid coding challenges just for the sake of it (especially if they have kids)
While 90% of the more junior developers are still too young to know what to look for in a good candidate. Unfortunately we are in a industry where often a 24 yo with 2 years of experience, and barely any real life experience, get to evaluate the performance of a 30+ yo, with 10+ years of experience.
This just doesn't happen in other professions such as Law or Medicine.
That is a very real, common, and true scenario.
Age and years of experience does not exclude developers from mediocrity.
I have seen countless 10+ YoE candidates with awful practices and performance.
Implying you need to be 26 or over to be a good director is ageism.
A good developer at 23, is probably much better at 30, after some real life experience in shipping many products.
A crappy developer at 23, may become better at 30, or maybe still be crappy at 30....
I'd take a good 23 yo, over a crappy 30yo, but i'd take a good 30 yo over a good 23. The good 30 will have the maturity that comes from shipping long multiple years projects, and have both successes and failures in their belt and learned from them.
So, I think experience (up to 10-15 years) makes an already good developer better....
A 30 year old might be half the developer they were at 23, due to burnout/disillusionment/less time. Developers can flounder after previously being excellent. I've seen it.
Something I've also seen a lot is developers getting promoted to levels they cannot handle, and imploding as a result (it's hard to recover from this).
Some people could not lead a project with 30 years of experience. Some people could lead a project with 0 years of experience.
Age is basically irrelevant.
Being a brilliant developer has absolutely nothing to do with being a brilliant or even effective interviewer.
Developers are not the sum of their raw coding ability. If you are a savant genius at coding but a poor communicator, you will be a terrible addition to any large organization.
Such mavericks might be effective in startup environments, but try joining a large org and start tearing up code without communicating effectively, you will be out within a month.
I'd even go as far as to say raw coding ability isn't even a developer's most important skill.
I'm a full-time developer and coding isn't even 50% of my job.
turns out my UI ideas are phenomenal, but the mere familiarity and habitual object oriented programming isn't enough if your first thought isn't to use events and presenters and cram that into a 4 hour coding stream of consciousness.
this one large prominent company didn't even tell me that they preferred it on github, as a way to further gauge how long it took and commit practices. the recruiter just said you could zip it up over email, and then the submission page on the activity website said submit the git link. but "could" meant, not recommended and creating it a git repository now wouldn't show the utility of the git protocol with one single giant commit at the end. "don't prematurely optimize but show us your best" meant show us your best. gaslight gaslight gaslight.
oh but $500K compensation packages are supposed to be hard to get and full of pretentious antipatterns, you should be happy you routinely get call backs in less than 48 hours and are considered for these levels!
k.
It is really bad. Removing yourself from it is probably a very healthy choice.
There's so much folklore, cult of personality, etc. behind all of this.
Hiring is REALLY hard. So exec's order a ramp up of head count. It falls on to managers to start the process. They either don't want to do it (why would they want to?), or they don't know how to do it...they are not technical, or technical enough, they don't have "hiring" skills, etc...
So the burden is typically dumped onto the rank & file "senior engineers" to do the dirty work. They don't know how to hire either. They don't like doing things in a "manual" way. Easy solution! Give them a coding test!
(this is tangentially related to the above fine comment :) I believe it went more like this.. The first time I saw this style of test, it was an engineering hiring group from Microsoft, testing for advanced math skills for some audio work in collaboration (?) with Apple. It was onsite at Apple (a long time ago). The first thing the interviewer said is "we want you to solve this problem right now" .. I said, I have a lot of code examples ready that are representative of my best work, and he was pushy and slightly nervous, and said "no we do not want to see that, this is the problem" .. I agreed, and I believe I did solve it awkwardly, but they refused to look at anything except the in-person test for that interview. It really surprised me, as at that time, I had never heard of this way of doing things. There was no whiteboard, it was a desk.
Sometime later, I believe Google famously used this testing method, on people from The Five university programs, of course. Google probably spun it more like a grad school situation, but I do not know that. Later, every EXEC wanted to copy Google, or wanted to learn bullying like Microsoft.
It is so obvious to me that this sort of test is partially a command-and-control exercise, giving the control role to the hiring group. Please recall there was/is a real tug-of-war between excellent engineers and companies at various points in the recent past.
You're statements alone have great truth to them, I'm just not clear how it differs from mine and why it's a rebuttal to them. Oh-well :-)
It wasn't healthy. It was competing against others that have much greater selective pressures to excel. You can see it on Blind how people from China and India talk about practicing and solving hundreds of leetcode questions to get better, and I'm just casually able to work anywhere at any time and won't be uprooted if I fail. It takes me 45 minutes to solve one problem, and if I don't really understand the concept then it takes me hours to brush up on that concept. I don't have time for that, and the "average good enough" scores in interviews are getting finer and finer.
Finance is easier and pays more. Any of us here could take the test for a FINRA license in a single week. And then use our coding skills and capital to automate most of that.
What is this deal about commit small and often?
I usually work on problems that requires a lot of thought, a lot of back and forth between solutions, a lot of a reach this state where stuff kinda work but it is not what I need yet and I am not sure it is gonna be in the final version.
Eventually I reach the final version and I commit all together, but the commit is rather big.
Do you guys come back and fix everything after? Like re-code the solution from scratch committing at the right moment. Do you just add pieces related from the final version in multiple small commits? And how do you make sure that each commit is correct? Do you re-run the test suite after each commit? While developing?
I must say that in smaller and simpler codebase than the one I usually work with it might be possible. But if you work in complex stuff that have a test suite that take hours, the approach of commit small and fast seems wrong.
Asking people to use TDD for a coding test is probably a bit much since it takes a long time to learn the ins and outs, but small commits give a little bit of the advantage and can tell you a lot: Does the candidate know how to break the problem into atomic changes? Do they commit only reachable code? Do they create simple, fast, and orthogonal tests?
The tools and options we have don't exist in a vacuum; you still need good judgement and practice to make the most of them.
I agree that TDD has enormous benefits when used with discipline, but remember to temper your expectations as to the actual purpose of tests. The fact that your tests pass doesn’t mean your app is bug free—it is only free of the bugs you were able to test for.
There’s a concise but somewhat counterintuitive quote from Dijkstra that summarizes what tests help you uncover:
”Program testing can be used to show the presence of bugs, but never to show their absence!”
How is that statement exclusive to TDD or something that makes TDD unviable?
TDD isn't about trying to pass some tests no matter how. It's supposed to make you think through the problem and define the interfaces before writing the actual code, when it's still very easy to rearrange and adapt the structure. So that the end result is not awful in the first place.
A short example. Let's say I had to write a hash function. If I were forced to write a test for it up front, I'd probably do something naive like ensuring the result of hashing("abc") was 123.
Now I start on my code. Oh, I need to return 123, easy enough. Tests pass, ship it!
If I implemented the function first, I'd then think about pitfalls -in- the code. Let's give it weird input. Let's give it empty things. Let's make sure that for 1000 random strings, the hashes are different. Things like that, because those are things I think about when -implementing- the code, and typically not prior.
I'm not saying that to be snide, that's really how my brain would work. I understand everyone is different, so I'm only speaking for myself.
This hits close to home for me, I was very recently rejected at an interview (a 1 hour pair programming session), because my approach to fixing the algorithm was 'not TDD enough'.
I think that it's really dumb to expect people to go full on TDD in an interview session, where the purpose would ostensibly be too demonstrate analytical and programming ability. Couple that with the fact that they didn't even tell me that there was going to be an emphasis on TDD, it basically amounted to wasting about 3 days of my time for the entire interview process.
It is unfair that companies can unilaterally make such demands on your time, while themselves not spending more than a few hours.
They need to do prototyping without considering all of the faults of their programming language, all of the concurrency bugs they could make, and all of the logical errors they could make.
TDD falls apart here because TDD helps you write code that does what you think it does. When you don't even know whether it's possible to implement what you want to implement, TDD doesn't help.
I think the TDD model of implementation is a lot better than most sloppy dev workflows I've seen, but I agree with OP that small commits don't work out in this case. When you aren't sure whether something is possible, you need to stay less detail-oriented. To do so, you switch from TDD mode to prototyping mode.
Personally, I have so little faith in my prototype code that I always rewrite it.
Of course, one can always rewrite the git history to lie about your process...
Tests (in my opinion) need to run continuously during development. I usually have tests run on file change - either in a terminal window or IDE. They also run on each commit when each branch is pushed.
If you have no option other than to run a test suite that takes hours, that needs fixing. You don't need to run all your tests during development, but pragmatically you need to run enough of them to give you some confidence that you haven't broken things. Some options you might have:
* Separate fast and slow tests (e.g. unit tests and integration tests). Run the fast tests frequently as you develop, and use the slow tests to sanity check what you're doing from time to time
* If there are no fast tests (e.g. everything is almost an end to end or full integration test), write some faster tests just for the feature you're working on and run those. It doesn't need to be perfect; just lay down a start that you and your team can improve on
* If all your tests are fast, but there are just a lot of them and they're so abstracted from the layer of code you're working at you can't tell which are relevant, you can add tracing statements to define a subset of tests that hit the codepaths you're changing. Run the small subset regularly.
* Spilt up your codebase - this doesn't have to be into trendy microservices if your problem domain doesn't support the use case. But you'll almost certainly be able to pull out modules or libraries into separate repos that have their own tests. Version them semantically.
As a discipline, we work across a huge range of problems and I'm wary of making assumptions about yours; but slow tests have a massive impact on how quickly you can deliver in my experience.
If you've ever played an Elderscrolls game:
do you repeatedly mash quicksave
save every milestone
save every x minutes of play
I'm sure amongst the people who code and play RPGs this will surface the different attitudes to this.
From a workflow perspective there are commits per feature/story, or commits per working unit of code (maybe TDD influenced), commit per decision, commit per unit time (commit at lunch or end of day)
Interestingly, with git you can kind of have your cake and eat it and achieve a mixture of all of those through rebasing.
From a professional point of view your git history is a valuable artifact of your development so having a clear and well documented history is a good working practice as some teams like git being a source of truth for a project.
Small and tasteful commits can be valuable in taking a newcomer through a codebase as some people like to read a codebase from its git history as it can be quite a useful contextual annotation of the evolution of the codebase - HEAD is only a projection of the final state of that and loses a lot of the journey.
So there are git commits for the reader and there are git commits for the writer. The workflows may not always be the same but they also don't necessarily need to extremely tightly coupled. Of course if you can find a happy medium of the two then you're winning.
EDIT
In the case of a hiring code test, the projects are (hopefully) small so sometimes the commits may feel a little contrived, but so is the whole exercise of a coding test.
That said if you personally value clean git history as something in your own ideals of code quality, it is worth communicating that through smaller commits in your code test for your candidate employer. Much like unit testing or documentation are signals of quality outside of the code itself.
These are part of the day job as much as slinging code or churning out features are (depending on the role)
The skilled developer is able to take a big/difficult/complex problem and break it down to small steps that translate to simple sub-problems and their solutions and map ideally nearly 1 to 1 in commits. How far you need to continue this recursive process of breaking things down depends on how big the problem is but basically you'd do it until your commits and steps are single cohesive steps. Certainly this might mean that you end up committing functionality that is not used at that point in time but will only be used by later commits. Some people might disagree with this approach and prefer to squash this but I find that this approach keeps things simpler.
For example if I'm working on adding a new feature X in my application I need to think about how to support that feature given the current components/APIs/classes/whatnot that I have in my system. Let's say that in order to implement X I need to add a new API functionality in components Z and W and then implement the new feature in a new component X.
High level this is already 3 steps. Maybe doing the actual changes for Z and W are still too large and further need to be broken down. Then there'd be more than 3 steps and subsequently also more commits.
Now often what I end up doing when I don't really quite know yet how to best go about implementing something I do all the changes at once and once it all works and I've found my way I spend a moment of reflecting what I actually did (and if there are indeed alternative ways). Once I decide to finalize the work I then go back to my changes and do the commits in steps using git chunks.
It's all about looking a head, working out mentally the roadmap in simple steps from current state to the next state and then taking those steps. I find that in my career learning this mental model and the ability to break things down has taken the longest.
How do you manage this without source control? The back and forth? Trying an experiment and then throw it away? Sounds like the perfect use case for private branches.
>I must say that in smaller and simpler codebase than the one I usually work with it might be possible.
On the other hand, if I'm working a large and complicated changeset I don't know how I'd keep the WIP organized if I couldn't use source control privately in the process of iterating on that changeset.
Doing the work in a private branch means I can frequently rebase on master so merging at the end is a cakewalk, I can also switch back to master to start a new branch to work on something else.
>reach the final version and I commit all together >Do you guys come back and fix everything after?
I make incremental commits on a private branch as I'm working. As I go it might not look like much more than an "undo" history, maybe a few abandoned branches. When I want to merge I'll do an interactive rebase. Reorder the commits to group similar work, in their implementation dependency order, then squash them into a set of logical commits and write some decent commit messages. It doesn't take too long, I do it as part of reviewing my code changes before sending them to the whole team. For example, my backburner task right now has 78 commits made over ~2 weeks, and will be squashed to about ~4 commits when I'm ready to merge.
My style is loosely based on the kernel patch series guidelines, https://www.kernel.org/doc/html/v4.17/process/submitting-pat...
People are competing for a job with these, potentially a big career step. A lot will spend every waking hour between receiving the task and the deadline working on it, because that makes sense when you're fighting for a job alongside a bunch of other people.
Anyone who actually follows the "don't spend more than a few hours on this" instruction is going to be at a huge disadvantage to the other people that almost certainly will ignore it, so as much as you try to make sure to reduce the pressure, you can't.
A start would be to time-box your interviews. Then maybe time out your own day and realize that spending 4 hours on an interview means that you miss dinner, sleep, your workout, and/or social time. I wouldn't want to force a future coworker through that mess.
So, now you've got to limit your interview to a couple of hours. Luckily, since 80% of industry is satisfied with trivia questions, you don't really need to try that hard to design an interview that helps you find better candidates than your competitors.
> That was all we asked; we intentionally left it open-ended.
> I let them know there was no time limit or minimum for how many hours they had to work on it
> But then over the next three paragraphs, they explained what they would have done.
> Normally, I would have replied with well wishes on their challenges
> When I came back, I had three replies in the email chain that just said, “Ship it.”
> Why did this work well yet was a big deviation from what we had planned?
> How did we get such a strong impression of them with very little code?
> Do we even need code? How much code? Let’s try less. Let’s try none! Do we just skip to the phone screen and talk about how they would start?
> Since then, I have moved away from coding tests altogether.
> ..that don’t require them spending a whole night coding for just a chance at an interview.
What's the painful part? That it could have been much more terse but you ended up having to read the whole thing?
If so, I kind of find your comment "painful" (not the words I would use if you hadn't) in a similar way at a high level, if for different low-level reasons. You provided very little information on what your found objectionable, and simply quoted select sentences from the the article.
I think the fact I'm not even sure I'm correctly interpreting your comment explains the problem with that.
I'm not sure the article is too long, maybe it is, but I definitely think your comment is lacking enough information to convey your point accurately.
This response (and how you responded to it) really illustrates, to me, the problem with hiring in the first place when placing those hiring choices in the hands of engineers that don't have hiring skills.
> What's the painful part?
The parts that I outlined.
> That it could have been much more terse but you ended up having to read the whole thing?
Nope.
> I'm not even sure I'm correctly interpreting your comment
Yup.
> but I definitely think your comment is lacking enough information to convey your point accurately
Right, so if it was an issue for YOU, one good choice in the future would be to engage me in a discussion. Ask a simple follow up question.
"Hey, I didn't understand your comment, what did you mean by that? I'm interested in finding out more..."
This illustrates the state of hiring in our industry. Snap, terse, misunderstandings without exploring the idea that maybe the burden is on you to get some clairity if YOU don't understand something.
"Hey, I didn't understand those lines of code, what did you indend to with that? I'm interested in finding out more..."
Typically it falls inline with that the author of this article posted, he passes out an "open-ended" coding task as a pre-interview, get's something back, most likely "replied with well wishes" and moved on. They didn't have the time, desire, skillsets, etc... to identify and engage.
Or was the issue that it wasn't asked in sufficiently conversational prose, which is the primary difference I can see between what he wrote and what you suggest?
"What's the painful part?" was the first thing I wrote.
> This illustrates the state of hiring in our industry. Snap, terse, misunderstandings without exploring the idea that maybe the burden is on you to get some clairity if YOU don't understand something.
I explained that I had a hard time understanding what your point was, I explained that it was because of your lack of explanation (you could even say it was terse, which is why I'm surprised you level that accusation at me), and I asked you to clarify.
Perhaps you're offended that I said I found it painful? I made sure to note I was only using that phrasing because you did. My purpose in doing so was to note that it's not a very constructive way to criticize. If you did find yourself ion any way offended by that, please consider how you originally presented your position, and whether that was a constructive way to do so.
> This illustrates the state of hiring in our industry.
This exchange? Maybe it does! I did everything you criticize me for here, yet you somehow failed to see it. I think perhaps there is a lesson here somewhere.
That said, I do think some of my criticisms hold weight. That is, that there wasn't enough actual content from you in your message to easily discern what you were trying to accomplish. I don't think I was the only one in that position. I did attempt to ask you what you meant, and that was genuine, I just also included my reply for what I thought he most likely purpose, which turned out to be incorrect. I'm not sure if you took that as sarcasm, or as a method to bait you, but it wasn't.
Here's to hoping we both come away from this slightly better than we entered, and with little or no resentment. :)
That's very funny to me :-)
> my original reply to you was probably a bit snarkier than called for
> At the time, I thought it warranted
> my misinterpretation of your comment
> what I thought the most likely meaning of it was
> and I intended to lightly reflect that in a way I hoped would illuminate what I saw as the problem with that approach
Wow.
I quoted a few sentences that were painful to me, and out of that came all of this?
A snarky reply, that you felt was warranted, followed up by you trying to educate (illuminate) me on why you saw it as a problem.
Throwing in some "social proof" for why all of that was okay:
> I don't think I was the only one in that position
Along with justifying most of your approach:
> That said, I do think some of my criticisms hold weight. > there wasn't enough actual content from you > to easily discern what you were trying to accomplish
Emphasizing how big of a person you truly are by putting in extra effort:
> I'm going to go out of my way to apologize
More just to explain what led to it, but you can interpret it how you like.
> Emphasizing how big of a person you truly are by putting in extra effort:
Actually, just trying to live up to the standard I set for myself, but don't always achieve. I chose an idiom that might have implied something I didn't intend. By "going out of my way" I was really just referring to how I was replying to myself, and not waiting for your reply to me, so try to short circuit that wait, and express my feelings even if you chose not to reply but still felt negatively about the exchange.
It is possible to apologize for how you did something, but not for why. This isn't some attempt and stealing a "win" in an argument through meta-analysis, it was just me deciding I had been uncharitable, and deciding to do something about it.
I'm sorry it sounds like you couldn't accept this in the spirit it was meant. It's probably not productive for us to continue further on this topic. I hope you have a good night.
Looking at the HN history, it's been a better part of a decade, if not longer, since people have been complaining about the hiring process.
It doesn't seem like much has changed except that it's worse, in the sense that you're usually asked to spend several hours on a coding project for screening AND have a subsequent whiteboard/leetcode interview or 10.
When you take into account all of the 'signals' that you're supposed to provide to even get a job, let alone be successful, maybe this isn't a good profession to be in anymore? Example:
- Several hours - several days coding projects. Some are screening tests.
- Having to practice leetcode continuously in order to keep it fresh in your head, because you never use it in actual work.
- Robust open source portfolio is highly encouraged, in some places required.
- Constantly keeping up with new libraries, frameworks and languages. It's unusual to get to learn these on the job, usually you have to do this on your own time.
Let's face it:
When you take into account all of that extra time, it seems like Software Engineers are not being paid very well.
Especially if you value your leisure time.
Why is it like this? Maybe it's just too competitive, too many Software Engineers applying for too few positions. Whatever the reason, it doesn't sound at all what it used to be, and maybe it's time to get out.
At the end of the day, I'm sure we've all met someone that justifies requiring sight of some sort of code of the candidate before hiring them. But different people will find different types of coding test better or worse ("I don't want to spend loads of my free time on a take-home project" vs "I can't code under pressure in front of a white board even though I'm great at coding in general"). So Hacker News will always be filled with endless debate trashing all forms of assessment with no agreement on what to use instead.
My experience is that almost every place I've interviewed requires both a coding project and at least one whiteboard interview, with open-source as a huge plus in your favor.
Maybe it's not an either-or thing and just requires a little bit more consideration of how it's affecting the industry as a whole.
- If someone has stellar experience on what's desired, they shouldn't have to jump through the other hoops.
- If they have a strong open source portfolio that reflects exactly what's desired, they shouldn't need the other hoops.
- If there's a whiteboard interview, don't ask about anagrams or anything that an experience dev wouldn't already be using in their day-to-day. Ask something relevant, like their understanding of databases, networking, etc if they are going to be Senior Backend dev. Ask about something a strong, experience dev actually knows that a junior doesn't. If you suspect possible gaming, then go deeper into the questions to filter out bs.
- If there's a coding exercise, keep it timed to a max of 1 hour if it's a screening test, up to 3 hours if it's a final interview. Have the project already set up for them so they're spending time on the problem and not on config. Clarify exactly what you're looking for in the results.
It's basic common sense.
I've heard on other threads a lot about being 'too old' to code. This doesn't happen in other engineering disciplines, where age is highly valued because of the experience it brings.
I'm suspecting this kind of hazing ritual to be screening out very good older devs that have a lot to bring to the table, but don't have the time to do all this extra stuff because they have a family.
A lot of these engineers work 8 hour days. A lot of 7-3 and 10-5/6 hours being worked. They learn on the job, because they understand that you aren't supposed to code for 7 hours a day. You are responsible for learning on the way. You communicate with your boss so they understand you aren't slacking off.
These engineers get cold emails from recruiters and complain about getting too many. The late 2010s are actually a golden age for SWEs (some exceptions around entry level)
1. Techno trivia on whatever stack the company was interested in.
2. Database modeling and queries.
3. Questions about past projects and the choices I made.
4. Whiteboard design questions (no coding)
5. Here is a skeletal project - do these 5 or six simple programming tasks. Use google if necessary.
6. Process, design and soft skill questions.
I have changed jobs 8 times in 24 years.
Since all of my experience is in the enterprise development/architecture space, I figured my best bet into getting into $BigTech without spending time “learning leetCode” was through their cloud consulting division. I tried my luck at AWS and I got in - no algorithms, not much techno trivia, just a lot of explaining past projects and soft skills questions on “Leadership Principles”. Also - fully remote with travel post-Covid.
To never lose at TicTacToe is not possible. If you're trying to never lose, you don't want minimax, you want a brute force of the search space
Seems like space is 2^9, but actually a few if statements is enough. You said I must never lose?? Okay I take the center (corner has same outcome, but center simplifies)
You now have 2 choices: corner, or side. Internally I will rotate the board so your choice was square 0 or 1
012
3x5
678
play(0)
if(yourplay()!=8) return play(8)
play(3)
if(yourplay()!=5) return play(5)
return play(6)
To never lose at TicTacToe is not possible
Am I missing something?
My main other point :/ :) is a full -never-loosing- AI takes 6 if statements, and minimax is technically wrong for that goal
I wonder if just asking for a one or two page design doc on something would be a much better signal, since that's probably what you'll need to do on the job anyways.
I agree that this would be a great question to answer, but I doubt it would be possible to answer at scale. I don’t trust people to give a thumbs up or down on how someone else thinks. It’s also so open to interpretation and that leaves lots of room for implicit bias. He was quickly considering all the variables. She was scatterbrained.
One alternative, focusing entirely on the code written during the interview, is much more concrete. It’s open to preparation (here’s what good and bad solutions look like) and group discussion after the fact. And it’s actually in the professional skill set (judging code) versus doing an amateur job of judging other people’s thoughts.
In fact, someone who is clear and sound in writing might get bogged down when writing code. Good writers are (obviously) not necessarily good coders.
I do think there is potential around the design doc. Design is a big part of a lot of roles. In addition, in a lot of these roles, having the "best" design isn't better than having a design that the rest of the team can understand. Still, I wouldn't do it for the purpose of identifying good "thinking". I would do it because I expect the people I hire to be good (or have potential) at what they will be expected to do.
Companies should remind themselves to help out the candidate, give them prep material, and create a safe and relaxed space if they want to see really how a person will perform in their daily job. I think that a lot of this is driven by big company ego where a high rejection rate implies a higher reputation because of course: "we only hire the best, that's why so many didn't make it" this is a total bias IMO, a sane metrics for a smooth interview process should not be based on the result of the interview solely. You should ask your self how diverse are the profile that you are able to attract, how long did they stay and how good their annual/performance review goes after 1 year, what's the feedback of their coworker and leads after 3/6 months. Getting feedback out of this and tune the interview process interactively and in a data-driven fashion, because this is what it is, a process that can and should be improved, not a sort of fetish medal to show people and (falsely) say: "Hey look how cool I am, I hire only `the best`"
Not sure what "style" means here, but this seems strange. Why does the author need to say this / why should style matter?
It sounds like the habit of the candidate about the position of braces had an influence on their hiring process, and after giving it some thought, they decided against it. This is the lowest signal one could possibly imagine.
Huh? Maybe because you only had one person to review who deviated from the norm?! StackExchange changed the way tech forums were ran by forcing everything into the Q&A format. It focuses people and prevents a lot of back and forth to understand the what the OP is asking.
Spec work sucks as a requirement in an interview SUCKs, but at least you can run it.
Why is that?
Go with your gut? Did I miss the part where the lesson learn was named? Learn what you're looking for first?
I would not hire this candidate for an IC role based on this submission. The main quality I'm looking for in an engineer is "gets things done". If this was a manager role, then sure. A manager could describe what they would have done or (preferably) what they did in a similar situation.
They were given two (!) weeks to complete a simple task. I saw no excuse for not delivering.
So you're saying that in your worldview, it is 100% acceptable for a pre-interview candidate to spend a full 2 weeks (or even a significant portion of that) coding something, for free, at the off chance they'll maybe get an actual interview?
If I want a job badly enough, I'm willing to spend 2 years preparing for a chance to have an interview there. For example, if I wanted to work at a top AI lab (like DeepMind, FAIR, or OpenAI), I'd better start working on a paper that gets accepted at NeurIPS, or a research project that puts me on their radar.
If that's not 100% acceptable in your worldview, then perhaps you're not who they're looking for.
I also wouldn't call this task simple, it's not fizzbuzz. It's going to take reasonable chunk of time even if you go into it knowing what you need to do.
If I were looking for a job, I'd welcome an extra hard test, because it's my opportunity to shine, and it reduces competition.
It is guaranteed to be screening out classes of people, like single parents, candidates who may have a full time job as they are finishing their school work, people who care for elderly parents, people with families, etc. Heck, anyone who's going through a crunch time at their work would also probably be unable to do this.
You might as well say that your company has a terrible work-life balance and we only hire people willing to destroy their health for us. Maybe that's what some companies want, but I don't think a company can grow only hiring like that.
Even the article said that by changing their test they wound up getting stronger candidates. It is clear that this kind of process does not produce the best candidates.