Becoming a dungeon master for an interview
propelauth.com
propelauth.com
I'm curious what changed around 2015-2016 that led to the current interview process. In the before times the interview was more of a technical conversation, sure they had some gotcha questions, but nothing too brutal. If you had real experience that you could talk about, maybe it was validated against references or a background check. People usually looked for a CS or adjacent degree, and that was enough. The quality of folks wasn't too different in my experience.
I really feel for those with a lot of experience that are being dropped into the fire, especially those with families who may not have the time to memorize patterns for 3 months, but are still very capable. Non-tech, low paying jobs now ask you to create a soduko solver or trap rain water within 45 minutes. There's not many places to go for those that don't 'adapt'.
Places just copied big tech companies that were already doing that for many years prior to 2015. I had leetcode style interviews well before 2010 even.
I also witnessed someone give a completely wrong (algorithm) whiteboard answer to a problem. Obviously they weren't thinking about the problem as they did it.
The problem is the sheer dumbness of those who believe it is important in the interview process
But once upon a time these leetcode interviews made a ton of sense. Just getting your program to run at all was basically a leetcode exercise.
see for example: https://arstechnica.com/gaming/2020/03/war-stories-how-princ...
I have my war stories too. It was fun but frustrating. Definitely not as productive as today.
Early 2000s internet was kind of like that too. The web was huge relative to what we had to work with and there wasn't the kind of products/services available then so asking leetcode stuff had a logic to it.
Admittedly, that's not where we are now for most programmers so it makes sense that the interview process should adapt.
So to me leetcoding is a waste of time and less fun than actually building something, having other people use what you have built. And there is infinite things to build there and learn.
Then again, I'm sure all of my leisure activities are a waste of time on someone's metric.
I probably learn more from the comments than the problems though. People post lots of interesting idioms and it's interesting to me to see how others codify various standard algorithms. Especially across languages.
It's a pretty specific scenario but it's happened to me
1. Interviewer: If you're a good software engineer, you can answer basic algorithmic questions.
2. Interviewees: Practice algorithmic questions so you appear to be a good software engineer.
3. Interviewer: People are just studying leetcode to get jobs, what can we do? Ask harder leetcode questions.
4. Other companies: Let's copy them since they're successful.
In short, the questions used to be reasonable until people specifically prepared for them. No one knew what to do about it so they just raised the difficulty, which made it even more unfair for people who don't specifically prep.
Another thing is an exercise for system design where hyperscaling is not required and the thing is actually quite simple. Many who have specifically prepared by leetcoding and reading "cracking the coding interview" 10 times over will naturally overengineer everything trying to fit this exercise to those book patterns dropping all common sense, all the while not having actually built anything meaningful.
I think these people will mostly try to rest and vest anyway. Truly passionate people will pass since they have actually built something and will understand the exercise.
The reality is a lot of systems (especially simple ones) run perfectly fine on a single server with next to no downtime and all the additional redundancies we introduce also add additional points of failures and without the scale that makes these necessary you might actually end up reducing your availability.
How did you come up with that number? I looked at the link and just one of the outages listed on January 10 was 59 minutes. That alone makes the uptime worse than 99.99% for the entire year before it was halfway through January.
(99.995% means at most 26.3 minutes of downtime per year. See https://en.wikipedia.org/wiki/High_availability#Percentage_c...)
if you click the history it says 98.452% over 365 days.
Yes we do have a DR box in another data centre now, in case of meteorite or fire.
This used to be the norm. A single hardware / software box CAN be engineered for high uptime. It's perfectly fine when we choose to go the other way and distribute software over number of cheap boxes with meh software, but I get pet-peeved when people assume that's the ONLY way :).
As to being a better candidate, there is little I doubt less* than simple observations I can make at work. We could split hairs over causality, but there is a clear distinction between the people who go home to delid their CPU and the people who have devoted their time to certs instead.
Of course there will always be those sages that dont really care much for videogames and have transcended the street knowledge, those people 1. Have better jobs to work 2. Are harder to find with little benefit.
It's hard to say for sure - both of these things are also part of being a good candidate and working well in a team.
But I do think that this experience is both because of and forms part of who I am, a certain troubleshooting & optimizing mindset and curiosity with machines. Is it strictly necessary, or will all people with this shared hobby/past be this way? I don't know. I do think its a useful or good fuzzy signal, to be best used alongside other signals.
Was it relevant for the job? Not for the job description. There were times when I used related experience to solve problems or smooth things over though, such as figuring out why a QA engineer's setup was bluescreening (faulty ram), or in having familiarity with tools built into windows for performance profiling and debugging memory & storage problems with programs
I heard this lyric at a formative time, and I’ve seen it be proven true many times. Including tech interviews. People continually seek out those signals that imply knowledge and experience and even shared culture, but those signals inevitably become too small (smaller = quicker and easier to weed people out) and then they become the very things that people practice in order to look like they have the knowledge, experience, or shared culture they need in order to get through the doors and secure the opportunities.
Then those signals get burned and the cycle starts again (in fact, in my experience the cycles concurrently overlap).
How did I do?
The function name is pretty misleading, since it looks like a constructor. Even if it's not a constructor, the functionality is pretty far away from the name you've chosen, so it's not a compelling API design either way. How would others use this function? How is this name self-documenting? The decision here is lacking consideration on all fronts.
Also, while I don't immediately recognize the exact language, the style looks off too. The function wetpaperbag is declared in all lowercase, but inside it's calling a function Quit that's capitalized? And, whether this particular language is camel-casing or snake-casing, it's clear that "wetpaperbag" isn't going to pass linting either way.
Overall, strong no hire. Next.
(/s... but only mostly, since the GP's point is that you actually can get a lot out of just asking simple interview questions, which I think we've helpfully proven out here)
(I am not saying that this strong correlation exists.)
The mistake is in refusing to have a separate job and hiring rubric for sysadmins
In my field, understanding signals and systems, probability, optimization, and numerical computing is very important.
Leetcode, in comparison, offers a more confined scope, making preparation more manageable and systematic, ultimately helping me break into software engineering.
But seriously... everything uses computers and programming ties everything together. I don't think you have to be a grand master programmer but having those skills make you a better employee.
The real pesky question is why do you think socialism, communism or the alternatives to "capitalism" wouldn't benefit from having people able to use better tools? Productivity multipliers would help "The People" and The Masters in charge of the authoritarian dictatorship alternatives to "capitalism"...
I have both EE and CS degrees, but I make literally more than double in CS-type jobs than I would back in EE. Even though back in the day, I was definitely better at EE stuff than CS stuff.
This was true very early career, and for now at least continues to be true.
---
I'm still of the opinion that software will eventually revert to the mean, comparable to other engineering compensation... some year. It probably won't happen until there's significant regulatory and liability burden that doesn't really exist in software yet. Like some in software complain about GDPR or DMA or whatever, but crank that up orders of magnitude and then add career-and-company-ending lawsuit threats on top, and then you have real engineering. Then it becomes much more expensive to do things, profit margins go way down, and pay does as well.
Who knows when that'll actually happen. For now, software is bonkers profitable.
(When software engineering becomes real engineering, I posit that it will also cease to be as highly paid)
If sounds like gatekeeping, it kind of is.
Compared to many other industries where people are required to go get expensive and time consuming "continuing education", I don't think keeping a timed flashcard deck of maybe a couple hundred items total in rotation is all that onerous.
Interviews should give information to both parties about what it would be like if the person joined.
Simulating a typical work task, as this article describes, might give you some of that. Though important to remember that's far from all that's relevant to either party.
A recently fashionable Leetcode hazing, on the other hand, is mostly "anti-pattern". It tells the company whether or not the individual has studied for that ritual specifically, and was willing to submit to it for the approval of this company (or is using this company as practice for the gameable ritual). And it tells the individual nothing except that the company thinks this is a good way to do interviews.
I have participated in interview processes where, at the last interview, the employer tries to talk the candidate out of taking the job. Discuss the problems, the warts, the intractable issues, and other things that are downsides of this job. Every organization has these.
I love this, because the employee is going to find all of the issues out anyway, and it's better for both parties if they part before employment begins if the issues are too much.
I do something related to this in the interview. Part of the interview is more like things you'd tell a colleague who was considering that: pretty accurate characterization of what you know of the work, your idea of pros, things that might or might not be cons for the person, etc.
Basically, they show up the first day, and their interactions with me are just the same (plus a little extra excitement on my part that they joined, and now able to share proprietary details but without unpleasant surprises). Six months later, and I'm still just the same (plus extra familiarity and increased mutual trust). And hopefully they also are what they seemed to be in the interview.
Though you can't necessarily be 100% candid about everything in an interview, like if you were doing an internal assessment a technology, product, vendor, or competitor. For example, there are personnel issues, company image, proprietary tech, etc. Also, you don't want to say something that ends up misleadingly quoted out of context on a jobs forum or word-of-mouth by someone with a different sense of professionalism or confidentiality, and you don't want to give ammo to a spy from a competitor.
In practice, the line of what to say seems pretty intuitive to hit, and I haven't knowingly had a problem with it. Though, when working at one company doing exotic-sounding, high-value-sounding stuff, I did have a couple candidates grill me on the tech and/or on resources. On those I try to respond something like, "Some of that is very proprietary, and unfortunately can't be talked about much right now until someone joins the team. But I can tell you some of the public information from news articles and such is A, B, and C. There's a few marketecture diagrams on our Web site, and links to some published research papers that detail those aspects better than I could."
I abstain, myself. I have too much real work that I'd like to push forward to be able justify spending weeks of free time on fake work.
That's who interviewers are looking for. Now, a case can be made that there aren't that many of those people and you might not need them anyway, but... if you can spend a few weeks to catch up that's not all bad compared to what high-end job offers in other professions demand of your resume.
The more we keep the interview process something demonstrable vs status/prestige/name-check/references, the better.
I work for a big tech company where people smell their own farts and think they are geniuses even though they work on CRUD interfaces for admin portals for Cloud. The interview process is well known.
I have seen wrong solutions made by these genius question writers in their own doc, or where they find out 4 weeks later that the time complexity of their own solution is not really what they thhink it is. Do you really think in a company with 50k+ software engineers, where many studied English (but like to conduct CS interviews like they’re CS researchers), that everyone will know and understand details like; how the expected linear time complexity of quickselect works?
It’s a nightmare; it’s blind leading the blind. It’s been completely gamified at this point and the only way to win is to play the game, maybe 10 years ago you could’ve passed on your own merit of having a CS degree and knowing basic DS&A, but I promise you that is almost the case nowhere anymore.
I'm sure you met a few flawed people like you describe. But they are not a large fraction.
It’s a reasonable filter for junior positions, but seems counterproductive if you’re looking to fill a position where experience would be valuable.
My willingness to practice leetcode isn't correlated with my interest in new work or the work any given job post describes. It's also not correlated with my willingness to invest time in exploring the position.
6 hours of leetcode practice teach me nothing about a position, the work it entails, the problems the company is grappling with, and the people I'd be in the trenches with. 6 hours in conversation or working close to real problems are higher leverage.
I have meaningful open source work with real users in the queue, and blog posts that won't write themselves. I feel like it's pretty reasonable to sort positions that don't compel low-leverage busywork that doesn't help me assess the position above those that do.
Sure, they might mean to exclude me, but I'm skeptical.
The job requires you to practice leetcode in order to get the job. If you don’t want to practice leetcode then you don’t want the job.
However if they just want knowledgeable, quality employees, some will self select out of that process.
Would you rather have an employee who wants the job, or an employee who is good at the job?
Coding is more like weight lifting sometimes then running. There are some weights that some people just can't pick up. And so if the job involves lifting things that are usually light but occasionally heavy, then a squat becomes a good measure for candidate.
But once everyone knows a big squat is the way in, they train squats, they show up in lift suits, they use amyl nitrates and the whole thing stops being a good measure.
Because on the job you don't squat in a squat rack, but pick things up where they are found, in the conditions on the ground.
Goodheart's law breaks things, because now you have a bunch of people optimized for a task that isn't exactly the job.
Companies lose another way too.
Folks who might be darn good at picking things up decide it isn't worth the focused training to get into companies that only seem to care about squatting, so they don't apply.
I think you meant ammonia. Poppers would cause an acute decrease in blood pressure which would be counterproductive during the lifting phase with a heavy weight.
find a way to simulate that I will do it some time over the next two weeks because the product manager reduced everyone's story points in half for this sprint in the grooming session, and the fact that I even did that gets me praised because nobody else on the team finished their tasks
While that guy may have an advantage over you in motivation, you might have an advantage over him in communication.
For a really unusual interview process, see how Jane Street, the trading firm, does it. They expect people to do well at this.[2]
Their actual interview process starts each candidate with a stack of poker chips. In each interview, candidates get to make bets on various things that look like financial trades. If they run out of chips, they're out.
"Is this bit of information available?"
"No"
Repeated for 30 minutes.
I've found that it's both frustrating for the candidate and doesn't give me much insight.
Such a back and forth to me is too close for comfort, even though I used to be into role-playing games back in the day.
Currently I try to use the little time I have to determine whether the person in front of me is a charlatan, so only questions that are easy to answer for someone actually having the skills advertised in their CV.
If you start by giving a little bit of context and then jump into some reasonable end of the backlog, then you get to see how they think of priorities, they get to ask you lots of contextual questions, you get negotiate with each other a thing to do, you get to investigate and/or develop a solution, and at the end of it all parties involved got lots of signal about what it’s like and what’s required to do the job. Not to mention you stand the chance of some normally useful thing getting further along too. Work with the person like they’re “consultant for a day” and see where you get to.
Why is our industry so resolute to come up with a bunch of bespoke, oblique representations of a job when the actual job is right there available to be used for assessment?
Of course, the downside is that it required me to spend four hours working on this at home, and then another several hours at the interview - but having to spend time interviewing is something that cannot be avoided, so I'd much rather the time be spent on something relevant that actually shows off my abilities, rather than playing Advent of Code with money at stake, or filling out astrological profiles.
I have bills, and free labor doesn't pay them. If an employer spends 1-2 hours talking tech, the role, experience, culture fit, and maybe a quick "yes, I can code" toy PoC -- fine, I'll volunteer. But if one interview process requires a day+ of investment...I'll pass, thanks. Ditto for whiteboard hazing rituals, whose prerequisites include memorizing algo' arcana and cute puzzles. I have personal obligations, which I won't sacrifice to practice dancing like a clown for a stranger with a fetish for reinventing bubble sort or whatever.
Either respect my time and status as your peer, or don't bother wasting both of our time. If that means fewer opportunities for me, then so be it. My deathbed reflections won't include "Damn, I regret spending so much time with loved ones instead of learning how to optimally stack rings onto three pegs."
Apologies for the rant. I'm clearly passionate about this lol
Do take-homes suck, and unfairly privilege people with more free time? Yes. Are they slightly better than the other asinine bullshit interviewers love to throw at candidates? Fucking absolutely.
As someone who struggles with ~parallelizing some tasks (especially when anxious), it would be great to have an up-front check in about how comfortable I am with the format of any technical ~challenge. (And even better to discover there's some wiggle room to accommodate.)
For example, I find it very difficult to code my way through a problem and narrate what I'm doing and take feedback while I know I'm being evaluated and the clock's ticking. If the interviewer says something important in the middle of this, I often have to put down any plates I've managed to get spinning (and maybe go offload some adrenaline) and then ask them to repeat before I can even parse the words they said, let alone figure out how they apply to the problem.
Two comments I'd make, that I hope you are already thinking about:
1. Role playing is a cringe-inducing experience for many (most?) adults. Please credit the interviewee with intelligence and don't try to actually role-play this. These are hypothetical scenarios, so discuss them as such.
2. As interviewers we are incredibly prone to seeing the things we know as obvious. When you devise an exercise, you can start to believe that everyone should easily see what you consider to be the 'right' way forward. In fact, seeing it may rely on your many years of experience in your company's culture, or your many years of experience with the interview exercise! So please, always remain skeptical of your assumptions about what is obvious, and avoid the trap of expecting candidates to see or follow the path that you think it's best. If any of your support scenarios are actually devious riddles that require one perfect question or leap, then whether candidates 'get it' or not will be random rather than indicative of better performance in-post.
Depends a bit on the how: I don't think most find fire exercises, military exercises cringy - although they are a form of roleplay?
A fire drill where the Health & Safety guy insists that we pretend it's real - yes, most people find this cringe-inducing. For more on this, see Dwight Schrute.
I don't see why simulating working a bug report need be cringe inducing... If everyone is just playing themselves.
[1] unintentional humor. I suppose there are some unfortunate people that have had to simulate helping someone with a mocha burn or similar injury. I meant "mock injury".
And which one of those do you think a tech support interview is closer to?
* Every candidate get an almost-identical interview
* The interview is not heavily dependent on the investment or natural aptitude of the interviewer
* Interviews generate data that can be scored by a rubric
* Interviews aren't scored by the interviewer, thus eliminating the intrinsic bias of being in the hot seat with the candidate
It looks different for every role. For instance, for a penetration testing role, I've run standardized interviews that involve generating and prioritizing attack surface lists and hypothetical bug lists. The output isn't "pass/fail", it's the list of places to test, the list of bugs expected, scheduling/testing budgets, and things like that. If I'm delivering the interview to the candidate, I'm presenting the panel of reviewers not my opinions on the candidate, but what they came up with.
The same approach works for general software development (we don't use it here, we have an even fussier process). You can do design exercises, you can work out the dependencies that are going to be needed for a particular complicated project, you can do estimation exercises, you can catalog likely bugs. Review some PRs together. Look at the day-to-day work your team does, and then make the exercise a model of that work.
It's not especially easy to do compared to just sitting down and asking a bunch of questions. Moreover, it's very hard to adopt it as a practice, because you have to trust the rubric (iterating on it over time, but using it enough for it to be meaningful) and not override it based on extrinsic considerations (referrals, misgivings about candidate background, &c).
But the upside is pretty obvious? Once you have it worked out, you get a repeatable process, something that is naturally geared towards iteration and improvement.
Standardize your interviews, generate data, have a panel make the decisions and not the interviewer.
You can still play the interview like D&D if you want!
In one, which I have described here before, an interviewer asked me to design a system for selling concert tickets. After half an hour of confused back and forth, it emerged that they wanted a network service that replied to requests with unpredictable random integers, which did not repeat, unless the service rebooted and then it was okay if there was an accidental repeat. Needless to say, I started with a lot of assumptions based on the idea of selling concert tickets, and I don't think it reflected poorly on me that it took so long to figure out that the real problem they were judging me by was so radically different from the one they asked me to solve.
I had another interview recently which had a similar wrinkle. The interviewer asked for a system to solve a very particular problem, which because of the particulars of the problem would involve receiving very small amounts of data from systems in the field. The interviewer asked multiple times about the cost of data ingress and data storage, and each time I deflected them, saying that it wasn't significant compared to other costs in the system. I was so focused on solving the problem as stated that it didn't dawn on me until later that the interviewer likely had a different problem in mind in which the data was much bigger, and they were evaluating whether I could be sensitive to those costs and design an economical system.
Interviews like these are very frustrating, and I don't think they will lead to great hires. You will end up hiring people who solve the problem you have in mind even though you told them to solve a different problem. These programmers will have the same biases and preoccupations as you, which might make them easy to incorporate into your team, but they will probably have the same blind spots as well, which means your team will be less prepared to solve new problems in the future.
Edit: Also I don't think a normal product requirements discovery process involves figuring out that you're actually being asked to solve a different problem in a vastly different domain. You shouldn't ask someone to build Netflix while thinking "let's see how long it takes them to realize I meant Twitter."
1: define the requirements for the job
2: devise questions that attempt to give insight into how well the candidate might meet that requirement
3: note down the candidates suitability out of ten for each question
4: TALK to the candidate in a free flowing and open discussion about their work, what they have done, how they built it, why they built it that way, their role in the project, what went right, what went wrong, what they would do differently.
5: Whether or not the candidate asks questions is a meaningless measure. What is meaningful is to sense their level of curiosity - about this job that you are offering, about computing and technology in general. Software development is hard and being good at it requires a level of curiosity. It's probably a concern if someone shows no curiosity in this job or software, though you need to be cautious about how you measure this - did you do a good enough job to explore their level of curiosity? Curiosity does not necessarily reveal itself without careful probing.
6: Discuss the actual work of this job with the candidate. Quickly run through the actual todo list of things that need to be done right now - talk through the hardest of those tasks - how would they approach it?
7: take a leap and employ the person you feel is right - do NOT go into ever deeper analysis, ever more interviews, ever more tests and checks and hoops. No matter how much interviewing you do, you probably won't really know how someone goes in this job until they've been in it 3 or more months.
8: Don't be so risk averse that you don't hire anyone. Do it based on your best guess. it might sound harsh, but do your best and be willing to fire them before their 3 month trial is up if they are not the right person.
9: Take into account enthusiasm - how much does this person want this job? And no, do NOT measure this until the end of the interview process - looking for enthusiasm in a cover letter is like wanting a girlfriend/boyfriend commitment before first date.
10: Pay more than people expect, if you can. $1K less than someone says they want/need reduces first day enthusiasm for the job, $1K more increases first day enthusiasm. Wind the numbers up and down for greater impact.
11: After the interview process is done, talk to referees, analyses interview results apples versus apples.
12: Interview gimmicks, leetcode, abstract questions should be red flags for interviewees. If you're looking for a job and find yourself in a silly interview, be willing to politely call it off and save yourself the time and hassle of dealing with a company that does not know how to recruit software developers.
The world doesn't work like that. Most companies are going to put you through the 10 round process or some other tedious process. Unless you're financially willing and able to turn down over 50% (probably a lot more actually) of job opportunities. And most people have bills, mortgages, rent, food they'd like to be able to afford. So it goes.
I don't like it either btw. I agree with you in principle.
Rest of your points are good though.
Importantly and usefully, simulations for cognitive work do not require any sort of physical fidelity, so it's really easy to set up basic scenarios like for hiring.
(The main cost in using simulation for training lies in designing scenarios that train for expertise. That takes slightly more advanced elicitation methods which fall under the umbrella of cognitive task analysis.)
I had one interview from a FinTech company which I actually thought went quite well, the next day I got a rejection stating "I need to improve my STAR technique". I just rolled my eyes thinking FFS
Another company the CTO wasn't on the call but had asked to record the last 15 minutes of the interview with questions he'd prepared, I was close to terminating the interview. If he couldn't be assed sitting in on the interview why should I answer questions the interviewer himself didn't understand. One of the questions was "What is triple D". I'm an experienced dev, fully aware there are too many acronyms in the industry but had never heard of triple D before, when I googled it after I was thinking FFS, that's just what any competent dev does by default. I guesses data driven development but admitted I wasn't sure.
Another interview the principal engineer yawned 3 times when I was talking before I even got to the half way stage, not one apology. I know it's just human nature but to not even acknowledge he was making me uncomfortable reminded me afterwards that it's probably a toxic work culture, which I've been told since is the case from people who worked there.
Thankfully at the start of the new year I was offered 3 roles, 2 of them I thought I'd screwed up the interview. The one I accepted, apart from taking on a task to review some code and raise issues with it, I was asked to describe an architecture of some system I'd worked on and enjoyed. I spoke too long, going past the interview time but didn't feel I explained the whole system.
I understand interviewing candidates is difficult, I've had to do it a few times in the past but the competency of interviewers and the process to score candidates varies wildly from organisation to organisation. A realisation for me although I was already aware of it, is the personality of the people interviewing you varies wildly, more often than not it's a good indication of the organisation itself.
After all, interviews tend to be a stressful and intense process...
With todays chatgpts, that reads like a pretty decent prep for a campaign as a DM.
Would love to see this more. There is also the inevitable future where someone publishes a book on it and it gets cargo-culted. C'est la vie.
Let the interviewer then dive into any technical challenges such a project may propose, in order to vet technical skills.
This allows a chance for candidate to show they can connect high level big picture thinking with low level technical skills.