Best engineering interview question I've gotten
quuxplusone.github.io
quuxplusone.github.io
1. grep -r '"append"' to find where it does "switch (cmd) { .. }" – append seems reasonably unique, but maybe there's a better command.
2. Modify switch to recognize "mult" command.
3. Find "append" function that does the work; copy/paste, modify to do multiplication.
4. Converting two strings to a number, multiplying the result, and returning the result as a string is a fairly trivial function.
Unless the memcached source code is hugely surprising, I might be able to get a first working version in 15 minutes or so on a good day if I don't mess up, maybe double on an average day. Add 15 more minutes to build memcached and read the question, and maybe a further 15 minutes of making it nicer/leeway.
A 3 hour limit is long enough so that no one reasonable will stress about running out of time, affecting the performance, and short enough that bozos can't easily trial/error it, ask on Stack Overflow, or whatnot.
Other than that, the stage matters a lot as well – is this the first interview question shotgunned to everyone who applies, the final one and you're hired if you pass, or something in between?
https://quuxplusone.github.io/blog/2022/01/07/memcached-inte...
A test might be useful, but I took a quick peek at the source and it really is as straight-forward as I suspected it would be: the input is already tokenized and you can just add a new entry in a large, if .. else if tree, write a multiplication function, and that's pretty much it. You can just test it by started memcached and sending it commands.
I'd whip up a quick patch, but the tar.gz is 14 years old and doesn't compile with latest clang without twiddling -W flags that don't seem to apply, and the git copy doesn't compile due to autoconf woes (sigh...) Messing around with autoconf is not my idea of fun and too much pain for a HN comment, so whatever. But you really don't need 3 hours if you vaguely know what you're doing and are a competent C programmer (which I am only barely).
The memcached protocol really is as simple as "CMD [params]", all of which is parsed for you, and this API is just "MULT num1 num2". There are no tricky bits or hidden complexities here as far as I can see.
If you read the introduction and then spend extra hours doing "other things" (that are bad), it could count against you.
In the past I've encountered companies that would just throw a multi-hour coding test at you without even giving you any idea of the role they are hiring for. If it's not worth employer's time to get the candidate interested in a position, the position is not worth applying for.
Used to be that it might be "Hey, let's have some conversations and then there's a little project..."
Now I've had multiple employers open with "Thanks for applying. As a first step, we would like you to perform this project. From there, we will let you know of next steps."
Uhh... no.
Well, see, many graduates or experienced job seekers don’t.
I tend to default to C++11 with C++[11,23]-portability.
In general, write maximally portable code without using vendor- or standard-specific features without a/some very good reason(s).
Also, C++11 is a monumental improvement over C++ from the days of rusty sharp edges of 90's C++ where one needed to follow strict standards to avoid nonobvious UB.
But what if my odds are only 10%? Then, on average, I'm going to have to apply to 10 jobs to land one. 10 x half a day is a full working week. Make that two weeks if my odds are only 5%, three weeks if they're only 3%.
That's a problem, especially if I currently have a job. I also have a life; I don't have three spare fulltime weeks to burn to try to land a new job.
I would hope Bobs Burgers aren't asking you to add a new feature to memcached in their interview process.
I’ve fixed a bug in memcached (improperly cleared buffer, null GETs resulting in zero prepend on next GET under high stress) during a five alarm fire and deployed it across a cluster in less than 30 minutes - and I would consider myself a mediocre devops hire.
If you want me to do work for you and judge me on that, I'm available as a contractor.
That’s weird. Seriously. That’s weird. Every job I have had in my 20+ year engineering career had a multihour interview panel. I’m talking like at least four separate hour long interviews all on the same day. Big company. Small company. It literally didn’t matter.
And you’re saying this has never happened to you? I have to ask, what has been your typical interview experience been, and how tiny are these places?
At some point, you have to wonder when know you’re the odd one out, but you didn’t. That’s on you.
Do better next time Tiny.
FWIW: Took me 20 minutes from start to finish, without editing tests, but including ~5 minutes taken to update the codebase so my new-ish Clang would actually compile it. I didn't touch the binary protocol, though; in a real setting you'd have to make sure the binary protocol extension would be compatible with other extensions made since this version.
I like it because the solution is simple (add an op code via find/replace), but quickly shows how a candidate approaches a new codebase.
Testing how one deals with existing code and perceived technical debt (i.e. Chesterton’s Fence) is hard to do otherwise.
Can they dive into the weeds?
Do they get stuck refactoring until realizing why some code path is the way that it is?
Praise aside, I’d hope most shops could craft an exercise that takes less than an hour!
I can't immediately imagine when you would want to multiply.
That’s more dividing than multiplying though
Cool challenge, but isn't it a bit of a red flag? I don't expect you to ask me to get familiar with a codebase, test, and implement a feature in 3 hours. Ever. It's also oddly specific, if you know what I mean...
In any given day, I might find an issue or missing feature in one of my dependencies, clone that repository, make a change, submit a pull request, and move on with my day a few minutes later. This is absolutely an important skill.
I don't expect someone to be able to become an expert with a codebase in 3 hours, or even become deeply familiar with it. I do expect someone to be able to get familiar enough to make a simple change, and to feel comfortable working in a codebase that's new to them.
I've done Josh's plan a bunch of times. It's why I prefer my dependencies to be open source. Everything is broken, but if it's open source, it's easy to fix it when it's noticed, push to production to fix it for my users, and send a PR to fix it for the world (eventually, if they take my patch; not everyone does, but I've had many accepted)
I can imagine in the real interview, there'd be more build up to the task, and a description of what the interviewer was actually looking for. That way the interviewee understands what the purpose of the exercise is and doesn't end up lost down rabbit holes.
The more I think about this idea, the more it grows on me though, if you frame it well and find a task that can be explored in the given time frame.
Do I have to modify memcached or can I just man-in-the-middle it with a python telnet client à la telnetlib?
Doesn't necessarily mean that it's a good question of course.
Note, I haven't used C in 15 years, have never looked at memcached code and in fact the last time I used memcached was before redis even existed.
It took me ~26 mins which included reading the article, googling the memcached repo, downloading all dependencies, compiling a first build, grepping for references to incr and adding mult with support for negative arguments, refactoring the guts of incr into two functions to support the changes, compiling, fixing a few compiler bugs then testing, given a few more minutes I'd probably be able to get it PR ready with a full test suite.
"and the whole thing needs to be atomic. Let’s look at how the locking works…” They spend all three hours getting deeper and deeper down various rabbit holes, and never produce anything that works. Candidates in this group don’t get hired."
The article mentions an `append` operation which is obviously not commutative; does that imply memcached needs a separate "appendable" string type? Of course not. Memcached only guarantees that the individual operations are atomic, it's the client's responsibility to avoid race conditions between multiple operations.
I begin to wonder about this programming culture. Speed at generating LOC, doesn't necessarily have to work IRL...
Really? You can't think of a single way for multiple clients to operate on the same data without racing? (Here's a hint if you're still having trouble: https://github.com/memcached/memcached/wiki/Commands#cas.)
I don't think memcache guarantees anything other than operations are atomic.
Having said that, it’s a great alternative compared to Leetcode style questions, where you’re penalized for just asking about the problem or not having the right insight from the get go, or not using the interviewer’s preferred formulation of the solution (such as using two queues for a graph search problem instead of just one).
if they are still using the exact same question, that's on them.
Do mathematiticans have to interiview like this? Medical doctors? Police officers? Rocket scientists? Chemists? Architects? I cant think off a single profession apart from the IT domain that does this.
I dont think there is a single profession that gets mocked on their interviewing strategies as much as IT still I see senior devs defending this insanity.
We either have to start requiring the same level of formal certification we require of other engineering professions or we have to deal with these types of interviews.
Previous comment on the subject: https://news.ycombinator.com/item?id=37989386
I've worked a lot in civil engineering, and honestly the worst working civil engineer I've worked with was still a lot better at their job than the worst working programmer I've worked with.
I suppose some kind of certification can set something of a minimum baseline, but I don't know – colour me skeptical.
The dichotomy lies in the fact IT and programming are not professions.
All of the professions you listed, you have to go to school for. For most of them, there is a body overseeing training, qualifications and conduct of the professionals, barring them from practicing for malpractice.
How do you evaluate Bob, a self-taught, remote developer, with no public projects, with 10 years of experience in no-name companies? You give them a real-world task and see how they do.
Even if I agreed with your premise that the prevailing paradigm is necessary for Bob (spoiler: I don't agree), I would rather it not be for Alice.
Alice might have the qualifications you listed, but never worked in fintech startups and therefore needs additional probing. Bob on the other hand might have none of those qualifications, but has worked exclusively in fintech startups, and would therefore require different evaluation.
This kind of context-appropriate evaluation ofcourse becomes prohibitively expensive when you're filling hundreds of positions per month.
I've done loads of interviews of "engineers" who could barely program at all. A short test like this can remove them completely and then you just need one discussion interview for the handful of candidates that remain.
I do agree that this probably has no value for the company. They could implement this easily themselves. Now, if they would give a different arithmetic operation to each applicant...
My take: There's some extreme sampling bias at play. Everyone knows at least one horror story where someone managed to bullshit their way to a position. Now they imagine that this must be a problem, while ignoring the vast majority of devs they've worked with that were legitimate developers.
The field of software engineering is also much more personal and opinionated than traditional engineering. This lowers the threshold for what devs. will classify as "frauds" - not because the people they accuse might actually be frauds, but because they are not up to their standards.
If your goal is to remove the bullshitters wouldn't a 5-10 minute, slightly harder than FizzBuzz, test also remove them just accurately and save everybody a bunch of time.
IT and SWE is very much unique in that they:
1) Ask highly specific technical questions
2) Require you to arrive at some optimal solution
It's very open ended and the notion of "optimal" doesn't really apply.
I was just commenting on the more general electrode approach to interviewing, which is endemic to the SWE sector.
Actually, this works in IT as well: if Donald Knuth, Yann LeCun or Zvi Gallil ever decide (however unlikely) to apply to a web shop, they'll probably get waved through on the strength of their credentials. For the self-taught J. Random Hacker, who can't show any of his old code because it's under NDA, there's the interview pipeline.
The awful thing though is that you’re not allowed to show your old employers’ code. And most of it is teamwork.
Even though the real-life work tasks could have catastrophic or extremely expensive consequences. As in shut down production on an oil rig.
I've also worked as an analyst / data scientist, and didn't see too many leetcode-like questions there either.
I think one big reason for this is that other professions have some quality assurance of their candidates through their education or certification. Whereas many companies are open to hire self-taught and similar people...yeah, I'm not 100% convinced about that either, because somehow companies will assume that a CS grad candidate can't code for shit, whereas traditional engineering companies will assume that a engineering grad candidate knows something.
It seems to be unique to IT
Architects bring a portfolio, and walk through it. And licensed architects have a majorly huge ordeal of a licensing process. After obtaining an accredited degree, you have to get signoff on performing 3,740 work hours in six areas of architecture practice (AXP, architectural experience program). You may need to work at multiple firms to get this experience, since not all firms do all types of architecture work.
You'll also need to take the ARE (Architect Registration Exam) and any state suplemental exams. The ARE is actually 6 separate exams, the shortest is 2 hours 40 minutes; total test time is almost 20 hours, but with breaks you're looking at 24 hours 20 minuted total appointment time. Most candiates prep and take one test at a time, but you've got a 5 year rolling window to pass them all, and the tests change from time to time. I'm not looking up how long the California suplimental exam takes, but I'm guessing it's at least a couple hours.
Some of the test is multiple choice stuff, but there's interview based tests and probably some practical work.
If you're thinking of becoming an architect because interviewing will be more straight forward, let me save you some of your life and say, don't do it. Anyway, most of the fun work is delegated to juniors/interns; if you want to make reasonable money, you've got to get up to partner, and then you're only doing client relations and very little design.
I took a government provided standardized test of computer knowledge to work at a school district in 1997, and it was full of useless questions about early 1980s tech. I can only imagine the depths of forgotten knowledge that would be needed for a Software Developer certification program. How many hours of debugging BASIC or bringing up a 68000 board from scratch would we all have to do in our work experience program? Or formal methods: in my 20+ year career, I've never gotten anything remotely close to a specification that could be used for formal methods, but I bet I'd need to get work hours and study up for the test.
> Type 2 looks at the problem and says, “Ah! I know just how to do this! Multiplication is just like addition, except wherever addition does +, I should do *.” So they copy-and-paste, change all the +s to *s, and they’re done in 90 minutes. Candidates in this group stand a really good chance of being hired.
> By the way, MemSQL-in-2013 was doing deeply arcane and high-performance C++11, so the fact that this challenge incidentally requires fluency in C was a plus, not a minus, for their purposes.
I think it's a good question because it heavily indexes on strong engineering fundamentals. Searching a codebase, reading code, understanding behaviour, modifying an existing codebase, blending your solution in, etc. Every strong engineer I know is very good at the things required to complete this task.
The problem is also not "fluency in C" because they require it for day to day work.
If anything sucks about this task it's that it also tests "set up a C IDE from scratch on Linux". This might reject a good candidate who nevertheless didn't recently set up such an environment. OTOH this one is easy to fix - either set something yourself or tell the candidate they will be expected to do this, so they will have time to prepare.
(And besides, "I need to replicate this addition-based functionality but for multiplication" does not seem like such an Eureka moment to me. The main difficulty for me seems to lie in navigating a big codebase, and exposing a new function on the API -- the difficulty of which depends on how well modularised the codebase is.)
> Via its incr and decr commands, memcached provides a built-in way to atomically add to a number. But it doesn’t provide other arithmetic operations; in particular, there is no “atomic multiply by ” operation.
Going from that to "maybe I should implement multiply similar to incr" is not a big enough leap to call it "Eureka" in my mind.
Previous discussion: https://news.ycombinator.com/item?id=31065143
(echo "set age 0 3600 2"; echo "mult age 10"; echo "quit") | telnet 127.0.0.1 11211 | diff expected.txt -
I don't get this comment at the end or the follow-up article? What's the difference, sure, FizzBuzz is a simpler problem, but I would imagine it being a common interview question for software engineers
To me, broadening to software engineering means things like requirements gathering, timeline estimation, prioritization, etc. If you just wanted to focus on code I'd choose things like writing programs that are extensible over writing extensions to existing programs.
Sure, it might not hit everything but it hits quite a lot of important skills.
Also, previous discussion: https://news.ycombinator.com/item?id=31065143
For Leetcode style questions, there is some amount of interaction from the interviewer, so the cost to the company is that employee's 1 hour wage. If as a company, you feel like this is an efficient and superior way of interviewing, consider giving that amount to the candidate.
But if you’re already shortlisted, I don’t feel it’s an unreasonable amount of time for a relatively basic assessment like this before they commit to paying you a significant amount of money by employing you.
Companies (as in its employees esp managers, processes, culture) do things that are not at all efficient, correct, in the best interest of the company all the time. Not to break out into another discussion - but the resent push to force employees back to the office could be seen as one such example.
that doesn't reflect my experience at all.
from the outside many weird things seem to have some deeper meaning and thought-out plan. But when you get an inside-perspective, you notice that often no meaning or plan exists, and the reason for the weird thing is something silly like "an important person made a suggestion, that someone else misunderstood, but also made it a priority", or "there was a good suggestion, but it wasn't implemented, because the person who suggested it, was in bad standing with the management".
I always wondered how workable that could be at any scale. We sometimes hear about company straight contracting small fixes to a candidate and use that experience to decide on hiring.
But most job interviews I've ever received where while being in a full time exclusive contract with my actual employer, and I wouldn't quit before getting the next job. Doing some coding interview on the side is of course fine, but the moment money gets into the mix, I feel it becomes a weird line to walk on whether I'm breaking my exclusive contract or not.
This line being iffier if I'm getting paid by a competitor in the field for the interview.
Isn't this collaboration? They've identified the codebase that needs changes, so it's not unknown to the company, just to you. They've designed the feature. They'll review your work in 3 hours.
I don't want to work at a company where it's not normal to get simple half day tasks with enough detail that you don't need further communication. I want to work independently, not be stuck communicating all day.
I would also not try and find a drummer for my band just by listening to them drumminggor three hours, instead I would like to see them performing a bit with other band members.
Granted that they would be generally undervalued in interviews, but for me a truly great interview question/interview setup is not about a clever technical question, but rather about being able to assess true on-the-job ability despite the interview setup.
Not an easy problem to solve I think.
The problem: we had a credit reset script that users complained about. Apparently it gave incorrect resets. We sanitized the db tables needed and introduced a couple more simple bugs in the data, sanitized the script and provided a couple different language adaptations with a couple added logic bugs.
The candidate was given access to the repo and it had a clean way to reset the tables. The candidate had 45 minutes to fix the issue in the associated bug report tickets.
We received so many compliments from candidates, even those who were not hired.
TL;DR: design a work sample test and give that. Base it on recent work. Give the candidate 3x the time to solve it than what it takes you or a midlevel dev on your team to solve.
Form 'Part 2':
> This is a great engineering interview problem because it pretty cleanly partitions the candidate pool into three different types:
> Type 2 looks at the problem and says, “Ah! I know just how to do this! Multiplication is just like addition, except wherever addition does +, I should do .” So they copy-and-paste, change all the +s to s, and they’re done in 90 minutes. Candidates in this group stand a really good chance of being hired.
As a non-expert in C/C++, I would definitely be of type 2, but I would be too afraid to apply for a job to work on a database engine's codebase.
For the simplest path I'm pretty sure I could figure it out. It's been a few decades since I touched C though.
Aside: currently job hunting and really hate the automated coding challenges. They all suck with too many assumptions. I'd much rather be tossed into a task like TFA in a language I don't even know.
„Let’s find the incr method and see how it works, take it as a base and work from there“
Is this not common, because some treat this here as some kind of „Eureka“ moment.
My absolutely 1 minute trash try would be to run the incr k times, see if it works and optimize from there.
To those who refuse engineering intervies that take time - what is your solution?
This is the same as take home assignments which plague the industry.
However, as technical assessments go this is very reasonable.
It’s not them getting work done for free.
It’s only very minimally contrived.
It’s not “leet”, as in it’s straight forward (you’d essentially copy and alter an existing feature) and absolutely the kind the task you could encounter during a day job there.
If I were interviewing you and you were not referred (referrals might be able forego assessments) and if you refused to do this, I would feel that your attitude was a bit unreasonable and it would make me think less of you as a candidate.
But really it’s a willing buyer willing seller scenario, so you do what you feel is important to you, and I would do the same, but don’t be surprised if it counts against you.
Don't worry, I would withdraw my application.
> But really it’s a willing buyer willing seller scenario
Yeah and my point is that I'm the seller and so should be every other candidate.
Without a strong referral beforehand, it's really you who doesn't need to worry, there would likely be no need for you to withdraw as most employers are apprehensive about making the financial and time commitment to an employee who is not willing to do even a reasonable assessment.
Also I would have been a type 1 person, mostly becasue I would have assumed there's some clever trick to it, since we're in an interview.
1. I assumed that low-level code and performance optimization was not the ask (given it was a mere three hour programming challenge with a unfamiliar codebase)
2. The very elaborate description of Incr, Decr with emphasis on atomicity and multiple clients -- was heavily leading us towards thinking about how to utilize the already implemented, stable, and presumably finely performance-optimized functionality -- rather than re-implement your own.
So I assumed it was a test of design thinking where you can apply the concept of atomicity, parallelizable operations, resolving conflicts / locks / race conditions etc.
If they are looking for talent in Type 2 space where they can do incremental edits to a codebase powered mostly by copy-paste -- I am not sure this intervew question is for roles that need the highest levels of generic problem solving needed from experienced Software Developers who will drive software design.
Also if implementing Mult was so straightforward that it can be done so thoroughly (including tests) in under three -hours by a single person that is opening the code base for the first time; Then why had this feature not been implemented already by then by the original project maintainers? I am sure I can find some use cases where there is a need to do multiplication of an in-memory numeric value say in gaming, cypto, etc. in order to justify this feature addition.
Hiring Type 1 is not a mistake unless that person is also rigid / headstrong and always goes for an overengineered solution.
So I'd also concur that someone who blindly copy-pasted a "+" and changed everything to "*", doesn't really understand the difference between + and *.
OTOH, it's 64bit and only unsigned, so yeah 64 bit gotta be enough for everyone (unlike 640KB mem)?
and besides, if addition overflows, then it behooves you to ensure your next implementation doesn't, otherwise you are just adding technical debt.
Exactly. That set off my warning bells too.
Very odd to me that you wouldn't see the question and think "multiplication is a pretty similar operation to addition, let's find the increment implementation and go from there".
Imo what he ends up calling type 1 is probably more along the lines of good engineering instead of copy and paste,which I would consider poor engineering.
Again it's not clear what is being tested for though.
This kind of problem/situation is a very common pattern. Honestly it sounds like you don't have good engineering intuition.
Type 1 engineers as he described them aren't great engineers because they tend to get stuck in analysis paralysis. They don't have good enough intuition to make the right assumptions about the problem that allow them to continue moving forwards.
Do you really need to dig into the e.g. locking mechanism? Probably not, because it should be abstracted away or be used/implemented in the incr codepath, which you can copy.
Why are you so scared about implementing new multiplication functionality? Do you not trust your ability to figure out a robust solution based on addition?
In my experience, Type 1 engineers do not tend to break problems down correctly. You need to break things down so that you can validate/invalidate your assumptions. Having vague concerns that a multiplication implementation might break tends to belie a lack of understanding and trust, since otherwise you would either have specific concerns or be able to use your intuition to make assumptions about the system.
"good engineering" is not about going through every detail of a system, it's being able to reason about problems, approaches and solutions coherently and pragmatically.
And ime, having a job hacking on Memcached, things like considering locks are extremely important for performance. And a project like Memcached, which is a general purpose network server, meaning it used with a large variety use cases and in different settings and also needs to be consistently fast, you need to carefully think through changes.
So again this question is poor imo, it's not clear what you are actually looking for with it. Do you want to see me write code that compiles and does the bare minimum or actually talk about designing a robust feature.
What's better engineering: not solving the problem, or solving the problem in a sub-optimal way that can be improved upon in future?
I don't disagree that performance is important for memcached, but your base assumption here doesn't help you make any progress. You're basically giving up from the start.
Assuming that everything you need lives in the incr/decr functionality is a reasonable assumption that lets you make progress.
If you assume that performance is important and a multiplication operator may have implications, you should be able to come up with assumptions/hypotheses about why that would be and try to prove or disprove them.
> Do you want to see me write code that compiles and does the bare minimum or actually talk about designing a robust feature
The great thing about this question is that it forces you to be specific. It's very easy to make general statements like "we need to consider performance implications" without having anything to back that up.
Why is the bare minimum insufficient here? Why do you need to deviate from the existing implementation and re-design this operation in a robust way? What makes the existing implementation unsuitable for this use case?
Those are the kind of questions I would expect to be answered by someone who is concerned about the performance implications of a multiplication operator.
I have found that strong engineers do not pose these vague, open-ended questions and concerns without being able to dig into the underlying details and be able to validate or invalidate their hypotheses.
From my hiring experience, it can be pretty hard to get bare minimum from candidates, so I've never looked for more beyond that, but I think some of the other people on my teams have looked for more in their interviews. I would rather hire someone who can write code that does what we agreed (or really, what I said) it should, but isn't great at thinking about robustness than someone who can think about robustness but can't write code that does what we agreed it should do. I've worked with lots of people who weren't great about robustness, and have a handle on how to get them up to speed. I've worked with a few people who couldn't code even though it was part of their job, and I don't know how to get them up to speed in a timely manner. I have helped maybe one or two people go from a non-coding job to a coding job, but it's a long process, and I shouldn't need to do it for someone who is supposed to know, so it needs to be tested for in the interview.
We've just completed a series of interviews and everyone below a certain age seems to think that we're pulling all kinds of tricks. During our interviews we'll ask a series of fairly simple questions about programming, operations, deployment, monitoring, all sorts of things. It's pretty clear to me that some candidates completely overthinks the problem. This was for a number of entry level jobs, so expectations aren't all that high, but we do want to make sure that people know the basics. It completely throws of a large number of people, especially those who have worked in startups and larger companies in the Silicon Valley / San Francisco areas.
Our older candidate frequently just straight up answer the question and just comment "Or where you looking for something in particular?" Normally we're not.
This is more a result of absolutely broken interview processes at other companies. I don't see us changing our interview process to include trick questions. Our questions are intended to make people talk, that's it.