A senior engineer's guide to the system design interview

Seniors should be able to build large, interconnected systems. IMO that's a basic skillset required to reach that level, and part of why I roll my eyes when I see people with 5, 4, 3 years of experience claiming to be seniors.

We call out in the pre-interview handout exactly what we are going to do and what we expect and we let the candidates bring what they think is best. For senior, lead, and staff candidates we call out that bringing something that the team is familiar with as well, will be beneficial to the candidate and to the team.

It's really allowed us to do our best evaluation of candidates in a reasonable amount of time. We have two technical exercises and they take ~1 hour total. We may still be getting false negatives, but we're not getting false positives. Hire hard, manage easy, right?

I also remember circa 2012-2014 being required to conduct what are now considered typical modern system design interviews (i.e. as an interviewer) while employed at Amazon, and 90% of (even senior) candidates had no idea how to even begin to approach the "design a system to do X"-type questions. I think this is partly because far fewer people in typical software jobs were building distributed systems back then anyway, and partly because all the YouTube videos and guides like yours didn't exist yet, so nobody was doing the kind of dedicated prep and rote-learning that seems increasingly expected nowadays.

Back then, when the problem was too far from the candidate's real work experience, and they weren't willing to make educated guesses and ask clarifying questions in order to move forward, it was often a struggle (for both of us) to pivot the interview towards something the candidate could tackle and demonstrate their design experience. ("Tell me about a system you designed", i.e. the "reverse" approach, wasn't an acceptable alternative in our rubric.)

Now, just like what happened with coding interviews vs. Leetcode, it seems that a more common challenge for the interviewer is telling the difference between a candidate who is applying real experience and understanding vs. regurgitating/performing what they read in a system design interview prep guide. But that's assuming one is actually more valuable than the other, and I don't have proof of that.

Asking for a friend who's going to have exactly that kind of interview in 10 days :)

This sounds like a type of interview that respects my time, which makes me think there is a good chance the company as a whole will respect my time and humanity as well.

It doesn't necessarily have to be something that you're working on. It just has to be something that you understand well enough to describe the design of and how things are inter-connected.

Sure, folks will ask you to drill down here and there for more detailed info, but if someone pokes on something where the details really _ARE_ proprietary, it's fine to just say that and offer to drill down somewhere else.

I would hate to be a stakeholder in a company whose value depends on employees not talking about and reusing the incremental skills and knowledge gained while working there.

Yes, it is not that hard to keep the proprietary business logic separate from the system design.

I'm fact I'd even say that even if you haven't built a side project, I'd still use op's technique and ask the candidate to talk about and white board a system they would like to build and then dive deeper from there. Heck I'd even tell this to them before an interview!

I understand that you see your past work as “proprietary”, but in the real world the vast, vast majority of us do not work on systems shrouded in such secrecy. There’s nothing interesting or proprietary about CRUD infrastructure and I genuinely can’t think of a situation where you’d be exposing yourself to any real risk by explaining it to a new employer.

The nice thing about this is you're giving the candidate a choice about what they want to describe. Good engineers will choose appropriate topics.

Good senior engineers can usually go fairly deep, they are honest about where they don't have as much knowledge, and are usually candid about what was good and maybe what in retrospect was a hack or a bad decision in retrospect.

That just seems rude...but to the larger point, there are a lot of signals that you can pick up if you have the ability...if you have zero ability to read people and you can literally only understand if they are answering abstract technical questions correctly, you as an interviewer would probably better be replaced by a web form.

We’ve found that this diagram exercise is actually a lot harder to BS your way through because the expectation is that we’re going to probe into your answers and it’s supposed to be something you understand well.

When was the last time building a database from scratch was a sane approach to building something?

I agree that you need basic networking and protocol understanding etc still. And that being able to talk through problems conceptually is important.

Not sure if you heard but there are a few other companies that exist outside the FAANG circle

Except those systems are owned by other companies and I have signed agreements not to talk about them.

The main tells are a mismatch between apparent practical experience and apparent knowledge of the design space. I always dig into a few specific technical aspects of the design as far as possible to see how deep the candidate can go.

For example, if there's a queue I'll ask what would happen if in production the queue starts to fill, and how to mitigate. Most candidates reflexively suggest making the queue bigger, some suggest adding more downstream capacity to drain faster. People with real experience will have encountered this problem and know that you have to spend time investigating to find the root cause of the backpressure, otherwise you could add capacity to random things without reducing the queue occupancy. Are your queue consumers getting throttled by some downstream service? Are your workers CPU bound? The "right" answer here is basically "we should figure out why the queue is filling up before changing anything", but very few candidates answer that way.

Another kind of mismatch is the red flag I listed, where candidates will basically list buzzwords or ideas regardless of whether or not they are related to the question I asked. For example if I ask a question where the user-facing API is very simple (ie, write opaque data to a log) then they start talking about the tradeoffs between graphql vs REST, that's a bad sign.

Caches usually reduce load on the thing behind the cache. They can sometimes reduce latency in ways that matter, but often don't. For example if the p99 requests all miss in cache, then the cache won't help.

I think it only takes an hour or so to become familiar with the structure. I can tell during the interview if that's the problem and I will be very patient in explaining what I'm looking for in that sense.

Now some interview prep might actually help a lot of people, but it will help you better present your existing knowledge, not replace it.

Big companies like Google have to sometimes choose mediocre processes that are Lindy and agreed on. System design interviews are less bad than coding interviews IMO, and they are much more scalable for Google.

I think I'm better at spotting genuine experience vs preparation than most interviewers because I've done a lot of interviews. On the other hand I don't think I can tell 100% of the time, and it's not black and white to begin with.

Another point, your specific background may not matter for L6 roles. I want to know if you can build the sort of thing you will be building in the role you're interviewing for. I don't care if you're good at building something else. This is less true for L4 and L5.

What's being called "bias" here may simply be "experience" and a fundamentally different understanding of the System Design interview's purpose.

Side note: Systems Design interviews are reserved for "senior" level candidates (L5+). It is a significant inflection point for expectations, as senior-level employees are expected to navigate through and resolve ambiguity. These interviews are not about determining if the candidate can or has solved a particular problem. When a candidate has a particular solution in mind for the presented problem, they better be prepared to explain and justify why.

(Take everything I say with a grain of salt, as there's no guarantee that an arbitrary interviewer you encounter shares the following understandings)

While Google's interviewing process for tech ladders is deliberately designed to minimize any potential interviewer bias in the process (i.e. the interviewer's role is structured to ask, as a starting point, approved questions and take notes --sometimes verbatim-- on the candidate's response for the hiring committee to make a hiring decision, the Systems Design interview is the one type of interview that does and --should-- rely on the interviewer's judgment.

The same Systems Design interview question can be given to candidates across a range of target levels.

What may be frustrating for junior candidates in particular is that unlike leetcode or cracking the coding interview questions, these questions are not intended to be or can be "completely solved." There are no specific "correct" answers. There is no book of solutions to be memorized for such questions. This is deliberate, knowing that people try to memorize answers for interviews. This does not mean a candidate can not, nor should not practice how to show their experience.

The tenets of systems design questions in engineering interviews are to:

0) Foster collaboration. The interview is about understanding how the candidate goes about solving a problem and understanding their experience solving problems (with others).

1) Give the candidate an open-ended question that is not meant to be memorizable, nor exhaustively solvable within the allotted time. This allows an interviewer/hiring committee to observe and judge a candidate's problem solving approach and experience, versus memorization.

2) Give the candidate an opportunity to show their experience -- The question and approach should be sufficiently broad to allow the candidate to surface areas where they have particular depth from their past work experience, and the interviewer to probe/explore those depths.

The interviewer's evaluation of the candidate responses and performance during a systems design interview should include the interviewer's expectations of what a candidate would at least ask or address with the presented problem. Better yet, the interviewer should include what they would expect a candidate for a given target level to address, and further specify what additional things an L+1, L+2, etc would have addressed.

Ultimately systems design interviews are not about a candidate's answers for "What" or "How" to build ______, but surfacing a candidate's judgement skills and understanding of the "Whys" along the way.

Sure, they can train you to learn Python, but they’re specifically looking for a Python developer. They’re not looking for a generic “person that can learn stuff” when there are many Python developers out there that can pick up the job day one.

In your case, you had to put in a lot of work because you essentially were switching specializations.

Having been on the "support" side of an engineering org (supporting teams building product features) the thing I've noticed is that if you don't maintain a certain "density" of people with these system design skills things start to go off the rails. People start showing up with designs resting on some very fundamental misunderstandings or designed with no eye towards simplicity and things turn dysfunctional quickly.

It's worth noting that there are also better and worse designed system design interviews. Good ones give space to demonstrate "taste" as well as technical chops.

A few years ago, I did my round of "big tech" interviews as a manager and was somewhat naive in that I did zero prep work, but I did have almost 2 decades of actually building systems. I found system design to be the "easy" and fun part of the process (including at Google). However, if I didn't have experience with scaling and redundancy in web applications, it wouldn't have been nearly as fun.

That said, once I was in the door (different big tech co), there was absolutely zero need for any of that knowledge. Now that I'm back at a startup, it's all relevant again!

This is something interesting that often is skipped due to "sterile" nature of system design questions(solve puzzle). 90% of system design problems are tightly connected with how your org works internally. Can you depend on particular team or is it better to provide requested functionality internally. This is very often neglected but this moves or breaks stuffs.

Imagine you have to build a complete traffic system architecture for the city that will start settling in a year? It will likely be ongoing thing for you, you will observe how street layout works and adjust, build new streets, make city highway from point A to point B etc. but over time. Doing this apriori is going to go horribly bad.

I mean, it's a cool thought process to go "Ooh, maybe I can store up repeated things so I don't need to repeat an expensive access process often!" It's a byproduct of modern computing hardware that we frequently have the luxury of not thinking about caching very often.

I don't think you should punish people for considering a cache. Write caching is a thing, it might just not be applicable for the system you're considering.

For your example, I would ask them to define what scale we were operating in, and what the footprint would be, etc.

Then you could start off with a more complex design (which seems like what the interviewer wanted.)

(As an aside, I think in recent years, the spirit of SEO has become more about just making good stuff and less about hacking...)

Anyway, it wasn't for SEO. It's hard to edit stuff well because we're so passionate about what we wrote here, and we probably should do another pass.

At the higher levels, they're also testing for humility.

This happens a lot.

I was once in the process for google and was asked a time series systems design problem. I said, “lets throw this in a time series db”. The interviewer had no idea what a time series’s db was. Mind you this was an interview for Google Cloud and the person worked on SPANNER. They also has been at the company for 2 years (by their own admission).

Left a sour taste.

It’s like trusting a doctor fresh out of medical school without residency.

Then we need a better test, but the gamification and badgering isn’t effective imo.

> And once in a while some genius superstar 10x programmer comes along who is too good for these silly interviews... well that's fine we don't need that arrogance either.

It’s not arrogance to not want to be judged by someone who literally hasn’t done and probably could never do what you’ve accomplished.

The unspoken secret is that most software engineers in the industry (even at FAANGS) can really only add features to existing systems. They can’t build green field.

Those are often the people with enough skill that they were put in a position that they could break it in the first place, and who provided enough value that even afterwards they were still allowed the opportunity to do it again.

Like the airplane bullet hole story.

p.s. I missed this ad hom bit (no, not the issue here)

> If you have an issue with people leveling up, good luck to you.

And BTW here computer scientists / programmers / developers are officially also not engineers.

When asked a reasonably simple question regarding expectations during interviews the google employees that were present each gave a drastically different answer(even after hearing their colleague).

The entire group covered the who spectrum of possible responses which lead me to believe that there was definitely a luck of the draw factor in who you get.

Perhaps they are more adherent to a rubric in 2023 which leads to these study guide type posts. Large company bullshit has never been my jam though so I never applied to find out myself.