Hacker News Reader

Comment by rednerrus | Hacker News Reader

We've gone away from system design interview questions on my team. We ask people to diagram something technical they understand well and the team digs in and asks questions to understand depth and breadth of the candidates understanding. For us it works much better. It's a chance to see how well candidates do in following instructions. It gives you a chance to explore depth and breadth of their knowledge on something technical that they claim to understand. Our philosophy is how well you understand something you claim to know well is indicative of the depth and breadth of your technical knowledge in general. There are plenty of opportunities in this to ask about design and get to know how people think when it comes to design. I think it helps to eliminate false negatives and false positives.

View on HN

I like this... in theory. But I don't think I'm allowed to give you a system diagram and detailed explanation of systems I've worked on, because obviously they're proprietary. That seems like a problem.

How do your candidates normally work around this? Does everyone just talk about hobby or open-source systems they've worked on?

The only really important details to keep secret are business-specific algorithms, so talking about the overall architecture of a system is usually okay, without those details. If you're building very special architectures where that actually is the business detail, then I suppose you need to figure out how to talk about something else similar or something. I just talk about the systems I worked on in generalities, so I would talk about how we get some telemetry from some devices, process it, store it, and etc., but only the mechanical details and not the meaty algorithmic parts (we use a JSON API to submit the telemetry, then we store it in a table, then we take that data in our other component and process it through our "algorithm", etc.).

Your ideas of what should be secret or not may not at all be what the candidate agreed to in their employment contract and NDA.

Exactly. Not to mention high level business logic could be inferred from system architecture.

Sure sounds like a good way to perform recon on a target company.

We say specifically “diagram a technical system you understand well.” It’s going to be challenging to work in technology if the only technical system you understand well is the one you’re currently working on and only the parts that are under NDA.

I have the opposite question from other people in the thread - this seems pretty open-ended.

Just as an example, I do bioinformatics work, and "technical system" for someone I'm interviewing (as I understand your question) might encompass any of:

- a workflow manager: how it works under the hood, common patterns for modularity/reuse or configuration

- a specific pipeline: the stages of cleaning and quantifying raw data, the rationale for various stages and the tools chosen to execute them (including how to benchmark various methods against each other)

- a specific third-party tool: theoretical details of the algorithm, practical considerations for when to apply one over another

- familiarity with common general-purpose packages or APIs (e.g. pandas, numpy, scikit-learn)

- QC: common metrics for QC'ing various experimental data, how to decide if data is "good enough", how to troubleshoot biological vs technical (lab) vs technical (computational) sources of error

- biology: technical details of an experiment, the technologies generating the data, or the underlying biology

- data analysis: how to choose and make relevant figures, any of many data-science/ML topics (e.g. clustering), connecting data to relevant domain questions

- devops: data storage/management on HPC or cloud, HPC job schedulers, any of many cloud topics (e.g. IAC, setting up a database or cloud execution of a pipeline)

I'm more curious about how you guide a candidate towards selecting a system that gives you the most relevant perspective on their thinking and skillset - do you provide any suggestions, or are there typically pretty evident choices based on the role?

For our more junior roles, we leave it more open ended because candidates may not have a similar knowledge bases to the panel. For more senior roles we advise that if they pick technologies that are pertinent to the role and likely to be understood by senior/lead/staff engineers, the process will go better for candidates and panel members.

If I were in your position, one of understanding many complex systems, I would advise you to pick where you felt both strong and the panel was likely to have some entry point into.

I literally can't diagram 90% of the stacks I've worked on in the last 8 years because they're highly NDA.

So draw the block diagram of something you worked on in school. Or, I don't know, a radio or something, if you're a hardware person.

Someone who answers, "I can't draw anything for you because everything I know about is somebody's private IP," isn't going to get the job. At least, not if I'm doing the interviewing.

You must know something about something else. Right?

Or just interview some place that doesn't give this kind of interview, like most companies. It's pretty trivial to get hired even if you can't talk about technical specifics along the lines of an architecture diagram.

Sure, I'd like to draw a vaccum cleaner, because this interview seems to suck and be unrelated to my job role. :)

"Great, that'll do nicely. Vacuum cleaners are complicated and interesting machines. What can you tell me about how a vacuum cleaner is built and how it works? Also, vacuum cleaners are pretty old tech. Is there room for further innovation in that area? If so, what angles would you explore if you worked at a vacuum cleaner company?"

This is a great way to get BS'd into thinking someone is qualified when they're not. Work sample tests are all you need.

"Awesome, then you've got nothing to worry about!"

Thanks, I can start on Monday

This is why this question is so informative. There are a ton of people in this thread arguing that they can’t describe any technologies they understand because of NDAs.

We call this out in our pre-interview handout. If you've got an NDA, choose something that isn't included in your NDA or figure out a way to describe it at a high level. We caution candidates that if we dig in and we run into something blocked by the NDA it creates challenges with our evaluation.

It doesn't necessarily have to be something that you're working on. It just has to be something that you understand well enough to describe the design of and how things are inter-connected.

>We caution candidates that if we dig in and we run into something blocked by the NDA it creates challenges with our evaluation.

So you're penalizing those under NDA and unecessarily limiting your candidate pool.

There’s no penalty. There are plenty of technologies that you’re working with that aren’t under NDA. We’re not asking you to diagram what you’re working on, we asking you to diagram something technical that you understand well.

Being able to use something you've intimately worked on as part of your job is a huge advantage since you're so familiar with it.

There have to be parts of what you work on that are not covered by NDA. Solving problems like this would be indicative of someone we would want on our team.

When I worked at Apple they made it abundantly clear that EVERYTHING was under NDA. Even things like “what version of the standard library do you use”. Or “what are computers for”. And that any minuscule transgression of the NDA would result in me Being immediately deported, divorced, and likely put in front of a firing squad.

That can’t tell you that you can’t explain how Kubernetes works or NGINX or the JVM. Any one of those things would be perfectly acceptable as a starting point.

The problem you're asking them to solve is a legal one, but you are not interviewing lawyers. I have never worked for a single employer that didn't require an NDA, and the NDAs are always written in a way that can be easily interpreted as "do not tell anyone anything about what you do here".

The question is not tell me what you do in your current job. The question is describe a technology or system that you understand well. Do you understand how the underlying systems work? You don’t have to tell me that your current company uses kubernetes to describe how kubernetes works. Your current job can’t keep you from describing how kubernetes works. Kubernetes is just an example here you could pick any technology that you’re comfortable with.

You're asking them to explain a technology they use while expecting them to tiptoe around what they're actually allowed to say about how they applied the technology. And you say this replaced your system design interviews. It sounds to me like someone explaining what Kubernetes is won't do nearly as well as someone who can explain in-depth how they applied Kubernetes to their job.

When I interviewed with apple they said pretty much everything they do is proprietary and considered secret. I imagine for some of these people working there for years it would be hard to avoid breaking nda

But, if you’re a web developer you should be able to diagram what happens when a browser makes a request

Ok yes because that’s colleges level material. The subject here is Staff Engineer (L6) interviewing. So that will require nearly all the demonstration of value to be of high level professional context.

But there’s lots of systems like this that you could diagram with arbitrary levels of detail without talking about NDAed systems

There’s no place where we are asking you to diagram the system that you’re working on. We ask you to diagram a system you understand well. Are you using messages queues, container orchestration, network components, cloud infrastructure, Java/rust/go/Python/etc.? Tell me about one of those components and how they connect together in theory vs the specifics of your system. Everyone who works in technology works with, mostly, complicated interconnected systems. The point of this exercise isn’t to design a system or tell me about a system that you’ve designed as much as it is about helping the team to understand the breadth and depth of your technology knowledge. Good engineers typically find creative ways help us understand their knowledge without disclosing NDA material.

You live in a world where most of those systems you describe are not covered by NDA. Not everyone does.

I worked at apple in the past. It would be a violation to answer your questions.

You can’t tell me how HTTPS works because of your NDA?

I feel like I'm either losing my mind or people are being deliberately obtuse now.

He's not saying they ask you to give a full overview of the way Apple architects their systems, he's saying they ask you to explain how a basic Kubernetes deployment might look, or how you might go about setting up some kind of backup system, or literally any of a million different technical things that you might have knowledge of and interest in.

How could things like this possibly be under NDA? If you weren't able to use any of this knowledge, you would be completely crippled in your work.

Nobody forced those people into that situation.

There's a lot of nobody in the context of interviewers making it expressly harder for those people to stop being that situation.

It's a reasonable hurdle; an employee who previously signed an NDA (or any one, for that matter) needs to be able to design new systems without regurgitating verbatim parts of an old one under NDA.

So it's really not a big ask to navigate their own NDA and come up with something interesting to talk about at an interview.

But it’s clearly a handicap.

It’s not a handicap. We’re not asking people to diagram their current systems. We don’t advocate for people to do that. You could just walk us through an HTTPS request and it would work, if you understood HTTPS requests well enough.

I’m going to be way more strict about confidentiality with an interviewer I don’t know at all than with a trusted colleague. For all I know you get beers every week with the CTO of my current employer’s largest competitor.

Besides which there’s a way more interesting variation on the question, which is “how would you design our product?” Got that one in my last big tech interview and quite enjoyed it.

I suspect I got my current job in part by correctly predicting (in some detail) the product they were about to launch but hadn't announced yet.

There's nothing 'unnecessary' about it. If you can't talk about technology in an interview for a software engineer, what else do you have to go on?

98% of what you'd be inclined to diagram out and explain for a system design interview isn't remotely proprietary. People are generally looking for a high-level overview of a system you've worked on to see a) that you've really worked on it and b) you can explain it such that it makes sense.

Sure, folks will ask you to drill down here and there for more detailed info, but if someone pokes on something where the details really _ARE_ proprietary, it's fine to just say that and offer to drill down somewhere else.

I'm not a lawyer, but wouldn't anything you or anyone else developed on company time at another company be owned by that company and thus proprietary?

In practice I can't imagine the spirit of any law would be violated by doing an architecture diagram, but it seems likely (to me, at least) the word of the law would be violated.

How could you ever interview anywhere if you couldn't provide details of the projects that you worked on?

I have a relative who works on very secretive stuff for big companies as a chemist.

His interviews are humorous. Those leading the interviews can't tell him what he'll be working on, and he can't tell them what he's done in the past.

The questions tend to be very theoretical, instead.

Leetcode questions?

(Don't kill me, you all. I'm joking).

Sure. But the level of description that you're giving in an interview can be pretty generic/non-proprietary. If that wasn't the case, then you really can't reuse any knowledge from job to job and that's just not so.

Sure, every company is going to have their 'special sauce' components, that ARE proprietary, but nobody's going to expect you to unpack those.

I doubt it's common, but my friend had an experience where they kept trying to direct him towards specific technical topics of particular interest to the business.

He worked for their competitor, but not in the capacity the interviewer was trying to delve into, so in addition to scummy, it was pointless and annoying.

At some point my friend cut the interview off.

Which is the proper thing to do.

> I'm not a lawyer, but wouldn't anything you or anyone else developed on company time at another company be owned by that company and thus proprietary?

In theory yes, in practice ... if that's true, why would anyone ever hire you for your experience? You wouldn't be allowed to use it.

(coming from the standpoint of a historically non-technical individual contributor who is leaning towards technical now, and asking because I want to learn):

But isn't that exactly why you as an interviewer would pose a problem, set up the context that is potentially similar to the work your project would require, and see how the interviewee navigates that?

Yes, it is. I would want to know how a person solves my problems.

And I sure would hope they bring all their experience to bear! Just because they learned about CDNs at their previous job and “everything is proprietary”, I don’t want them to suddenly forget how CDNs work.

Very little is actually unique between software businesses. We’re mostly just doing data bureaucracy.

So no one else can never build todo app or blogging system?

There is term called prior art an for me if you write bunch of props to database like text/numbers it is basically done in every other system.

Unless you build novel db system of course.

Most people barely consider this. In the old days, before leet code interviews, I had people give me code samples. One guy gave me a code sample from some telecom carrier's billing system. It wasn't good. It was also bad that he gave us a proprietary code sample, so he got no points for that either.

So, nobody can never do again anything that he/her do to become a expert?

> 98% of what you'd be inclined to diagram out and explain for a system design interview isn't remotely proprietary

Maybe we're just on different wavelengths, but literally every single thing I work on that would be a good candidate for this sort of interview is extremely proprietary...and I'm just working on regular ol' backend services at [generic big tech].

If the entirety of your web development knowledge is NDA'ed, then what good are you as a potential employee anyway?

You wouldn't be able to build anything, as all your knowledge is NDAed!

What you learned isn’t the same as what was built.

But you learned how to build things so….

Well yes, I hope someone learned how to build many things not just recreate an architecture they’ve seen before.

The flaw in this argument is assuming that you could do a good job of diagramming a system without describing the use cases of the system. My web dev skills are not proprietary, but the processes and systems I have worked on are.

The engineering knowledge isn’t under NDA but there’s no way I could fill an hour long interview about the systems I work on without getting into enough specifics about our business processes to violate NDA.

If I were asked a question like this I’d wonder if I were being tested to see how loose-lipped I am with sensitive information.

I get it...I tend to think in point solutions as well, but getting stuck there is the problem and moving past that to abstract point solutions into a systems design that talks about the architecture of the solution and the 'why' behind the design choices is what gets you out of NDA-land. None of that stuff the next level up is proprietary. Specifically how it was applied in that problem domain is.

There's a difference between being asked to design a system of the interviewers choice and being asked to design a system you know deeply.

This comment is a prime example of how the non-technical founder understands the world.

Proprietary systems are company IP. Talking about the design of the system is protected by the NDA, which the exercise described above is asking the candidate to do.

You hire for the skills to develop your own proprietary system. The employer doesn't own the skill. The employer owns anything those skills produced for the employer.

You wouldn't want your employees giving the recipe to your secret sauce away to your competitor, right? That's what everyone is talking about.

If your employees go to a competitor and make a secret sauce that tastes the same, how do you judge what is the source of knowledge?

They can have used their skills to come up with a recipe that tastes the same because the requirements led to that outcome, or they simply copied your recipe.

If somebody from OpenAI goes to a competitor and creates something like ChatGPT, where is the line crossed that shows that company IP is transferred? The employee knows how to create the system because he knows how ChatGPT works. If he recreates a similar system but doesn't say that it is a copy of ChatGPT, is that applied skill or is that a violation of the NDA?

If it is a violation, how could he forget the structure of ChatGPT to genuinely come up with a new structure?

Legislatures write rules, people enter contracts, sue for perceived breaches and courts rule on these disputes. People then adjust behaviors to get desired results without incurring legal liabilities. Results vary across jurisdiction. Such questions can't be answered in principle. California famously won't enforce non-compete clauses, and that apparently helps innovation. China caught up industrially with the West within one generation by flouting every IP rule we have here. On the other side of the debate, patent trolls...

> you hire for the skills to develop your own proprietary system. The employer doesn't own the skill.

Oh OK! Then I guess a good engineer should be able to make it through a standard systems design interview process, and shouldn't complain about it!

I am glad you agree that it is totally fine to ask engineers these types of questions, and there is no problems with NDAs that will get in the way of them demonstrating basic skills!

That was my point entirely.

Yes, I totally agree that an engineer should be able to go through a tech interview.

And the engineers who are complaining about the fact that they have an NDA are wrong, and they should stop complaining.

If the NDA is so significant that it prevents you from doing an interview, then you are worthless as an employee anyway. But really that should never happen.

I think they mean something like:

[Client]->[Cloudfront]->[Load balancer]->[ECS service in a VPC]->[RDS/other internal systems]

* Replace the above with any other cloud provider, or just commercial/open source tools that do the same job running in a VPS or whatever *

I can't see how any of this could be claimed to be protected by an NDA or similar, as this is just a highly standard architecture.

If you work on a proprietary system that has a specific architecture just don't mention that of course. I think the above allows us to discuss things pretty well, talk about introducing caching, why each layer is there, TLS termination, authentication/authorization concerns and implementation, secrets management...

It’s the difference between being asked “how would you design a Twitter clone?” and “How did you design [current employer’ssystem]?”

The first is generic (unless you happen to be a current Twitter employee). No NDA impact. The second is asking you to reveal details of a proprietary system covered by NDA (or if you’re a Fed contractor, covered by various classification laws).

What on earth are you even talking about? I can't build anything, because the stuff I have built is covered by NDA?

That would fall into the 2%. At least in my experience, the 98% number is in the ballpark. If you are working on backend services in big tech, what you are building is quite different than most software systems.

System diagram of a CRUD SAAS isn't going to be a critical business secret in the same way it might be for a company that does stream processing of marketing/advertising data off of twitter or video/radar analytics. Or a real-time fintech trading company. Many companies are a Python/Django monolith on Postgres with redis & || memcached. That system diagram isn't going to let you disrupt the online medical services industry but it's what a lot of them use.

At this point I just build on top of other take-home projects I've done and pretend like its my portfolio.

That, I must admit, is extremely tempting.

Show me a system you’ve worked on that wasn’t put together from knowledge from previous education and jobs (yours and others), documentation from vendors, stack overflow, code generators , etc.

I would hate to be a stakeholder in a company whose value depends on employees not talking about and reusing the incremental skills and knowledge gained while working there.

This is a pretty common situation in the US for folks who work on government contracts.

Yup. I’m outside DC. Somewhere near half my software engineer (and adjacent fields) friends are under contract to either the military or NSA/CIA. Most of them with TS/poly clearances. Ask one of the what they’re working on and the most you’ll get is something like “dig data on platform X”.

If platform X is something public, they could talk about the platform. But, talking about any project specifics, implementation details, etc would be a felony.

NDAs don't care where you sourced your knowledge from.

Im pretty sure I learned a full set of new skills at my current job.

I don't think knowledge on any particular api is particularly useful compare to the more generic "can you read docs and use apis" skill, which is directly applicable from something like high level hard ware work making an ftdi USB chip go brrr to making dynamoDB work nicely, but I definitely didn't know anything about dynamodb before doing a server/web job

Async distributed systems and FPGA vhdl have some overlap in how you build things, but doing an interview based on my old FPGA skills likely won't show much of my servers and databases systems design skills

In my first semester as a freshman in an American university, I took a quiz in the Calculus class that had a question based on American football. Having never played the sports or even knowing the rules all that well, I aced the exam.

Yes, it is not that hard to keep the proprietary business logic separate from the system design.

See the reason ops thing works is this can even be applied to something you haven't worked on but purely as a passion project on the side. If you are Saying "oh what if I don't have side projects" you do realize people spend/waste 3+ months going these interviews + tons of mock interviews with companies they have no intention of joining. What does it say about if you are telling me you'd rather waste valuable personal time that way instead of building something fun?

I'm fact I'd even say that even if you haven't built a side project, I'd still use op's technique and ask the candidate to talk about and white board a system they would like to build and then dive deeper from there. Heck I'd even tell this to them before an interview!

I’ll tell you precisely what it says about me:

I put in 60 hour weeks at my current job, except for a month before I change jobs, where I put in an additional 2 hours/night to prep for interviews, because that’s what will benefit me for every other company.

I care about my career and my family, so I don’t have the luxury of spending it on passion projects outside of work. I’d rather spend the precious little time I have left on my real hobbies and family/friends.

It doesn’t communicate that I’m a bad engineer. On the contrary it communicates that I give my all at work.

But I never said you had to spend all your life on passion projects. How is leet code fair game but asking how they would build something they would to even if they never did or do somehow an indication that I think you are a bad engineer? Because I see today's state of system design question interviewing pretty much like leetcode no (asking this as a faang interviewer)?

Having said that what you said about spending 60 hours at work tells others you will be a much better employee. Which is probably the signal they are looking for?

Leetcode isn't fair game either. It's a despicable practice.

Is that 6 AM - 6 PM nonstop work M-F? Or 8 hours on the weekdays and Saturday with a 12 hour sprint on Sunday?

How many of those 60 are spent in meetings?

Superhuman working hours for someone with a family.

I am not going make judgement on people's loyalty. If someone chooses to do 60 hours works more power to them. If people don't want to do side projects that is perfectly fine too. Seriously my motivation wasn't to suggest id penalize those without a side project. It was more from my observation that (from several l5-l7 candidates):

1. Despite the "let us work together" intent it is still an exam where you are penalized for taking too long or making a "wrong" assumption or missing a few bits,

2. Given above or not when confronted with a random question it can be easy to get very nervous unless you have built a similar system from 1 to 1b user scale or you have done tons of practise interviews with companies.

So apart from interviews being hard and what they are purely because of supply and demand I wanted to see if you can learn just as much from a more open book approach (with either a side project or candidate knowing the question ahead of time say the night before). It could even be more inclusive that way!

What is the vector you’re concerned about here? You sketch out a diagram so well that the company you’re interviewing with takes a photo and secretly integrates it into their stack? Then your past employer catches wind of this, realizes you were the only person in the world to understand this system, so they go after you for damages? Sorry man, I don’t see it.

I understand that you see your past work as “proprietary”, but in the real world the vast, vast majority of us do not work on systems shrouded in such secrecy. There’s nothing interesting or proprietary about CRUD infrastructure and I genuinely can’t think of a situation where you’d be exposing yourself to any real risk by explaining it to a new employer.

One place I interviewed with last year did something similar in that it was more of a BYOSD (bring your own system design). So it meant I got a chance to think through complex systems I've worked on, mock up a diagram beforehand, and then present it to the interviewer. They then drilled down on a lot of components, why decisions may have been made, etc. Sort of like what you're saying.

Out of all the similar interviews I did last year for that type of round, I enjoyed that style the most. Rather than your typical "build me an {ecommerce site, social network, video streaming site, url shortener}".

Square/Block (used to?) ask only 1 system design question and the recruiter would let you know ahead of time so you could prepare for it. Probably because it allows for deeper questioning and answering of relevant technical skills.

It’s hard to tell between those who know their stuff, and those who are good at memorising all the various courses that tell you what to say. By telling the candidate what system to design you make the latter’s job easier.

If someone can quickly learn enough about a system that they can convince a domain expert they understand it well in a deep technical discussion, they're probably a good candidate for hiring, since your business likely consists of many complex systems, and the candidate will theoretically be able to onboard faster.

Attaining high level understanding and speaking convincingly about it are entirely different skills from knowing how to build it. Interviews are always a proxy anyway, your job will never be solely comprise of convincing people of your expertise. Sooner of later you have to do exercise it somehow. Unless you're the CEO, that is. The higher you go the easier it is to make a career out of being in the right places at the right times.

I’ve seen first hand people hired who’ve aced the system design interview but then fail to get much done. Building systems in the real world is much more than a few high-level components and algorithms - it’s about all the edge edge cases, tricky stakeholders, ambiguous and changing requirements, sequencing and coordinating small pieces of work.

That candidate is the best one... the one that can understand and apply any knowledge to resolve the problem in front of him.

If you can't distinguish between those two types of people by asking additional questions about their design, are you capable of accurately evaluating them in the first place?

How did it go?

Asking for a friend who's going to have exactly that kind of interview in 10 days :)

I work for a big tech co, and our interview training explicitly says to not ask candidates about systems they have built in the past, but to ask them to build brand new systems from scratch.

This seems completely backwards to me. Is like saying "hey, do you know that relevant on the job experience that you have? we don't want to hear anything about it, here you have a made up scenario".

Ok ok, that is a bit cynical. Asking to design novel systems can give you some good insights about the candidate, or help with people who don't have experience building systems yet. Still it seems to me that asking candidates to describe real world systems that they have actually built is much more useful at checking their skills than building imaginary systems.

One argument against this is that candidates can "cheat" by preparing in depth a made up example. But how is that much different than the current approach?

I imagine a concern is that for many candidates who are under NDA with the current and past gigs, sharing detailed system design info is not something that they would be willing to do. Do you want to penalize candidates who are careful about leaking confidential IP. Given that case the fair evaluation is to start from a blank slate to see what they would build given the requirements and constraints shared in the interview.

Also big tech, we ask senior candidates both hypothetical systems and ones they have experience with.

We don’t ask people to describes systems the worked on previously. We ask them to diagram a technical system that they understand well.

If you are an expert one one system that you have built and maintained for ten years, then that’s all you know.

> asking candidates to describe real world systems that they have actually built is much more useful at checking their skills than building imaginary systems.

Except those systems are owned by other companies and I have signed agreements not to talk about them.

Few people at software companies are designing systems from scratch nowadays. If they are, they'll be gluing together third-party middleware applications like database systems, queues, proxies, load balancers, etc.

A general understanding of computer architecture, programming language theory, networking and common protocols and formats is more valuable. At that fundamental level, they won't be using recent fancy buzzwords to sound cool, but could be using their brain to come up with something original.

If you havent designed a system from scratch, or designed a major change, you arent a senior engineer.

these "system design interviews" often focus on systems at the scale of major consumer internet companies, e.g. "design youtube", "design twitter".

How many systems of that scale even exist? 100? Only the core developers of those systems would be considered "senior engineer"?

Someone isn't going to design one of these system or has designed some of these systems themselves or during an interview. However for something like YouTube one could say Ok to start simply I would have a frontend form to submit videos, which sends it to a video conversion API queue that calls back when it is completed or could be polled for progress. Now obviously what YouTube really does is much more complicated. But the follow up questions could be like OK your form works and now your video site becomes enormously successful and your video transcoder is overloaded, what would you do? Well I could parallelize the consumers of the queue to some large number, and scale based on load...

So isn't going to be something like I would architect my own massively parallel converter database and binaries written from scratch to process all the various formats which is probably closer to what is done in reality. But senior and lead engineers should indeed understand tradeoffs, parallelization, queues, data storage concerns. They are given as questions because people know from a use case view how they generally work.

When I interviewed at Facebook my system design question was for a Yelp-ish clone that would be rolled out to Facebook as a whole. The interviewer told me the design had to be scalable to 1 billion daily users on day 1.

It doesn’t take a ton of scale to hit limits in traditional 3 tier architecture. I think many many companies have similar designs but built with OSS.

If you carry this attitude, you aren't a senior engineer.

This is exactly why we do this. It gives candidates a chance to choose what they want to diagram for us. What do you know well?

> database systems

When was the last time building a database from scratch was a sane approach to building something?

In 2023 knowing which AWS database to choose, even if it’s hosted for you is probably ok for the system design level. Knowing when to use a queue or a load balancer etc is ok even if you didn’t engineer the queue. The company you’re going to probably isn’t implement their own either.

I agree that you need basic networking and protocol understanding etc still. And that being able to talk through problems conceptually is important.

> We ask people to [discuss] something technical they understand well and the team digs in and asks questions to understand depth and breadth of the candidates understanding.

This is how I conduct all of my interviews. I can't stand the gotcha-centric, pitfall-laden Jeopardy contest format of interview interrogations.

We think this is the fairest way to evaluate candidates. I had one standout interview that was very gotcha centric. I didn't think the way the interview was conducted was fair or respectful of my time. I also don't think it did a good job of assessing how well or how poorly I would have done in the job. It seems indicative of poor management. I think it's very important as managers that we respect people and their time, always. This feels like as fair of a way as we've come up with so far to do that. We do two technical exercises, diagramming being one of them. We've had really great results and it takes ~1 hour. That seems fair.

> It gives you a chance to explore depth and breadth of their knowledge on something technical that they claim to understand.

Assuming you know enough about what they're doing to actually dig in - which, granted, maybe you're only considering interviewing people who are working things you're somewhat familiar with.

Ultimately, I don't see how this is different from a technical design interview - in that you can be susceptible to hiring charlatans, and pass on people who'd rather say they don't know something than try to BS most of it on the spot to sound impressive.

The two people could have the same knowledge. You're more likely to hire the charlatan. Maybe that's what you want :shrug:

We've had very few false positives. It's a conversation and we understand and appreciate when people say they don't know. We understand that really good engineers say "I don't know how that works." If all of your answers are "I don't know how that works.", it'll be red flags enough for the team. The team of interviewers has a very wide and deep knowledge of technologies (it's kind of a virtuous cycle). We call it out in the pre-interview handout that if you pick something super obscure it'll hurt your chances as the team will struggle to ask good probing questions.

The nice thing about this is you're giving the candidate a choice about what they want to describe. Good engineers will choose appropriate topics.

Is this sort of interview common in the industry? This is the first time I've seen someone describe a technical interview I know I could pass with flying colors.

In my own experience interviewing people about systems they have worked on (as opposed to hypothetical on-the-spot design), we have been highly successful in routing out charlatans, as you put it. They very rapidly get hand-wavy and are only able to give the most shallow of answers. In a few cases they will go in-depth, but reveal truly bizarre decisions or lack of understanding of the platform.

Good senior engineers can usually go fairly deep, they are honest about where they don't have as much knowledge, and are usually candid about what was good and maybe what in retrospect was a hack or a bad decision in retrospect.

> Maybe that's what you want :shrug:

That just seems rude...but to the larger point, there are a lot of signals that you can pick up if you have the ability...if you have zero ability to read people and you can literally only understand if they are answering abstract technical questions correctly, you as an interviewer would probably better be replaced by a web form.

You’re ruling out all candidates who work on classified systems and those under an NDA

We assume that good engineers will know something about technical systems that are not directly related to the exact thing they're working on now.

Yup, the key thing is let the candidate pick -- any technical system they understand.

I have seen this process described elsewhere as 'reverse system design', and it is my preferred approach to evaluating senior candidates as well.

Feels like an Architect interview more than a Senior interview though

As a former (reformed?) non-coding architect, non-coding architects are a scourge on the industry. I know you didn't say "non-coding" but if it's a separate job, they're very likely not spending enough time coding.

Seniors should be able to build large, interconnected systems. IMO that's a basic skillset required to reach that level, and part of why I roll my eyes when I see people with 5, 4, 3 years of experience claiming to be seniors.

I somewhat take issue with this. (3+ decades of very hands on (read: coding) architecting, including some learning experiences in orbit. I've been coding code-doodling since teenage years.)

A lot of what non-coding architects traditionally brought to the table has been taken over by experts designing OSS protocols, data formats, etc. Take things like data frames that are now exploding in ML space. Seniors today, agreed, should be able to integrate (sub)systems and that may in fact be enough. But a competent systems architect (who have never touched code) should be able to also define data formats, patterns of movement of data between sub-systems (for say optimal performance), the actual computing platform considerations, etc. Also, sometimes when being too close to code, things degenerate to debates about tools, etc.

Naturally my points here gain more validity as system size (or its open-ness requirements) increase.

I always thought that non-coding architects still had at least some background in coding. What does the path of a never-had-coded architect looks like?

DBA or sysadmin, I would imagine. So more “never wrote application code” than “never wrote a single line of code for anything ever.”

Possibly effective doodling in the design space of systems? I personally hacked at untold number of experiments with different approaches, and was doodling like mad over at least a decade. I have bookshelves full of notebooks covered with object graphs, system sketches, etc. It's not just coding.

Architecture and coding are two distinct skillsets. I know that most senior developers haven't touched a book about architecture for a long time if ever but still feel competent about such topics - falling in the "You don't know what you don't know" trap. This is as problematic as the architect with rusty coding skills or who hasn't worked as a professional developer beforehand. Coding should be a small part of the architect role but this role is usually reserved for very complex systems where the majority of your work isn't anymore about coding because there is enough to fill a full time job otherwise there is no need for an architect at your company.

Why are books important? Senior engineers should be seeing design documents and code reviews that involve architecture and design patterns all the time.

Sure, there's room for external learning, especially at companies where software isn't the primary focus, but almost none of the best software architects I know spend much of their time reading about architecture.

Speaking personally, I've learned more about architecture from reading code and system design interview prep materials than I ever did from any book. The closest I found to a useful book was the "architecure of open source applications" series which is essentially reading code with a senior eng to walk you through it.

I hear the "Why are books important" pretty often when guiding developers. All of the best architects i know of spend much time honing their skills. That won't happen if all you see is just the company you're working at. The worst architects i've worked with stopped learning outside of their job tasks. It's also a step to understand that design patterns and system design are all within the core competency of the senior developer and that the role of the architect has many other responsibilities.

While I'm sure that the best employees make their entire lives about work skills, I just can't imagine that being a fulfilling life personally. If i have to spend my nights reading books about work, outside of work, then I don't want to be the best. I have other hobbies I'm more interested in than, "becoming a better employee".

Now, I do feel different about academia though. I could understand doing fundamental research and dedicating most of your living hours to those problems. But i guess I just feel like the research mission is infinitely more valuable than the corporate one. I say this having never had the chance to really contribute anything meaningful to any company I've worked for, and never having worked for a company who's mission felt personally fulfilling to me.

I'm glad that there are people who work their butts off for work in important places but I just can't imagine doing that myself and not just disintegrating with regret when I turned 65.

> I just can't imagine that being a fulfilling life personally. If i have to spend my nights reading books about work, outside of work, then I don't want to be the best. I have other hobbies I'm more interested in than, "becoming a better employee".

I'm the guy who reads books about my career (not about my work) outside of work (in my free-time) and during working hours as well. I love my career (computer science) and I love reading well-known tech/cs books. I don't do it to be the "best employee". I couldn't care less about what I do at work (I, like 99% of the people here, work for a totally useless tech company that nobody would care about if it disappeared tomorrow). What I do at work is stupid distributed systems in Go + Kafka + postgres + k8s (totally uninteresting stuff, but pays very well though). I genuinely enjoy readings books from Stevens, Kerrisk, Kleppmann, etc., just like I enjoy reading sci-fi/drama/etc. novels. I usually end up learning stuff that's actually useful at work, and once in a while I apply such knowledge at work, but I do it only for the raises and promotions.

There are people out there who doesn't give a fuck about tech companies, but care deeply about tech.

People can claim to be seniors because the gates for getting that title are pretty permissive. But even if you had the conviction to re-evaluate yourself ("do I deserve to be a senior engineer?"), you'll find tons of opinions on what it means to be senior, what knowledge they should know, and what they should bring to the table before claiming that. It's difficult to build yourself a widely accepted roadmap.

You have to watch out for people being too exclusionary though, almost getting into no-true-scotsman territory.

For instance: you can be a valuable contributor to a large, interconnected system, explain each part of it, but maybe not have the skills to build it from scratch. I would call that senior, but I wouldn't say architecting it from scratch is a basic skillset of this position. My opinion is if you could build large systems from scratch for a company, I'd say you probably need a title and pay higher than senior.

If you did want to roll architecture into a senior job, I'd imagine the pay is higher than non-architecting seniors, but it's hard to quantify the architecture contribution to the salary's bottom-line. If you took the non-coding architect's salary and added it to the senior's most of us would be clearing like 300k.

Personally, I'm pessimistic around job duties because you'll find so many companies looking for do-everything one-man armies at lower-end wages.

The nice thing about it is that it scales to whatever level you need it to. We've had junior candidates with no real Ops experience diagram synthesizers that they've created. Then we ask questions to gauge their level of understanding. We have a general idea of what it means to be at each of the clearly delineated levels on the team and we ask questions to what seems like the appropriate level. We've had candidates come in to interview for senior positions and during the interview the team drilled down to what we felt like were lead candidate levels and we offered the candidate a lead role instead of a senior.

We call out in the pre-interview handout exactly what we are going to do and what we expect and we let the candidates bring what they think is best. For senior, lead, and staff candidates we call out that bringing something that the team is familiar with as well, will be beneficial to the candidate and to the team.

It's really allowed us to do our best evaluation of candidates in a reasonable amount of time. We have two technical exercises and they take ~1 hour total. We may still be getting false negatives, but we're not getting false positives. Hire hard, manage easy, right?

Hey, Creator of the guide here

Reverse system design interviews have a a lot of untapped potential. They just started getting a bit more popular in the last year or so (however, most companies haven't caught on to the trend yet.)

You're one of the trendsetters @rednerrus

I had a "reverse" system design interview at Amazon back in the 2000s. Possibly before the modern "forward" system design interview became so popular.

I also remember circa 2012-2014 being required to conduct what are now considered typical modern system design interviews (i.e. as an interviewer) while employed at Amazon, and 90% of (even senior) candidates had no idea how to even begin to approach the "design a system to do X"-type questions. I think this is partly because far fewer people in typical software jobs were building distributed systems back then anyway, and partly because all the YouTube videos and guides like yours didn't exist yet, so nobody was doing the kind of dedicated prep and rote-learning that seems increasingly expected nowadays.

Back then, when the problem was too far from the candidate's real work experience, and they weren't willing to make educated guesses and ask clarifying questions in order to move forward, it was often a struggle (for both of us) to pivot the interview towards something the candidate could tackle and demonstrate their design experience. ("Tell me about a system you designed", i.e. the "reverse" approach, wasn't an acceptable alternative in our rubric.)

Now, just like what happened with coding interviews vs. Leetcode, it seems that a more common challenge for the interviewer is telling the difference between a candidate who is applying real experience and understanding vs. regurgitating/performing what they read in a system design interview prep guide. But that's assuming one is actually more valuable than the other, and I don't have proof of that.

To be honest, since I have to go through preparing for the standard style of interview before a job search anyway, all these novel forms seem like annoyances.

We’ve found that candidates prefer this because it doesn’t require a ton of prep. Talk about something you know well in a conversation with other engineers. Do some minor diagramming as you go to help us understand. It’s thirty minutes with no gotchas.

This sounds like an enjoyable interview, even from the applicants point of view.

Do you follow up on your decisions to see how it held up regarding false positives you did accept, if any? I mean as related to the heuristics, not performance review generally.

When doing IT people often say "Well, I don't know anything about computers", which I generally found pointless, as with about 30 seconds of talking I can tell if you know anything or not. Your story passes my smell test, as getting someone talking is 90% of telling what they know.

People with broad and deep understanding of technology can typically gauge if someone knows there stuff or not with 30 minutes of them describing a system to you and answering questions about it.

How many rounds of interviews do you typically do? What other interviews if any? Curious what good orgs are doing in this regard!

We do phone screen, team fit, technical. Technical is two part, the diagram and a group exercise debugging something. That’s it. It’s 3.5 hours total.

This is pretty cool. I’d probably have fun walking through some systems I’ve designed

hmm interesting approach, really like it. Where do you work? and are you hiring now?

I would also like to know these things.

This sounds like a type of interview that respects my time, which makes me think there is a good chance the company as a whole will respect my time and humanity as well.

This is exactly what we’re going for. It lets candidates know that we value their time and have worked as hard as we can to create an interview process that lets us gauge whether or not you’d be a good fit for the job in the least amount of time as possible.

best way of doing interviews imho. apart from anything else, it gives you a chance to show that you know how to design/implement a system that is perhaps more complex than the one you are being interviewed for

Still sounds like a system design interview to me, just a bit less structured.

We've found that candidates who better understand how things are connected make better candidates and operators. We need some way to gauge how well they understand how things are connected. This is as fair of way as we've come up with.

I totally relate to that. In the university setting, I see many students more interested in learning keywords instead of “key concepts” behind such keywords. Think of preferring to learn “k8s” instead of “resource manager”. Today, very few of them knows that etcd is one of the key components in “k8s”.

I guess this behavior is similar to the transformation that happened in other non-CS fields.

I don't know if you work in FAANG or no, but most FAANG interviews just don't work this way. So, as much as I like this style, it's just 1 company and definitely not the norm. The rest of us are stuck with design Uber, Whatsapp, crap...

I’ve done multiple interviews like this - “tell us about a system you built”

Not sure if you heard but there are a few other companies that exist outside the FAANG circle

And they pay well like FAANG? Examples?

It still looks design interview

where do you work?

I don’t like this approach because it doesn’t provide measurable or repeatable criteria for comparing candidates. It also suffers from a sort of Gell-Mann amnesia where a candidate who spouts good-sounding BS about a topic you’re not familiar with is assumed to be competent across the board.

The process is repeatable. We ask all candidates for the same role the same questions and ask them to do the same technical tasks. The other component of our technical interview has something more similar to scoring.

We’ve found that this diagram exercise is actually a lot harder to BS your way through because the expectation is that we’re going to probe into your answers and it’s supposed to be something you understand well.

Reply on news.ycombinator.com