Hacker News Reader

Comment by nijave | Hacker News Reader

I've always found system design interviews to be a bit... weird. Designing a complex system generally takes at least a few days (at smaller places) up to weeks or months at larger places. In addition, the solutions always seem to be biased towards solutions that fit with existing technology/knowledge (which is good).

It's not very easy to encapsulate domain knowledge and research heavy questions into a small interview slot so some of these can feel quite contrived.

View on HN

If you've done complex system design for a long time then it is possible to breeze through these interviews almost on autopilot. The key is not the "design" per se but asking questions that rapidly constrain the design space and understanding the implications of second-order effects. If you do this regularly, like I do, it feels like a straightforward mechanical process rather than something that requires hard thinking. Muscle memory. System design is largely independent of a specific app or tech stack.

I tend to burn through these interviews so quickly that we have time leftover. It was always the easiest part of the FAANG interview process for me.

Can you give an example of what you mean by 'understanding the implications of second order effects'?

That comment made me think of things like data relations that would require indexing or lookups on foreign keys/IDs. Or bottlenecks in components that perform hefty transformations due to the design of the data structures or unbalanced data sources, like where you need to merge infrequent but large chunks of data with smaller but much more numerous chunks from another async source.

Computers are closed finite systems, use of resources is a zero-sum game. You cannot optimize aspects of the system design in a vacuum because they are all competing for the same set of resources. Resources applied to the optimization of one thing is the pessimization of something else. You have to apply them where they make the biggest net impact.

A concrete example is cache replacement algorithms for storage. The ideal metric is cache hit rate i.e. the percentage of the time that the storage you were looking for is in the cache -- higher the better. Literature is full of academic algorithms that focus on improving cache hit rates under a variety of workloads.

In many real systems, the cache replacement algorithm is in the hot path. Most "efficient" algorithms in literature either have poor CPU cache locality or thrash the CPU cache, causing significantly worse performance than is offset by the marginal gains in cache hit rates. Knowing this, good cache replacement algorithms are explicitly designed to minimize CPU cache locality/thrashing problems, at the cost of slightly lower cache hit rates because it has higher throughput in real systems. In this case, allowing the CPU cache to work efficiently is more important than a better disk cache hit rate.

In complex systems, the optimal algorithm in isolation is almost never the optimal algorithm in a system. There is a global resource budget. By "second-order effects", I mostly meant understanding how subtle design choices for one part of the system will tacitly interact with the performance of other parts of the system by virtue of how they use global resources. More broadly, this is the discipline of accounting for the hardware resources available to the software at any point in time and identifying conflicts for those resources.

Hopefully that gives you a sense of what I meant.

Yes thanks, I took it to mean understanding what are likely to be the bottlenecks in a particular system and how your design assumptions might shift those bottlenecks, which seems to dovetail with what you said above. Having said that your example above is interesting, I hadn't thought about conflicts between the cache algorithm itself and the cached data in memory.

The thought of these questions, and engineering more generally, boiling down to intelligent dialog trees hit me like a ton of bricks a few months ago. I've been investing time into deeply thinking about these trees, and it has been one of those things that have made me feel like I've 'leveled up' in my career journey. It's very validating to read your comment, and it makes me feel like I'm on the right track. Aside from plenty of practice, do you have any good reference material or advice to share on improving this skill set?

Yeah, dialog trees is a good way of describing it. Every answer to a question directs the subsequent questions. After a long while, you've more or less memorized the common branches of the tree. I benefitted, accidentally, from working in wildly different systems engineering environments, which gave me intuition for large parts of the phase space e.g. exotic silicon supercomputers, massively parallel clusters, federated systems, cheap-ass computers, etc for various workloads. It gives you a better appreciation for the tradeoffs. This also requires understanding how silicon actually behaves, not idealized abstractions.

The other complementary skill is resource accounting -- knowing what everything costs in terms of bandwidth, latency, storage, compute, etc and how they interact. This allows you to look at any combination of hardware system and software workload, and quickly identify the resource bottlenecks. Again, if you do it long enough it becomes very intuitive. Old hands can accurately predict the performance characteristics of a software design on given hardware before a single line of code has been written, even if the design is novel, just through resource accounting. The application of these resources involves tradeoffs i.e. you can trade an excess of one kind of resource for another resource that is scarce with clever algorithms and architecture (e.g. classic space/time tradeoffs but more so).

Unfortunately, I don't have any reference material. I learned by doing over a very long time.

> Designing a complex system generally takes at least a few days (at smaller places) up to weeks or months at larger places

That's designing it completely. In the interview, you're not expected to produce the final result; more importantly, it's not even about the result, but about the thought process and conversation.

Please. I’ve been asked why I didn’t use specific features of kafka in my solution and “because I’ve never used it before only these other pubsub systems” was not the right answer.

The fact that somebody found a way to misuse a tool (system design interview) doesn't mean that there's something wrong with the tool itself.

That anecdote only means the interviewer was bad

The point is that the open ended question of system design can lead to nonsensical expectations - No True /Interviewer/ fallacy or not.

Without the context, it is hard to know what this means. If they are a company that heavily uses Kafka, then there could well be an expectation that you would have researched it and at least learned enough to have an informed discussion.

Other employers might expect someone who is "Senior" to know about what different languages offer. As others have said, the title "Senior" means different things to different people and not all skills are transferable.

As others have said, if they don't do interviews well, don't sweat it, you probably don't want to work there anyway!

It's a maximally open question which turns into a guessing game where you are trying to match what the interviewer has on their mind at a given point. Some of them just want you to collect requirements, which makes them feel smart because they didn't go too much into details in a short time, others want you to actually describe how you'd implement that something and you have to come up with an imaginary users, imaginary infrastructure and all the world around it just to ground your concocted solution into it.

Some (big tech) companies tell you to prepare for the test, because they try to standardize the system deaign open ended question. They end up selecting those that dedicated weeks of time learning for their version of the test. For the others it's a psychological game trying to predict what is on the mind of the person asking the questions.

This is a very negative view of what an interview is. Most interviews, however carried out, couild be described as "trying to predict the mind of the person asking the questions" but most interviewers are trying to get a realistic view of the applicant with very little objective information.

The bigger the company, the more likely people have "trained" up to try and give themselves an advantage over others. I don't think companies are necessarily looking for people who have dedicated weeks of learning, but that learning can certainly make the applicant look smarter than they might be.

As someone who does “system design” for a living - I specialize in application development, but I get pulled into the architecture side also - if you give me a set of requirements including your requirements for scalability, high availability, disaster recovery, do you need immediate vs eventual consistency, networking requirements etc, I can usually walk through and diagram an ugly (not aesthetically pleasing) whiteboard solution relatively quickly. I can get you a good diagram in a half a day. Most of the time, the actual implementation is not too far off my initial design.

Knowing how to do system design is no different than knowing how to do DS&A (which I’ve never had to do at any developer interview between 8 jobs over 25 years). It’s all about recognizing patterns.

What really amazes me about the software engineering interview process at larger companies is that they are given theoretical system design scenarios instead of asking them to walk through their real world experience.

Both in the latter part of my interviewing in the real world at small companies who needed someone with experience to help them scale and my interview for my current cloud consulting position at $BigTech, they wanted to make sure I had real world experience and understood the consequences and trade offs of my choices.

System design is the blind leading the blind. An interviewer whose never written and wouldn't recognize a "scalable architecture" before asks a candidate to design one in front of them. The candidate pretends to write a scalable architecture, and the interviewer pretends to know when it's correct.

System interview skills are best taught with a class on how to use "Microsoft Visio".

as with other things: it helps to understand the format and to know what to expect.

also, when I am asking a system design question I am not looking for a perfect design or even a design in particular. at the end of the interview I ask myself the question: given enough time could this person evolve what they have into a design that would work in the wild?

Reply on news.ycombinator.com