What we talk about when we talk about system design
maheshba.bitbucket.io
maheshba.bitbucket.io
Already it suggests a surprising idea, which may or may not be fleshed out elsewhere, that a good way to understand an existing system is to try and solve the same problem without reference to how the existing system works:
> You will often expend more cycles understanding the existing design than you would solving the problem from first principles. ...
> Even good solutions can bias your thinking towards a particular part of the design space. ... A great time to look at other systems is after the Design phase, to see if you can map those solutions to your space. Even better, you can often reverse-engineer the details of solutions simply by understanding where they fit in your design space.
If it doesn't already exist somewhere else, we can call it Balakrishnan's Law - the best way to understand a problem is to solve it!
By trying to solve it from first principles, and making some progress but most likely failing to completely solve it, I get a good sense of that the design forces and challenges are. Then, when I go back to study the state of art, I'm much better placed to understand what I'm studying.
Another good thing about this approach is that I minimize pre-biasing my thinking by what is out there now. Then, when studying existing afterward, solutions, I can more clearly begin to see gaps in them that I can direct my efforts to solving.
There is also value in early binding combined with a willingness to iterate. There is a lot of knowledge to be gained by trying to do things rather than staying on design level. I‘m making the comment not to contradict the author but to stress that exploring the design space can also mean doing things for real. The risks with a wrongly applied late biding are superficial designs and design paralysis.
Design Workflow:
- spend little time making a couple design decisions -- lots of handwaving and "we'll figure it out"
- bind these decisions, then write a tiny app to see what it looks like. What are the technical consequences? What are the business consequences (including risks)? How does this affect development?
- repeat 2-4x. The goal is to maximize speed of feedback, at the cost of minimal scope and low quality. That's fine.
- iterate more, expanding the scope and quality as needed
- you're done!
Use high-level diagrams to help focus the effort. This dramatically helps discussions with stakeholders and developers.
Source: writing a book on feedback loops
Try things, take risks, etc. The really amazing thing with software is that the cost of iteration is basically zero (+ your time). You don't even have to go buy new paints or brushes periodically. You can reset your digital canvas a billion times per day if you desire. You can even cheat and set waypoints in time that allow you to instantly teleport to any arbitrary moment with perfect recall. You can create infinite copies of your work at various stages. You can trivially blend your works together. There is no other media on earth that comes close to possessing these same attributes.
I find Developers like to have a specific Goal, then explore the space towards it. An Artist does the opposite: they make random marks and explore the marks that they "like".
Art iteration is also almost free. In fact, the "cost" of physical media (pencils and paper) helps the human brain understand and integrate what it's learning, vs digital is harder for ideas to stick.
Developers could gain by doing the same thing as artists: do random things to gain knowledge, then create a Goal and work towards it.
Recommend checking out Lynda Barry, who's written multiple books helping students unleash their creativity -- exercise https://www.youtube.com/watch?v=gtb2M2SmeuA
To me, it’s clear that designing systems is the way to maximum ongoing profit. Yet no-one I’ve come across in the world of business (still clinging on to hope) seems interested in an intentional, on-going system design process - especially when it comes to the software part, and they’re often not very good at the non-software part either.
I find myself a little bit stuck doing hands-on software engineering for companies who’ve gotten themselves into a system design hole. Companies are willing to throw tremendous resources at paying for system design mistakes, but not at avoiding them, or correcting them.
I don’t really know how to profitably find my way upward, except for becoming a manager or a consultant.
This is a huge claim. If true, then the way to profitably find your way upward is similar to what the linked article itself is doing:
Write about it, in a credible way, build your reputation as a person that can drive change leading to results, and then become a consultant.
I think systems suffer from interaction permutations complexity and the approach to look for a fundamental building block fails, because There Is No Primal Particle™. So we try introduce fundamental particle such as "everything is function", "everything is a list", "everything is a type", "everything is a file". It doesn't work. We do it to keep it in our heads but as soon as the design hits a real interaction, we have to get detailed again.
The C++ spec is enormous and very detailed. Kubernetes.
How does multithreading interact with garbage collection? How does async Rust interact with the borrow checker? How does your type system interact with - everything else? How does DNS going down interact with the rest of your cluster? How does autoscaling fail if your Docker repository is down? Can you rebuild a machine if your debian packageserver is down? How does POSIX interact with security?
What's the building block between things itself?
I am currently trying to design tooling around state space exploration and new type generation from interaction combination information by modelling interactions directly. And the fixpoint of interactions between interactions. (I am inspired by the Nothing I understand of TLA)
My dream: I am told by the computer what interactions I need to handle to handle all the cases I throw at the computer.
It works the other way around. You as the designer tell the computer which interactions it should handle and how. There are sadly no ways around that.
This is so true. It’s very common to see infrastructure being built in terms of the application that will run on it instead of designing the application to run on the infrastructure. A lot of brittle, unmodifiable messes are the result of this.
https://google.github.io/building-secure-and-reliable-system...
> hearing a lot about how distributed systems are overused and usually a monolith can do a better job etc
> improve my understanding of monolith vs microservices, when to use each, and the tradeoff of preferring one over the other
A lot of this will depend on where you're working. Where I'm at, the preference is for single tasks up to somewhere in the neighborhood of 200 gigs of RAM and commensurate CPU. Our individual servers have just stupid amounts of RAM and CPU on them, and our deployment stack has..... nontrivial amounts of overhead for.... reasons.
But if you're e.g. deploying on AWS, the price optimization point is gonna be different. And if you're deploying while working at Amazon it'll be yet a different tradeoff (cynically, having more to do with pager duty boundaries)
Except that as a developer when I hear "solution Y is just X with one change", I know that this is BS and that there won't be just that change which will be annoying because I'll have to refactor later..
Any good resources on how to properly define the design space? What makes it a space more than a list of requirements?
I could be wrong, this is, as noted, just an interpretation.
All i need is just simple webpack config to serve my customers.
The application is a monothlic. Who needs system design ?
A normal system design, derisively called a monolith by some, is much clearer and explicit. It's less code. It's more reliable, less brittle. It comes with less footguns.
Distributed systems are not a better technology, like weaving looms were, they are simply an alternative design that is presently overused by incompetent software architects.
So it's not a luddite movement, it's an anti-complexity-for-the-sake-of-complexity movement. Try to not call us luddites again. Personally I view people that advocate for such designs by default as incompetent and that they should never be let anywhere near system design as they clearly don't understand the costs.
Simpler is almost always better, and there needs to be extremely good reasons to switch to incredibly costly distributed design patterns.
The goals of system design are a) a system actually works, b) the system is not a colossal mess.
In a similar manner, the challenge of any engineering field is to not allow things to become overly complex.
> A normal system design, derisively called a monolith by some, is much clearer and explicit. It's less code. It's more reliable, less brittle. It comes with less footguns.
You're presuming that your average monolith is the result of a refined design. It is not. The main gripe that the luddite-like movement directs at systems design in general is basically the reality that software projects actually need planning and a working software architecture, where in monoliths they can just pile stuff in there without thought or criteria.
"Can" and "do" are two different words.
"Monolith" is often a cop out to hide amorphous big-ball-of-mud projects. In fact, the luddite-like movement's aversion to system design is rooted in the fact that it's not necessary, and their irrational aversion to microservices and distributed systems and systems design lies in the fact that, unlike monoliths, they are far less tolerant of hacking together big balls of mud and demand instead some degree of discipline to design a system and comply with the system design.
A demand that is often poorly met, leaving one with a distributed ball of mud that is more difficult to reason about and refactor.