I was wondering what is the best way to learn more about these distributed systems and form better mental models when dealing with them. Any resources or tips would be appreciated.
Thank you in advance! :)
https://www.youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_H...
I also enjoyed Release It! by Michael Nygard to learn about making distributed systems more resilient.
- it takes a while to read DDIA. Probably around 6 months of focused reading. Perhaps more
- one can learn a really good chunk of theoretical stuff... but probably not applicable to day to day work
- zero practical experience will be gained regarding Kubernetes, Spark, Kafka, EMR, Redis
So, I would recommend a more practical approach:
- start already reading the documentation of K8s, Kafka, Spark, etc. Choose one and go for it. I would recommend Kafka since its documentation is well written
- while reading documentation of the tooling above, one will inevitable stumble upon theoretical stuff that will not be explained in detail: that's exactly when you pick up DDIA (or similar books) and try to find the topic in the index and read it.
http://erlang.org/doc/reference_manual/distributed.html
Following, as you understand the concepts, you can dive into it‘s internals. Not by studying Erlangs compiler, but by trying to solve distributed computing by yourself. For this, at first you should learn about RPC. Then have a deeper look at sockets. Definitely, you should go with C to understand these techs. High level languages are abstracting RPC and sockets way to much.
Formal study seems to work best after real experience. I read Martin Kleppmann's Designing Data-Intensive Applications based on its inclusion in teachyourselfcs.com.[0] I did not find it useful because I had nothing to apply it to once I finished. However I don't think this will apply to you as it seems you already have some problems in mind to consider.
Even if incomplete and many times broken, the effort in attempting at least a PoC or a toy implementation has helped me immensely in understanding the challenges and limitations of each.
It’s also important to play with real, production quality software, and stress it and understand its limitations under load or failure. Distributed systems fail in surprising ways.
Distribution means different things based upon who you ask, but it largely speaks to separated across one or more networks. If you really want to understand distribution model for third order consequences, such that A gives a thing to B who gives it to C. What does that mean to A if A and C share no relationship or connection? The answer to that question differs based upon the thing that is shared.
For a more practical example I wrote this point to point file distribution application. The point to point nature of the application forces privacy in that each end point knows and trusts each other. The application is also built around a windows like GUI, so user B can easily drag and drop a file from user A to user C which could violate A’s privacy or compromise C.
If your workplace has written postmortems, read them. Ask more senior engineers about the times things went wrong. Figure out how to politely spectate, or productively involve yourself, when emergencies/incidents happen.
Try to understand the pros/cons of the choices your workplace has made (people will certainly be whining about the downsides!). How could different choices have avoided the cons? What different downsides would those choices have introduced? (Be careful not to come off as critical. It's common to see, with hindsight, that different choices would have worked out better, and sometimes anti-social to point it out).
1. Read the problem statement (or context, or motivation) in a particular system's design doc.
2. Flip through DDIA (the book others mentioned) to find relevant material, and try to decide how you would solve the problem.
3. Go back to the design doc and see what your colleagues decided.
By trying to solve problems yourself, and seeing what colleagues did in the same situation, you will learn to improve your thinking.
The good thing about doing this within your existing environment is that you can ask colleagues when you don't understand "why didn't we just use X for this system?".