Ask HN: Best resources on abstracting and dealing with complexity
What would be your recommendations on resources that teach on how to abstract away and cope with complexity and hierarchy in technical domains?
Had a major impact on how I build software
I also keep a very sort overview of imho good resources on Problem Solving Methods at https://www.bm-support.org/problem-solving-methods/
Relational modeling can be the ultimate answer for wrangling complexity. It supports real world messy things like circular dependencies without a hitch. You can achieve the same in OOP, but then go and try to serialize or persist an instance of such a thing... you will find a very unhappy variety of dragon lurking in that stack trace.
If considering not just software, but engineering or systems overall, best of all I seen is Goldratt, Theory of Constraints (usually named TOC).
Resources (mostly books) are many, they even created their own accounting system and simulation software. For intro good books The Goal (1,2,3), they are for different markets and shows look on different aspect of TOC, and easy to read.
The Goal 2 https://www.amazon.sg/Goal-2-Eliyahu-M-Goldratt/dp/818598410...
People name "The Goal 3" https://www.amazon.com/Necessary-But-Sufficient-Eliyahu-Gold...
"The Theory of Constraints International Certification Organization (TOCICO) is an independent not-for-profit incorporated society that sets exams to ensure a consistent standard of competence"
https://en.wikipedia.org/wiki/Theory_of_constraints#Certific...
Also exist lot of business organizations (official), who provide support on implementing TOC best practices, and sure, non-formal fan-clubs and lists of books and lists of other support files (learning software, accounting software).
1. Simple Made Easy by Rich Hickey (video https://www.infoq.com/presentations/Simple-Made-Easy/)
2. Thinking in Systems by Donella Meadows (pdf https://wtf.tw/ref/meadows.pdf)
3. Raymond Hettinger’s content about CHUNKING is classic for this (one example video https://youtu.be/UANN2Eu6ZnM?si=TnLLj1CASwubkAS2)
4. The classic tome on Abstraction is Gödel, Escher, Bach: an Eternal Golden Braid, by Doug Hofstadter (https://en.m.wikipedia.org/wiki/Gödel,_Escher,_Bach)
5. Finally, I’ll cut myself at lucky number five by pointing you toward Jüergen (You-Again!) Schmidhuber’s fascinating webpage: https://people.idsia.ch/~juergen/
Honorable mention to Margaret Hamilton’s T-Maps and F-Maps. Use a hierarchical numbered outline.
Zoom in and out of the fractal network. Remember your working memory holds seven (7) concepts, plus or minus two (2) … so chunk size around five (5) or less is ideal for cognitive accessibility.
Anywho, this is arguably the overarching reason Design Rules: The Power of Modularity[1] was written, at least in the abstract way you are asking it.
The book tells you about the process that led to the creation of IBM's System/360, and it shows how a set of patterns and rules have converged in parallel industry sectors. It was honestly one of the most interesting books I've ever read, and that's exactly what it is about. Highly recommend this book to virtually anyone on this board.
[1] ISBN10: 0262024667
data:text/plain;base64,aHR0cHM6Ly96bGlicmFyeS50by9maWxlZG93bmxvYWQvZGVzaWduLXJ1bGVzLXZvbC0xLXRoZS1wb3dlci1vZi1tb2R1bGFyaXR5LTA
Not a converted EPUB, either. ^^
the concepts here are re: functional programming, but they apply to everything tbh.
whether your system is a single monolith or a bunch of microservices, the core ideas hold true: think about your system in the form of actions (mutations), calculations/computations (reads), and data as data.
To that end, I heavily stress that there is no single "data as data" view of any data in the system. It is all a transformation of some sort from somewhere.
This is not to say that I disagree with your general statement. I would stress that the same "data" can have multiple representations in the "data" world. (This is not uncommon for outside of programming, either. Easy to consider that 1/3 is .33333... or 3.3...e-1, etc. Canonical formats for any data item are things you have to work to build, they do not come for free.)
If you can think in data, as you say the underlying mechanisms for storing and retrieving that are less important. They are when it comes to performance and persistence guarantees, but if you abstract things correctly, that becomes easier.
To add to my initial comment... Two things that come to mind as vital to a healthy system: SOLID principles (https://en.wikipedia.org/wiki/SOLID) and Clean Architecture (https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-a...), and Hexagonal Architecture (https://en.wikipedia.org/wiki/Hexagonal_architecture_(softwa...) come to mind.
Recently I saw a post here that clean architecture was dead. This is hilarious to me and not true. If you are writing a low-level script, sure you can avoid that and optimize ... but for larger systems (ie, anything that is outside of a single process), clean+hexagonal is the way to go.
It deals with more than just software, as it can be applied in many systems.
This textbook, an introduction to the principles and abstractions used in the design of computer systems, is an outgrowth of notes written for 6.033 Computer System Engineering over a period of 40-plus years. Individual chapters are also used in other EECS subjects. There is also a web site for the current 6.033 class with a lecture schedule that includes daily assignments, lecture notes, and lecture slides. The 6.033 class Web site also contains a thirteen-year archive of class assignments, design projects, and quizzes.
https://ocw.mit.edu/courses/res-6-004-principles-of-computer...
And learn TLA+. The core idea of TLA+ is that any problem, how ever complex can be represented as a state machine i.e a sequence of variables or states that change over time.
It can be used to describe software, hardware and theoretical concepts like consensus algorithms.
I suspect one low hanging fruit for TLA+ is to use it for describing physical phenomena gravity, magnetism, boiling etc. This I haven't seen work using it for that yet.
I will never forgive my college professors for not teaching this to me. Many years later, when I discovered SICP on my own, I recognized that my high-school CS teacher wanted expose me to this. I was too young at the time to understand. Bummer.
SICP is about managing complexity.
Erlang actors excel in part because they are structured in supervision trees. There's that word again.
If you can, structure things as a tree. If it is not possible, make a DAG. Only use arbitrary graphs as a last resort.
This is why so many OO codebases turn into a mess, every object has references to who knows what, forming a complex dependency graph full of action at a distance.
You're describing a major part of Software Engineering. Most reasonable university/college SE 101 courses will focus here. Which is to say, if you haven't (or it's been a while), check reading lists for university SE classes and pick the ones that focus on modeling. steer clear of process/agile material.
Commonly used and recommended books would be GoF (if you're not familiar with the patterns already), Code Complete, and The Pragmatic Programmer.
https://hiandrewquinn.github.io/til-site/posts/the-rule-of-f...