Yeah, dialog trees is a good way of describing it. Every answer to a question directs the subsequent questions. After a long while, you've more or less memorized the common branches of the tree. I benefitted, accidentally, from working in wildly different systems engineering environments, which gave me intuition for large parts of the phase space e.g. exotic silicon supercomputers, massively parallel clusters, federated systems, cheap-ass computers, etc for various workloads. It gives you a better appreciation for the tradeoffs. This also requires understanding how silicon actually behaves, not idealized abstractions.
The other complementary skill is resource accounting -- knowing what everything costs in terms of bandwidth, latency, storage, compute, etc and how they interact. This allows you to look at any combination of hardware system and software workload, and quickly identify the resource bottlenecks. Again, if you do it long enough it becomes very intuitive. Old hands can accurately predict the performance characteristics of a software design on given hardware before a single line of code has been written, even if the design is novel, just through resource accounting. The application of these resources involves tradeoffs i.e. you can trade an excess of one kind of resource for another resource that is scarce with clever algorithms and architecture (e.g. classic space/time tradeoffs but more so).
Unfortunately, I don't have any reference material. I learned by doing over a very long time.