> Plan9 keeps being mentioned, but do regular users understand it? (Is there a "this is why we do things, and this is why that's good" written for people who haven't written their own compiler?)

Yes, you can start with the main Plan 9 paper: http://doc.cat-v.org/plan_9/4th_edition/papers/9

And follow up with The Organization of Networks in Plan 9: http://doc.cat-v.org/plan_9/4th_edition/papers/net/

And The Use of Name Spaces in Plan 9: http://doc.cat-v.org/plan_9/4th_edition/papers/names

This will give you a good overview of the basic design decisions in the system and their rationale, for further details on how Plan 9 deal with issues from toolchain design to authentication and security see the rest of the papers: http://doc.cat-v.org/plan_9/4th_edition/papers/

They are a wonderful read even if you never touch Plan 9, they are full of insights, ideas and criticisms of existing approaches, and many even include discussion on how to apply them to existing nix systems (sadly most of this has gone almost completely ignored by the nix community).