Unix: small and hacked together, success.
Plan 9: carefully engineered, fail.
I'm not sure what the lesson is here... there's that "worse is better" essay... maybe it's that when great engineers let their hair down, they do much better?
ADDED: And Christensen in "Innovator's Dilemma" http://www.amazon.com/Innovators-Dilemma-Revolutionary-Busin... shows how slight improvements can gradually replace something that was originally better.
"A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system."
With respect to Plan 9, it had to compete with unix which was already established.
Note that during this time, another OS also became popular: DOS. Like unix, it ran on smaller hardware (actually, even smaller than unix could run on... giving DOS a foothold advantage denied unix). And clearly, DOS had some luck with respect to all the other small OS available at that time, being put on the IBM PC. There were two other OSes available for the PC (CP/M and UCSD p-System). Why did DOS win there? I don't know, but it was much cheaper, and possibly, MS BASIC compatibility had a role. Before the IBM PC, personal computer manufacturers would tout "MS BASIC" compatibility as a benefit because it gave users access to existing software.
Once you establish a platform with applications, it's hard to beat you. People don't want the platform - they want the applications. It's a network effect.
By targeting low-powered hardware, you can get established ahead of others, so that by the time the hardware has grown powerful enough to support a "real OS", you have already won. It's a disruptive innovation.
Plan 9 lacks adoption for an entirely different reason. It is very portable, and in the earlier days (~1995) ran on Suns, SGIs, Alphas, some 68k machines, and the PC. As non-PC hardware has become less widely-used, the other ports were not kept updated. However, compilers exist for the MIPS, 68k, ARM, AMD64, Alpha, x86, SPARC, and PowerPC architectures, and it is not particularly difficult to port to a new machine. Unfortunately, Plan 9 was only available via closed license until 2000. Had it been given freely from the earliest days, it very likely could have taken what is now Linux's place as the primary free OS.
The reason UNIX is so widespread is because it was the right solution at the right time. It was freely available to universities and, like Plan 9, pretty easy to port. The interface was simple yet powerful, and the introduction of pipes meant you could write stupid programs, then chain them together for powerful functionality. It really served to end the days of "One OS per Computer" and the kind of monolithic programs one found on such systems as VMS, where every program was an island.
The original Unix was put together in a thoughtful fashion with an eye to both hardware efficiency and powerful features. However, it was unfortunately written before networks and graphical interfaces became widespread--those portions were "hacked on" later, giving us network sockets and X, two systems often considered the worst parts of Unix.
My favorite part was:
"He allocated one week each to the four core components of operating system, shell, editor and assembler."
Now that is a productive four weeks!
Maybe the lesson is, if you don't have a working prototype after four weeks, you're doing it wrong. At least when it comes to software. I mean, if that's what it takes for one guy to bang out the first draft of Unix...