Ok, first question, why aren't we all using Plan 9 instead of a linux-based OS? Maybe more importantly, what could the next upstart kernel/OS do different than Plan 9 if it wants to be more successful?
However, I would guess that had it been freely available when it was released in the early 90's then it actually would have more of a presence vis-a-vis Linux today.
1. As others pointed out, Plan 9 was not initially free. If you go back and look at the very oldest messages from the 9fans mailing list (http://9fans.net/archive/, starts in 1993) you'll see lots of messages from people just trying to figure out how to get a license. At the same time, you could get Linux for free or very cheap.
2. Plan 9 is unfamiliar for new users. Face it: most people define "intuitive", "powerful", and "useful" as "that thing I'm already used to". Programming in Plan 9 is pretty similar to programming Unix, and the shell feels much the same, but you still need to learn new concepts and people don't like that.
3. Plan 9 is not Unix. People have, from the beginning, wanted to reshape Plan 9 in the image of the Unix/Linux environments they are familiar with, even when it doesn't make sense. They want Emacs or Vim, without having even tried Acme or sam. They want bash, ssh, X11, Firefox, and GNOME. Some of this software, like ssh, X11, and even Vim, have been ported or re-implemented because they're useful for interoperability with Unix or simply because somebody wanted it bad enough to do the port (Vim).
Those aren't the only reasons, but they're pretty substantial.
As for what the next kernel/OS should do, I'd say it's not enough to just exist and be good. You have to be used widely and have enough developers become familiar with it. If, for instance, some engineer at Facebook comes up with a nifty kernel and convinces them to start using it on all their servers, that kernel will probably be successful outside because there's suddenly a massive deployment of it.
Alternately, if you can get into a market that's less entrenched than the desktop, you may have a chance. We saw this to some extent with netbooks (some of the earliest ones shipped with Linux by default), we've definitely seen it with smartphones, and I have a feeling that we may see it in tablets. The problem with the desktop frequently comes down to: "Why can't it run Firefox and open my Word documents?"
The command language is powerful enough to do what I want day-to-day. It's easy to apply an editor command across every open buffer, if you need to refactor for instance. Like emacs, you can write applications which run in Acme; we have a mail client, news reader, CD player, and IRC client (among others).
When it comes time to compile, I can execute a "mk" command from within Acme by simply typing "mk" anywhere and executing it with the middle mouse button--this isn't just for mk, you can do that with any command. It'll put the output in a new buffer, and if there are compilation errors, you just right-click on the filename:line-number (foo.c:42) and Acme instantly jumps you to that line of the file.
There are more cool things about Acme, but I can't sit here and type all day! It's basically what you get when you take the Unix concept "It's all just text" and make an editor out of it: text is content, text acts as commands, text is for searching.
As as example:
Edit X/.* / ,s,loginAdmin,loginIdiAmin,g
I edit this text (in a guide file for the current directory) to suit my current need; highlight it by dragging while the left mouse button is down; and then middle-click anywhere on the highlight to run it. After I run it, every open buffer that was changed gets a dirty-bit marker, and I can either middle-click the word Putall (usually in the top row) to save all the files' changes, or Undo (in any buffer's tag) to undo all of the changes across all buffers.
Edit is a separate executable which Acme runs. It talks behind the scenes to all the control files Acme publishes for the files it edits, just like any program you could write yourself. So the editing features are not built into the Acme executable, and are independently changeable. This outsourcing of even core (to other editors) functionality inverts the emacs approach, which brings everything into a big global elisp space.
X/.* / says to address every open buffer, since .* matches any pattern. There should not usually be a space after the asterisk, but HN's formatdoc makes the following text italic and removes the asterisk if I don't put a space there. (In Inferno Acme this particular case appears to work anyway, probably because of the space after the filename in buffer tags.)
,-before-s says to edit the entire content of the buffer, not just a highlighted section.
s,loginAdmin,loginIdiAmin, is a tasteless replacement command in Edit's dialect of sam (and in ed, sed, vi, and vim). These commas could be replaced by any character, as long as that character is not in the set of characters to be replaced, or in the replacement set.
g-after-, says to replace every instance of loginAdmin, not just the first.
* Do not be hostile to your potential users. (The community that formed around Plan 9 as it went open source were quite hostile, there's no need for public beatings just because someone suggests to port X11 to Plan 9 even if it's a bad idea).
I remember at FOSDEM a few years back, there was this Spanish dude and some of his friends, who were frothing-at-the-mouth fans of Plan 9, and, as they became progressively more inebriated in the place where a bunch of us were staying, he kept ranting louder and louder (and more by himself) about the perceived injustices and deficiencies of a world where Linux was more popular than something so beautiful, so elegant, as Plan 9.
It was ... a bit disturbing to watch. That kind of passion should be reserved for relationships with other people.
I think part of the problem with its diffusion was that it only became free software a decade after Linux, with no clear niche to attack (see: Crossing the Chasm) and make its own.
For what it's worth, I use wmii on Linux, which uses the Plan9 model for configuration. It makes it super-scriptable and really easy to use.
Plan 9 wasn't always free.
Also, its GUI sucks. That's sort of important for end users.