If you’re talking about original man pages from volume 1 of the Unix manual (as found on e.g. http://man.cat-v.org/unix_v7), then yes, they indeed make superb reference documentation. However, even their authors recognized that reference documentation alone is not enough, and thus included a collection of (much less streamlined) papers on various parts of the system in volume 2 as a sort of extended introduction and rationale. Reading man pages, even ones as good as the v7 ones, when you want to achieve a specific goal but are not familiar with the system in general, is an absolutely miserable experience. (I imagine it was worse at the time, when OSes were much more diverse.) Man pages are not the be-all and end-all of documentation for the same reason that even the best-written JavaDoc/Doxygen/... reference without standalone prose isn’t.
So, my first caveat is that reference ≠ (comprehensive) introduction, you need both. My second caveat is reference ≠ specification ≠ introduction, thus you actually need all three. When people started to create reimplementations of Unix utilities, it turned out that they were massively underspecified by the man pages. The precise specification is more or less what POSIX is, and it’s significantly less readable than the original man pages even if you also format it as man pages and take out all the extensions. It’s not that original Unix userland lacked specification documents, it’s that their role was served by the source code of the libraries and utilities themselves (for all the copyright problems that it eventually caused). The kernel too, presumably, but people famously thought that wasn’t enough, and thus Lions’s commentary was born.
I mentioned above that the papers from volume 2 also served as a rationale and an annotated bibliography, which is also a good document to have, but probably not as essential as the previous three. I’m also not saying that they all be separate documents: Unix merged the introduction and rationale but separated both from the reference; except for the tutorial, Python manages to merge the introduction and reference for its standard library surprisingly well, but lacks a good specification; the TeXbook tries to be everything at once but is in my opinion not that good at any particular task. What I am saying is that they are all different facets that have to be present in the writing.
My third and final caveat is... Not really a caveat for your statement, but still a problem if you’re a documentation writer: The concise style and general structure of the Unix man pages only works well as long as the individual pieces are simple enough and there is not a lot of them. You can’t write man pages for complex programs; or rather you can, but the format starts to show its limits, as aptly demonstrated by bash(1). It’s probably not a coincidence that Stallman wanted to replace the Spartan man page by the hyperlinked info node. You can say that one oughtn’t make things that are that complex, and I agree, but that’s not the point of view of the documentation writer.