I used Docbook SGML (and later XML) for my book about Webmin. It wasn't worth the pain.
I've worked on another large Open Source project that used Word (yes, really!) and OpenOffice with custom templates and conversion tools (they would, for the most part, save as HTML, and then convert from that to PyDoc, .chm, and several other formats).
I later converted the Webmin book into wiki format (TWiki, specifically), and now we're in the midst of moving it to Markdown (with extensions). I like Markdown best of all, and is now what we're using for everything (forums, tickets in the issue tracker, and all documentation).
Don't underestimate the cost of hard to deal with formats. Docbook produced beautiful output, and in many formats, and was useful for pulling data out automatically. But, the effort involved in writing the docs was atrocious. Even with good tools (most competent XML-capable editors can work with Docbook pretty well...I used jEdit in the end), you'll produce half the docs with twice the frustration. With Markdown, when you need more control you can drop to straight HTML.
Markdown works great in revision control systems, and is easy to generate programmatically (we've started exporting our API doc PODs to Markdown, for example). Nearly everything support it, as well. Our new CMS is Drupal, and it has great Markdown support, and there are converters from various wikis (we have more than a thousand printed pages worth of docs in two different wikis to be converted).
I can wholeheartedly recommend it. Textile is another popular minimal markup that I imagine has pretty much all the same advantages as Markdown.
Pandoc might be worth a look, too. I haven't had need for it, yet, but it's got an awesome feature set, and can do PDF from Markdown (so I probably will use it soon, as we get pretty frequent PDF requests).