A typical CLI might make a terrible piece of code to integrate with. I wonder if you can design CLIs that are better to integrate with, though. To some extent, this is what Git's 'plumbing' layer tries to be, although i don't know how successful it is.
One advantage of building on top of the CLI rather than via a library is that it enforces the rule that any functionality available in the GUI is also available in the CLI. It also means that any functionality added to the CLI is also immediately available to the GUI. With the library approach, it's easier for one to get ahead of the other.
Mercurial has an interesting take on this - the command server:
http://mercurial.selenic.com/wiki/CommandServer
This lets a process spawn a sort of daemon version of Mercurial, with which it can communicate via its standard input and output. Communication is a machine-friendly framing protocol which allows the parent to execute Mercurial commands. These are the same commands as you would use on the command line (interpreted by the same dispatcher, i believe, so really exactly the same). So, it's good code to integrate with, but exactly as powerful as the CLI.
The one real let-down is that the output is, AFAIK, the same as Mercurial's interactive output, which means the parent has to parse it with various command-specific regexps. If you were following this approach from the beginning, you would probably structure your CLI so that commands emit output in some structured form (eg a tree of named, typed values, or a table of named, typed columns), which is then rendered to the console somehow. You could then return a machine-friendly serialisation of that in command server mode.