I lived in San Francisco around 1992 and got to know Ted a bit, along with a few others in the Xanadu team. At Ted's houseboat in Sausalito, I witnessed a few demos of the ever-in-progress Xanadu software (and related projects), and saw some of the code.
Ted has always had very strong ideas about treating copyright as technology (see his 'transclusion,' etc.). That's why he's always been bitter about the web winning the hypertext game, without the necessary (in Ted's design) two-way linking with automatic credit (& royalties!). I can imagine Ted being vociferous (in his unique way) in opposition to a blanket release of source code via the distribution technology of the time -- largely Usenet News, FTP sites, and BBS systems. The contemporary landscape of open-source licensing had barely evolved by the early 90s; if you released something on the net, you were pretty much putting it in the public domain. That was antithetical to Ted's goals of creators owning their content.
Even if he had been convinced to release the code, it was in Smalltalk. (It had started out in C, but had been rewritten by the time I saw it.) The culture of Smalltalk never seemed to be very interested in methods of distributing source code. (Or even binary code: ST apps included their entire runtime environment, making them really heavy.) Even if the code was released, the number of people running Smalltalk systems was miniscule.
Finally, Xanadu, for most of its active life in the early 90s, was very much in experimental development. Although Ted had been thinking & designing & writing about Xanadu for decades, I believe this period was the first time he was actually able to commission running software that implemented hypertext. (Ted's not a programmer, nor an engineer.)
As for ‘publishing specs’ — most of the details of Xanadu are described in Nelson’s book _Computer Lib / Dream Machines._ Ted’s written extensively about his hypertext ideas (as this very post points out). I don’t know if Xanadu itself was ever documented, but you could read his book and probably implement most of it — if you dared!
Although Ted has always stood up for the 'regular person' computer user (that's partly what _Computer Lib_ is about), in actuality both the UI of the software, and the inherent concepts, were very weird and outside of what most people would understand. I'm not sure Xanadu would have ended up being any more influential than it already was — Berners-Lee certainly knew of Nelson's work as he was created the WWW.
(@terminalcommand: You seem to appreciate Nelson's work more than most, so please don't take the above as any sort of argument. But because you touched on the open-sourcing issue, I replied here.)