It's not so much that they couldn't sell it, but adjusted for inflation to today's money, a well equipped Xerox Star workstation with reasonable amount of RAM, HDD ($20000+ in 1981 money), would be the equivalent of $70,000 in today's money. It was very much a high end system affordable only by a few niche users.

It's not clear to me how much it cost them to build the Star, but it might have been pretty close to that 20000 figure if they included all the R&D costs that went into it, and the fairly specialized high end hardware at the time like 20MB HDD. And it was being done in fairly low quantities so not much economy of scale.