For Lisp in particular, you're probably right about many of the tradeoffs. You won't benefit from a large ecosystem, you won't find as many job candidates (albeit likely a higher quality of average candidate), and it's probably not the best language for large decentralized teams that need to coordinate development (although, what language is?).
The lack of knowing you reference is exactly what I'm riffing on here. There are people who have leveraged Lisp features to huge benefit, and there's just no easy way to explain those features to people who don't use Lisp. "There's no way to say it in that language". To outsiders then, Lisp looks like a back-patting cult with no tangible benefits and yet an air of elitism. Similarly, those with "people skills" can't always explain the mental framework and skills they've built up to make themselves excellent at working with other people, but they can easily recognize those skills in another.
EDIT: Here's the article.
http://www.paulgraham.com/avg.html
The whole thing is well worth the read, but for what I'm talking about here, you could skip down to read about the Blub paradox.