I've chosen to specialize in Clojure and it hasn't been difficult at all, in fact in a lot of ways it has been easier. I was already a senior dev when I started with Clojure, but having Clojure experience on top of that has made me a very valuable commodity on the (admittedly smallish) Clojure job market.

It's a trade-off of course, there are fewer job openings, but Clojure isn't that niche, and everyone is aware that it is more difficult to recruit from the small pool of Clojure developers or those willing to learn. So when they find someone with seniority and experience putting Clojure into production they will court you pretty aggressively.