Reliance on the Java ecosystem has been the biggest issue for me in trying to use Clojure. I love many things about the language, but I have zero experience with Java or the JVM. A number of my attempts at using Clojure for a project have been hindered by my lack of understanding of the underlying runtime and the tools. In a way, it almost feels like Clojure simply isn't for me.
On the other hand, I've had much more luck with ClojureScript, which I'll happily use over JS any day, though the build system still occasionally mystifies me.