The downside is that it's a difficult tool to become fluid in and so it's difficult to find experienced hackers, so scaling the team is going to be slower than with other, more popular languages.
There are alot of caveats here, and it highly depends on the teams, projects, maturity, quality of tools, and so on, but there are not alot of weird concepts in clojure that makes it that hard to understand (as a counterpoint to eg: rust borrow checking rules, or C++ templating, and so on).
The hard part with clojure is having just enough discipline to keep things in check.
On the other hand, the ROI from hiring a couple of developers with Clojure/CL experience is almost always guaranteed to be higher than hiring a bunch of "regular" programmers. Niche languages like Clojure usually attract more experienced, tired, "seen some shit" developers who have likely worked with multiple languages before. Their broad experience typically leads to higher productivity and better-quality code.