The "j" is always what put me off from learning Clojure. I've had too many bad experiences with JDKs and JVMs that I stay a thousand kilometers away from any thing that has "J" in it unless it's referring to JavaScript.
It's pretty easy to switch to writing JVM Clojure if you're familiar with Babashka. Most of the libraries written for Babashka are designed to work in either environment.
That said, there are reasons you may want to use the Clojure on the JVM later on. It might be interesting to read the replies to another poster with similar concerns about the JVM: https://news.ycombinator.com/item?id=40445415