If you are running Clojure you need to run at least clojure.core. Loading that is handled - surprise - by the JVM. As would be expected, if you run other implementations (like Dalvik), this can change dramatically (for worse, usually).
So it is correct that loading the JVM itself is not what's taking most of the time. But loading take so much time because of the JVM.
Here's a pretty good (if a bit outdated) analysis on Clojure startup time: http://clojure-goes-fast.com/blog/clojures-slow-start/
The question is, could Clojure be rewritten in a way that loads significantly faster with the same behavior on the JVM? I'm not sure.
And especially on big projects, the lack of DCE on the JVM can lead to loading a ton of useless code that significantly impacts startup time outside of the Clojure language.