You've just pretty much described the motivation for core.matrix: it's an API that wraps various other back end vector/matrix libraries (including JBlas etc.) with a nice, standard Clojure API.
BTW, as a total aside, I'm a big fan of Clisk. I think you nailed the API for functional image manipulation. I'm interested in seeing an OpenCL backend for it. If I get anything working well, I'll be sure to contact you.