Have you used transients in Clojure? http://clojure.org/transients

They allow mutation as long as it's localized.