In my opinion, this turns out to be a pretty major win. Code is still data, it's just that this data is a mix of lists, vectors, and maps, and now you can type associative data structures into your code as easily as you can lists. Hash tables in vanilla Common Lisp are a nightmare (there is a nice library that at least gives a Clojure-y initialization option via Reader macro), and a glance at the Racket documentation suggests the story there is probably not much better.
Clojure code in the wild uses the destructuring on maps a lot like, e.g. Haskell code uses pattern matching, and Spec is an interesting core library aimed at allowing libraries to insist that keys be present in maps and so on.
Anyway, I had occasion to interact with some Common Lisp code I'd written several years ago, and I was struck by how annoyed I was at it for not being Clojure. Part of that is the Lisp-2 issue, but I also really missed how lightweight and easy Clojure maps are, and how easy they are to use for control flow. (I will say that Clojure's clunky recursion annoys me, but much less than trying to deal with Hash Tables in CL)
Also, and I know you say you don't care about Java, but ... sometimes you have to interact with some annoying outside system (like, a database or a web browser), and there is a high-quality Java(Script) library that makes this task trivial and lets you work on what you want!