Crawlers naturally want to be stacks with pipelines and expressing them as tail recursions over URLs curried into transducers, etc was easy enough conceptually, by difficult in reality because you want crawlers to be stateful. I think in clojure you end up hacking state by just attaching stuff to the outputs of your functions and building of more complicated objects down the line.
I discovered that stuff which is cake in SQL or a language with data frame support is often typically hard in Clojure. Eg eg joining data , aggregation, etc. The few clojure libraries for this sort of thing are complicated to use because they have a mechanical sympathy with how clojure executes. Eg https://github.com/nathanmarz/specter/
So I wrote some of my own libraries to get over this hurdle. I then realised that there isn’t a single decent statistics library that still maintained, and that the java bindings were far from simple to us.
At that point I gave up on clojure for my use case. That’s not to say that there isn’t something for which it’s awesone. But in my opinion it isn’t data processing.