What I've needed from D3 is the math-y bits: calculating layouts, paths, etc. What I don't need (or want) is for it to ever touch the DOM.
But that's what it wants to do most. D3 is sort of the best of the jQuery-inspired "select and mutate" generation - it does data binding in a way that really makes sense as long as you let it handle DOM operations.
But following React, there's been a pretty big movement away from procedural DOM manipulation and toward the declarative re-rendery "DOM as a pure function of state" approach. In that case, letting D3 mess with the DOM (especially in the way that it does, where complex links between DOM and data are maintained in the background) is bad news.
If you already have a way of getting elements in and out of the DOM and keeping their state aligned with the data that you like better than the select-and-mutate model, a lot of what D3 does seems unnecessary or introduces problematic interference. React and D3, for instance, tend to not play very nice without a lot of effort.
I've had projects where programmers lean on D3 because that's the way they've learned to get SVG elements into the DOM - they'll take an example they've found somewhere that mostly only uses simple shapes, modify it a little bit to do what they want and ask me for help adding that visualization into another project. Sometimes in these cases, D3 isn't getting used for much except for "insert an svg element with these attributes for every node in this data structure", in which case it's total overkill, and I end up stripping it out entirely. I sometimes end up keeping it around for things (I'm probably not going to write my own Reingold-Tilford tree layout algorithm), but often in these cases I have to do extra work to keep it from conflicting with other things.