I must be missing something, this doesn't answer the question at all. Literally everything you said is true for just building the graph in JS/SVG. What does D3 give on top of that?
Consider how much JS you'd have to write to even start creating all the frameworks to build in native SVG structures... a lot.
I created html graphs with tables in the late 90s, made Flash graphs in 2000s, upgrades to numerous JS based charting libraries after this, building custom graphs the whole way.
I had looked at D3 so many times over the years when it first showed up online. But it "looked" complicated, no obvious "chart" library, etc...
Finally, when I tried D3, I was like "why the f didn't I start with this?"
Basically, D3.js strikes a good balance between ease of use and expressive power, that leans pretty far towards expressive power, while still being far, far less work than rolling your own.
Somewhere else someone described D3 as a hammer, and that's a good analogy, because if you don't start with a hammer, it's the first tool you'll make for yourself when you need to pound nails.
In another case I might not care about it, but use d3-selection, a mechanism for creating a data-driven DOM tree — but it's pretty clear that you can achieve similar mechanics with React.
Importantly, you don't have to pull the whole thing into your dependencies just because you use a bit of it, and it's light on own dependencies. So just using a small subset of functionality is "cheap."