d3 and direct manipulation of SVGs are both so low level that make a decent looking chart requires a bit of boilerplate. That boilerplate should get wrapped inside of a directive and once you've done that, the specifics of how the directive is implemented don't matter too much.
It would be nice to use one language for data binding; I think angular would end up being a lot more verbose and prevent some of the nice things you can do with .call() and .transition() though. Additionally, I don't think 'creating a hole in the DOM' is necessary a bad thing. Sending graphs only the information that they need separates concerns nicely, resulting in more modular and maintainable code.
If you haven't read it, http://bleedingedgepress.com/our-books/developing-a-d3-js-ed... is pretty good.