ggplot2 is much nicer than matplotlib and d3 as long as you stay on the beaten path; which is wide enough to accommodate the majority of use cases. But the second you step off that beaten path, it's hell.
It just looks like there's a fundamental tradeoff between visualization expressiveness and API complexity.
Btw, Seaborn is a framework on top of matplotlib that replicates the semantics of ggplot2 if that's what you're after.
There's ggpy (formerly ggplot), but that also doesn't have a grammar going on under the hood, and on top of that it pretty directly translates ggplot's API to python. ggplot's API might be fine for R, but it's wildly unpythonic. ggpy doesn't understand where it's coming from or where it's going to. Plus it's buggy and incomplete and seems abandoned.
The one I'm currently looking at is plotnine, which keeps the API but at least it also keeps the grammar.
I was working on my own library until I found plotnine, with IMO a better API (and a rudimentary CLI). Now I'm looking at building it on top of plotnine, and when I have a POC I plan to get in touch with the author of that and see if he's interested in adopting it. (I think this is a long shot, but worth trying.)