matplotlib is an example of unnecessarily complex and confusing "organic" API. That's why there is so much resentment to use it; trivial things need non-trivial internal understanding and confusing boilerplates.
It is designed to be familiar to people who already know MATLAB, and it does that quite well. So it is not "unnecessary", it's like that for a reason. I agree tho' that someone who has never touched MATLAB might want to plot directly from Pandas, or maybe use Seaborn.
I think "organic" part of the API is very well integrated and completely optional in most places. For instance, you can use the matlabish shortcut "subplot(111)" or you can spell out the parameters in a pythonic way as "subplot(nrows=1, ncols=1, plot_number=1)".
Yes, and this was a mistake. Other than logical-indexing, there's one is better off forgetting _everything_ about MATLAB.
For specific examples, I've looked idly at doing chord diagrams and trees in ggplot. Chord diagrams don't seem to exist, you can do them with another R package but I don't think it interacts with ggplot. There's a ggtree library that does interact with ggplot, but in kind of a weird way that I wouldn't describe as "you can now draw trees with ggplot".
(I haven't explored too closely. This was in the context of "I want to write a ggplot for python, and I'm curious whether supporting chord diagrams and trees at some point in the future is plausible".)
ggplot2 is much nicer than matplotlib and d3 as long as you stay on the beaten path; which is wide enough to accommodate the majority of use cases. But the second you step off that beaten path, it's hell.
It just looks like there's a fundamental tradeoff between visualization expressiveness and API complexity.
Btw, Seaborn is a framework on top of matplotlib that replicates the semantics of ggplot2 if that's what you're after.
There's ggpy (formerly ggplot), but that also doesn't have a grammar going on under the hood, and on top of that it pretty directly translates ggplot's API to python. ggplot's API might be fine for R, but it's wildly unpythonic. ggpy doesn't understand where it's coming from or where it's going to. Plus it's buggy and incomplete and seems abandoned.
The one I'm currently looking at is plotnine, which keeps the API but at least it also keeps the grammar.
I was working on my own library until I found plotnine, with IMO a better API (and a rudimentary CLI). Now I'm looking at building it on top of plotnine, and when I have a POC I plan to get in touch with the author of that and see if he's interested in adopting it. (I think this is a long shot, but worth trying.)
Building abstractions is just a matter of writing very simple imperative functions. Modifying existing abstractions is a matter of copying and pasting out of the source code for existing ones.
It can involve lots of tedious trial-and-error, and the documentation is a little terse, but it's about as close to drawing by hand as you can get.
Ggplot2 is great for prototyping, but when you want to really own your graphics, go for Base.
There's also a middle path, called Lattice. It's built on the same library as Ggplot2 (called Grid), but lets you dig down into the guts a little more easily, at the cost of your graphs looking "older", since it's based on Trellis graphs from SAS.
I will agree on reproducibility and portability across graphics devices (not to mention actoss installations). Grid takes care of so much annoyance in that regard.
https://cran.r-project.org/web/packages/gridGraphics/
I've said it before, but the attempts to replicate ggplot2 usually fail to implement the full stack. As a result we have somewhat incomplete implementations of surface features of ggplot2, without the depth that is afforded by the grid / ggplot2 combo.
On the other hand, matplotlib is ridiculously powerful as an embedded plotting library; the limiting factor there is definitely not flexibility, but rather performance, which limits applicability to larger data sets -- you can work around that in many ways, or use something different. (I wrote more than one application-specific OpenGL plot renderer). Performance also limits interactive use, especially on non-desktop devices.
[1] Disclaimer: Very few technical docs I've seen for "modern" software are of convincing quality. Most are poor or don't exist in the first place. Further Disclaimer: I don't know how to do better and write docs I'd consider at best "meh-ok". People able to write technical texts well seem to be incredibly rare (likely an educational gap; besides 08/15 standard English courses I've never seen lectures or courses on technical writing at an university).
There's a reason Technical Writer[0] is a profession in of itself, this stuff is much harder than people give it credit for.