Interesting. I thought about writing a Matplotlib version of my Veusz GUI plotting package [1], but at the time Matplotlib was far too slow.
Veusz is quite nice in that the Python-based widget-based plotting library is the backend to the GUI. The saved documents are essentially the Python API calls used to recreate the plot. The plotting library holds the document as a tree of plotting widgets (e.g. axis, graph, function plot...) with a set of properties.