I would love to see an integration with excalidraw where you write text to seed the diagram and then do the touch ups with something like excalidraw. That would solve I think your workflow.
Please give me a combination between Excalidraw and Graphviz, with stylesheets, and layouts generated via a straightforward Python API, immediately displayable in a Jupyter notebook, with a display widget having hooks to move items around, giving auto-updating a Python property dict.