It's subjective, in my opinion. I might be fine with inline SVGs if they were "smaller", ie didn't have the doctype and root element and general XML verbosity on all nodes. PlantUML / Mermaid have a much more to-the-point syntax.
Also, I personally hate markdown tables since editing even a single cell means that I likely have to then resize either the entire row or the entire column, or both. So when I do need tables I use HTML tables anyway.
At any rate, a middle-ground might be to have the diagram source as references at the end of the document, and reference those from the use site, like `![Chart][#chart]` or something. So you don't have to see the diagram source inline if you don't like it, but it's still there in the same file.