It's nice to have the diagram source in the same file that uses it, instead of a separate file that needs to be kept in sync.
So, the question is would you rather have SVGs inline?
Personally I think the reference to another file is a perfectly fine compromise, as the content is sufficiently different to no longer be markdown.
Also, I personally hate markdown tables since editing even a single cell means that I likely have to then resize either the entire row or the entire column, or both. So when I do need tables I use HTML tables anyway.
At any rate, a middle-ground might be to have the diagram source as references at the end of the document, and reference those from the use site, like `![Chart][#chart]` or something. So you don't have to see the diagram source inline if you don't like it, but it's still there in the same file.