Hi jefft255,
I think you are totally right. As far as I understand, CNNs rely on the correlation of neighbouring pixels. And so directly applying them to vector graphics won't work.
However, vectorisation has massive downsides as well.
I will try to summarise it in some article. This had nerd-sniped my brain and so I need to write it down. Even if I have no solution for the problem yet.
Happy to send you a link to the draft once I have it so you can comment.