It would be really cool to have an invoice format that contains payment and tax information in a machine readable way and a way to send that information around with a verifiable channel.
pdf.js lacks capabilities to extract the XML or verify signatures, so the usual way will be to use Acrobat Reader or the usual bunch of "industry-standard" invoice-processing crap that now suddenly has to deal with malicious input.
The idea to do it differently might be nice in theory, but is lacking a smooth way to change over from the old paper-invoice ways. PDF will be the thing for some decades and we will have to deal with it.
This means that while you might be able to use something other than PDF for the human-readable part, I don't think anything other than PDF will be used. All the other stuff (XML with embedded SVG or PNG, Word, plaintext) will have acceptance problems in one form or the other.
EDI is big business to big business, as evidenced by you mentioning SAP. There, you may be completely right, I don't know.