I am not sure why there should be an emotional factor here, as expecting everybody to conform to some pronunciation ideal they have no experience with is arrogant, to say the least.
But see, that's the point. We're not speaking German when we use a borrowed word in English. It's no longer a purely German word, despite its origins, just as "xylophone" isn't a mispronounced Greek word, nor "Handy" a misused and miscapitalized English word.
That's not hubris, it's just descriptivism.
"Wagen" on its own would probably be pronounced like way-gen, if my english intuition is not fooling me. Instead it is pronounced like wuh-gen.
I don't really care about this either way but if you are bothered by something the weird mixture is.. a bit annoying.