Letters and punctuation are not icons at all. They are symbols. Icons are images, i.e. likenesses of concrete or abstract objects which they refer to. Letters have no meaning on their own, they represent sounds, which differ based on language (e.g. c sound like see, chee, key, gee, that &c in different languages) or its position in a word (caps, ace, click).

And if its not an abuse of fonts to use them as icon packs, its an abuse of the users who wants to use their bandwidth for better things (e.g. cat gifs). I have them disabled and my day is full of fun: What this "f" means? What happens when I click "ff" or "p" or what not? And some are kind enough to provide alt texts, which is soooo helpful: https://imgur.com/cHdSxcO

Letters are not icons.

"Letters are things, not pictures of things." -- Eric Gill

Github did a great writeup on the perils of using fonts for icons. One reason was the one mentioned here, users overriding them with their own preferred fonts.

https://github.com/blog/2112-delivering-octicons-with-svg

I think the issue is less about philosophy and more about technology.

SVG icons are better than icon fonts in pretty much every way. The only example I can think of where an SVG could be considered worse is if the icon is particularly complex. Even then, the only thing that is worse is file size. At that point, a PNG is probably better, anyway.