We must not also forget that web fonts have replaced text graphics. They can be small, but when used on every header or with multiple states in interfaces, they added up quickly. Not only are web fonts lighter, they are real text. We don't want to revert to text graphics.
What we have is the designer on one end, and the user on the other. The designer wants to decide how the page is served. The user wants to choose how they consume the page. Some users may add salt and pepper, or remove all the tomatoes. Giving users power cannot be a bad thing. But do we want features that automatically change our food no matter where we go? And there may be some allergic to everything. Maybe they'll turn off JS and not load any images, and be okay with half the sites being broken. But they will always be a minority that other's shouldn't be too concerned about. And they will always do whatever they want.
To know if a web font is ugly requires tasting it first, so at that point, you've already paid for what you may have tried to save. But considering how little weight web fonts are and how important they can be as part of what is being expressed by the composer of a page, ignoring it would be like randomly removing an ingredient deemed important by the chef.
Why risk those great sites for the one's that suck? Why would we allow the worst food to dictate all of our meals? Shouldn't we just call out bad web sites for what they are, instead of crippling our hard earned technology?
Imagine iOS 9 with Arial. Fonts are important.