If people don't view the ads, the site shuts down. Maybe you don't like that site all that much, so it's fine. However, more and more sites would shut down due to lack of funding. Or, perhaps, they find different manners of obtaining money. Maybe they enforce pay walls, maybe they have product placement in their articles.

The fact is, most advertising is not obtrusive. Despite the problems that Google Ads has, it is not visually obtrusive. Hell, half of the time I don't even notice them. Does it hurt me? No. Does it help the site sustain itself? Yes. Would I notice a pay wall or product placement? Yes. Would it obstruct my enjoyment of the website? Yes.

To directly answer your question: you should care if you hurt society. If people stop being altruistic altogether, politeness and civility go out the window. An Ayn Rand style system pops up, with people being concerned how to give themselves the most power and goods while screwing others over the most. Maybe you're fine with that; I'm not.