It's unfortunate that there's the collateral damage that affects non-obnoxious sites; and, while I whitelist the sites I frequent, and many here may as well, I'm sure we're the minority.
That being said, here's about as close to a real-world analogue for my thoughts as I can get: in NYC, I'm sure there are people in Times Square trying to gain signups for Greenpeace and other "ethical" purposes mixed in with the other people hustling CDs, comedy club tickets, and the like, but I just ignore everyone as I walk by, wholesale, because stopping and seeing what everyone's about will turn a brief walk into a pain in the ass.
I'm not sure what the answer is; perhaps the various adblocking technologies could make whitelisting a site even more obvious? With AdBlock Plus, you hit the drop-down and choose Disable on foobar.com, but it requires an active interaction to seek that out. GlimmerBlocker for the Mac is even more buried (as it's a proxy, it's in System Preferences.) Maybe switching to putting placeholders where the ads would be with a short message and a single click to whitelist would make it a little easier for users to both see how many ads a site's pushing as well as whitelist the site with the minimum amount of effort? I donno.