I think the problem is in the default behaviour of being on for all domains by default. Which even when you do your job right, by not using overlay or otherwise distracting ads, blocks you by default, without giving the user a choice to decide.
Another problem is that the ads the user deemed to be acceptable visually don't necessarily carry a correspondingly lower risk of malware.
Yet another problem is that the user likely thinks that by allowing ads for domain X they are allowing X to serve them ads. When in reality, they are allowing X to add a third-party service Y that sells access to Z, and that neither X nor Y currently have a way to guarantee that bad things won't happen from letting Z (or even who Z is, or that Z was a possible participant when X signed up for the service).
Lets compare it to cancer. Maybe the users don't want brain cancer, but they should have the option for skin cancer.
Nobody wants cancer. Users who install adblockers don't want ads.
personally i compare them with housekeeping. I don't like it, i can decide to pay someone to do it, or i've to accept them. Which could kill me by poisonous cleanser (malware).