I am largely ad blind. I only notice them when they are annoying, and when I do notice annoying ads I generally leave the site.

It is hard to comment on your situation without a link to evaluate. I would suggest that if you are thinking this way, you probably already know what the answer is.

Perhaps some A/B testing is in order to tailor your site to your audience.