Two questions: how much time do you spend selling/negotiating with advertisers? And you say you could earn more from a network: have you researched at all exactly how much more?
Naively, it seems that maybe advertisers would pay you more because 100% of the visitors to your site are relevant to their products (but admittedly I'm out of my realm of expertise).