All you need is a niche market and your first advertiser. Then start contacting the publishers in that space. You'll need somebody to help the publishers embed the ad codes. A decent developer could track everything or you could just use Google Analytics or some other commercial tool. Most publishers are willing to try something new, but consider the ad inventory is worth much more to Google (Adsense) because they have a ton more advertisers bidding on the inventory and they're tracking people's interests and so forth.
On the one hand, Google completely ignores 99% of its publishers. On the other hand, the bigger publishers receive advertising proposals all day long every day. So yes the advertiser can contact the publisher directly, they don't really need you, which (last I checked) is against the Adsense/Adwords TOS.