If ads were served by the publisher than the advertiser would always think that the publisher was cheating and claiming they showed more impressions and generated more clicks.
If ads were server by the advertiser then the publisher would always think that the advertiser was cheating and claiming that they showed fewer impressions and got fewer clicks.
Both could believe that a third party is honest. Both will feel even more assured if the site has 50 different trackers from 50 different third parties tracking the process.