If the browser is waiting for it then the server would be waiting for it too. The endpoints for adexchanges are the same and wouldn't be any faster just because it's coming from a server instead.
I suppose ads could be cached server-side, but it would probably be more efficient to cache an ad client-side. If ads are personalized few users of a website may see the same ad, but an individual may see the same ad on different websites.
If a smaller site uses and ad network on the server side then ads cannot be cached on the server, because the ad network always serves the highest paying ad at the moment which constantly changes according to the ongoing ad auction and inventory.