Here are some links:
https://discussions.apple.com/thread/2778603?start=15&ts...
http://reviews.cnet.com/8301-13727_7-20026241-263.html
It's something to do with Apple's CDN, as I remember it.
Problem is, the CDN returns a server near the IP address of the machine doing the DNS lookup, and that's generally your DNS server, not your own box. If you use your local ISP DNS server, that's close enough you do get a nearby server. But if you use a third party DNS server, your DNS lookup might happen from a DNS server in Ashburn VA while you're actually in Chicago. The CDN would balance you to VA instead of Chicago.
That's why using a local DNS provider gives faster downloads when the download comes from a multi-POP CDN.