With out going into specifics there was a lot of effort at Google going into pushing 'popular' videos closer to the likely place they were going to be played. Any caching system or content delivery system, is by definition limited to the amount of edge cache it has and its ability to predict where that content will be.

One of things putting pressure on that cache is probably Google's efforts to put 'premium' content near possible subscribers. Clearly people who are paying money have a higher expection of people randomly watching the latest cat pratfall. So if you're Google and you're doing more "TV" like things, you may have reasons to reserve some of that cache space for your premium content whether or not folks have asked for it from that part of the network 'yet.'

Another interesting area is in transit networks. Generally the tier-1 networks who provide peering connections have done that on a quid pro quo basis (basically I'll peer your traffic if you'll peer mine). However as web 2.0 companies like Google or NetFlix start creating very large flows, suddenly if you are 'peering' and your peer is putting a LOT of traffic on your fiber but you have relatively little for their fiber, then maybe you start thinking you want to be compensated for that. We talked about Level 3 and Comcast going at it in: http://news.ycombinator.com/item?id=1954077

So an explanation might be that your transit network is 'throttling' its peering relationship. I could easily see someone agreeing to 'peering' terms where they take the traffic they've sent and use that as a 'credit' and provide full rate service for an equivalent amount of traffic coming from the peer, but excess traffic might get throttled rather than being 'free.'

This is a very active discussion in some circles. I would not be surprised to see it come up in any discussion of throughput to 'YouTube' vs local throughput.

That doesn't mean that they have infinite pipes to Google's edge caches, though (you're not exactly the only one pulling data through those pipes :-). Or that they're even pulling their data through the right pipes. I suggest talking to your ISP. If enough users complain to them, and the problem isn't in their network, they'll complain to Google.