I worked for an ISP (in Russia), and in the early 2010s, we had quite a lot of complaints about YT videos buffering. We've checked our network very carefully and found no signs of any trouble. And we've confirmed issues and were able to reproduce them even when the network was least busy. We've tried to re-route traffic via different uplinks, even messed with the DNS records a little bit (I admit this grave sin) - sometimes with some temporary improvement effect, but mostly to no avail. And of course, we've talked a lot to our peers. This was quite a major headache, and we were out of ideas what else we could do (we weren't anywhere large enough to get a Google Global Cache server). Then Google did something (IIRC, mid-2012), and we've stopped getting the complaints.
With VPN you're probably hitting different YouTube servers. I guess we should've tried to route via some VPNs too, not just physical peerings we had. Never occurred to us at that time.
I left that company in 2015. Don't know how things are now.
My point is, even without a malicious intent sometimes things look the way you describe.