Weird that you don't have that resizing/offset issue when exiting fullscreen, but rather when toggling between video sizes. Everybody seems to be having different experiences.
I was using Chrome 16 to test the cache. For the HD stuff, I must say: I can't find many HD videos to try it on. HD videos are more likely to be supported by big corporate accounts, and they'll opt for ads in their videos, too, which means that the video won't be HTML5-compatible (that's the real distinction). If the video doesn't have Google Ads, then it can be played in HTML5. Otherwise, it has to be Flash.
Yeah, it took me a while to find some HD videos. I noticed it when stumbling across an advert for Breaking Bad which was uploaded by a user, if I remember correctly.
I haven't tested on all of my computers yet, but I've updated to Chrome 17 on my Mac and the caching issue seems to have gone away. I believe I still have Chrome 16 on my Windows machine, so I'll hopefully be able to test them side-by-side.