I haven't experienced the buffering issues you describe, but I've definitely seen the resizing issues from leaving fullscreen (screenshot here): http://cl.ly/2y3F0B3V0X09013U0710. I've noticed this doesn't happen after full-screen, however; there's resizing issues anytime you toggle between different video sizes.
I should point out that my connection is pretty fast, around 75 Mbps.
I also checked out the caching issue you pointed out: you're dead-on. There's no caching whatsoever with HTML5. In general, the buffering behavior is kind of odd. Doesn't behave at all like the Flash player.