1. Chrome’s buffer size seems to be very small, enough to cause problems if you’re trying to watch very high resolution footage or play multiple videos together.
2. If you serve over HTTPS, Firefox sometimes seems to warn about mixed content even if everything is properly encrypted.
3. Some popular mobile devices/browsers don’t send cookies when requesting video files, which is a serious problem if you rely on those cookies for, say, authentication of logged-in users.
4. The canPlayType API is still one of the most bizarre and awkward I have ever encountered.
5. There is no standard for how video controls appear, or indeed if they appear at all before the video is actually playing. Moreover, some browsers will completely change their presentation of the controls as part of a software update from time to time. This makes it all but impossible to make clear to the user that something even is a video that they can click to play, if the video appears as part of a larger general content page rather than the obvious main element in a dedicated player page like YouTube. It also causes problems if you have short videos and include captions/annotations in the lower part of the video, as the controls may just stay there and obscure the content, which can be very frustrating for users.
I think HTML5 video will be a great development in time, but right now, it is still quite an immature technology. You can use it fine in trivial cases, which is sufficient in practice for a lot of useful pages, but all kinds of not-that-tricky-really things are still broken. The more worrying thing is that some of these issues have been broken literally for years, and the browser makers show no great enthusiasm for fixing them, which hampers developing anything using the HTML5 technology that is more interesting than another basic video hosting site.