The <video> element is very capable, but the amount of combinations of possible video files has greatly increased. When it was developed, the biggest complexities were codec compatibility, and fallbacks.
While it would be possible to have the ability to select different video sizes/quality rates based on HTML elements and attributes, it is likely impractical to handle, say, adaptive video streaming (where the quality of the video is based on the connection), purely in HTML.
I'm no fan of Javascript for everything, but handling video is a scenario where there's generally a need, due to the complexities of video streaming.