So what you're saying is that it's probably better to use a browser, because your bandwidth between your PC and your TV is likely to be much better, as well as having the capability to cast not only web content but also anything on your desktop. Plus a mirror is much easier to control, and some cast capable devices may not have the memory to "download" video files.

What am I missing here? You seem to think it's a bad thing.

While this is true, newer Chrome versions have a feature where if you fullscreen a video in a tab, it will (within a few seconds, if the video is in a directly supported format) attempt to send the video directly to the Chromecast instead of mirroring the tab. It doesn't always work, but it's much smoother when it does.