So what you're saying is that it's probably better to use a browser, because your bandwidth between your PC and your TV is likely to be much better, as well as having the capability to cast not only web content but also anything on your desktop. Plus a mirror is much easier to control, and some cast capable devices may not have the memory to "download" video files.
What am I missing here? You seem to think it's a bad thing.