The thing is, executing javascript is not that much different from rendering all those things, and anytime you allow your computer to do some computations using any program on data from untrusted source, it's a risk. There might be even bugs in decompression library, so let's not download anything.
There were plenty exploits that used html/css parser bugs with no JS at all. I think I even heard about case where guys hacked into car using radio and a bug in mp3 decoder.
And it won't get any better unless we maybe start writing on something more safe, like Rust instead of c/cpp.
The way JS engines are implemented and used makes it an easier and more obvious target, but not really enough to just turn it off by default. Doing so is just a superstition, we could make just a little step further and rip off all our computers IO to be safe.
Running browser in VM might be a better idea, but again, in theory one could escape VM.
Sadly, it feels like we're at least a couple decades away from owning secure systems, if that's even possible. Ha, I just thought, that we might get strong AI before that :)