I do agree that there should be a fast and easy way to experience VR, uh, experiences without having to download native installers for every one. I'm not sure what way that will be. I am extremely unsold on the browser being the solution. But if it works out then that'd be pretty nice.
Maybe webVR won't be the way to get AAA titles in VR, but it will certainly lower the barrier to 1) development of VR content 2) distribution. I think webVR can do a lot for the adoption of VR without "poisoning the VR well."
Unreal Engine in asm.js looked pretty impressive to me. If we can get VR to work on the Galaxy Note, albeit with custom hardware access, then Firefox on a desktop doesn't seem unreasonable.
Also, I think you're exaggerating. I'm sure http://timeinvariant.github.io/gorescript/play/ would work fine with VR.