Have you ever seen the Google Lightfields demo? They have a rig they concocted to essentially capture a "volume" of video to allow for the stereoscopic effect in VR AND which then cleverly presents a different combination of the footage it captured based on your precise head position, so it makes up for these distortions. I found it absolutely breathtaking... first time seeing VR for a space that actually made me feel like I was in it. This was A LONG time ago and I suspected I'd be seeing a lot more of that content, but I was... very wrong, it seems.
Your point is completely correct. Even Apple's awesome new stereoscopic 3D short film for the AVP immediately loses what it could be its total awesomeness from this basic fact. The perspective being perfectly fixed will never quite be there to fool our brains so used to dealing with these micro-movements.