Normally you don’t notice, but trying to keep things at 60fps it shows up rather quickly.
N=1
There's also a very recently landed change that might help, if the problem happens to be excessively long minor GCs (which is what I've seen with similar sites): https://bugzilla.mozilla.org/show_bug.cgi?id=1751162
Shuffling data to the GPU could be an interesting case too, but that's not handled by the JS engine itself so I'm not as familiar with how that works. It sounds like it might be worth a bug if there isn't one already (and feel free to file; we'll triage and determine if it's a duplicate.)
Last time I checked (which was admittedly 2-3 years ago), Chrome had considerable lag with VR whereas Firefox managed much better. Might depend on the GPU, of course.