Neat!

Have you considered rendering the WebGL scene to an Offscreen Canvas in a Web Worker? You lose the ability to process PointerEvents (since Web Workers can't access the DOM), but you can serialize PointerEvent data and send it to the Web Worker via postMessage. That does introduce a bit of latency (~0.5 ms) that might cause some jank since PointerEvents are emitted ~8ms. It's something I'm playing around with.

Neat project though! Stick with it! If you're looking for some ideas/inspiration, check out FreeFlyer: https://www.youtube.com/watch?v=Cwau4GrxuUU