Have you seen the HoloLens by Microsoft? They've developed a custom silicon chip to off-board all the SLAM processing, and they use that for realtime localisation. Current SLAM techniques are absolutely fast enough for this, VR manufacturers just haven't gotten on board yet.