Once they can put high quality positional tracking in an iPhone, the whole problem is basically solved, they can throw it in a GearVR-like chassis, add VRKit to iOS and instantly be the biggest VR platform in the world. They'll simultaneously be the biggest AR platform in the world with hand-held iPhone-based AR. They're technically already there with Pokemon Go.
One can quibble about the handful of areas where iPhone can't compete with other more specialized AR/VR hardware (head mounted AR, high performance realtime graphics, etc) but content creators are adept at overcoming limitations like that. An iPhone 8 or 9 with positional tracking will be basically Good Enough for any <1hr experience you could think up. Content creators will adapt to the device market, and the best content will run on iPhone, probably iPhone first, with lots of indie experimentation happening on PC but big brand name experiences generally starting on mobile.
I'm hard pressed to think of a seven figure audience who would need head-mounted AR or high performance realtime graphics. Airplane mechanics, gamers, etc, will stick to Windows/PlayStation, but all of the mass market stuff will be iPhone first. Even if the iPhone is mostly limited to composed prerendered light fields, I don't think that's going to hurt them. PC gamers overestimate the importance of GPU flops because current VR engines are new and unoptimized. Once we get the next generation of engines, hardware won't be nearly as big of a limitation.
As for competition, maybe Google somehow gets in their first, but I'm just not sure they can package it as slickly as Apple and get the content creators on board. We'll see, Google is trying to pick up those skills. I don't see Oculus or Valve really competing in the mass market space because allegience to gamers is in those companies DNA.
Not a big fan of Apple, but AFAIK they have a decent camera on the back of that thing capable of high-FPS video capture (google tells me on iPhone 7 it’s 240 fps for 720p video).
You fix e.g. 3 balls in your room near you, from that image it should be possible to track position by looking at the video stream. In robotics, balls like this are sometimes used for similar application: http://www.precisionballs.com/Satin_Finished_Balls.php
You can even go without special objects in the view, if the background is contrast enough, ain’t too far and ain’t too close.
Not sure about latency, though.