The accuracy of ARKit-style tracking is low. It’s mostly good enough for the use case where you hold a 5” window in your hand to peek into the AR world... But if you strapped it to your face, the latency and positional fluctuation would give you motion sickness very soon.

Yes, take a look at the Oculus Santa Cruz. It's still in development, but it has I think 4 very wide angle global shutter cameras on it. It is using a similar kind of Visual Inertial Odometry to things like ARKit and ARCore. Like the other posters have said, there is no way to get the kind of precision required for immersive HMD with a single mobile quality rolling shutter camera/lens setup and IMU.

Pavlov is correct, the tracking in ARKit and ARCore are just good enough for small mobile screens, and not good enough for anything immersive.

Windows Mixed Reality headsets use AR tech for positional tracking, but the MR headset tracking is made for a maximum of room-scale experiences (as opposed to hololens which can go multi-room).

>with only a camera on the device

Sounds like it's going to be slow compared to other methods.

A key to a good VR immersion experience is very low latency and high framerate.