I don't think it would be too bad. Cameras are tiny. The processing would still happen on the PC, and you could delete the lighthouse tracking sensors. I guess the hardest part would be sending that much camera data back to the PC over the cable.
And once you have the pipeline and computation power to enable inside out tracking all on device, adding an OS is essentially free.
As for sending data over a cable, there's nothing inherently laggy about it. After all, the display signal already travels over the cable, and the cable transfer is by far not the limiting factor in latency. The camera data is lower bandwidth than the display signal, too.