The camera shots that are shown in that paper are really restricted in what they can see. How can the model possibly estimate the position of an arm that is not visible?
Errors are also not exactly small, ~6cm average, although it's more like 1-2 or 5-12, depending on body part. I think this would very likely be noticeable in VR applications, but it is still very impressive accuracy overall.
The question also nicely highlights the disadvantages of trained algorithms: No one knows for sure, and it certainly isn't obvious by looking at the network weights...
On the other hand, when looking at the pictures, I felt like the authors: There should be enough information in there to get at least a good estimate. And it is extremely useful that one can nowadays "just" train a model to confirm such theories.