On the software side of things, the good news is that the combination of a decent conservative advancement detection scheme and speculative contacts can get you down to microsecond tier fidelity pretty cheaply. (Incidentally, this specific problem is why I built that particular form of continuous collision detection :P) You're definitely right though that it's not going to magically solve away hardware level concerns.

On that front, one of the things I had to end up doing to ensure physical consistency was bite the bullet on a level of indirection. This can be flat out necessary on a gameplay level- allowing full 1:1 control often leads to pretty nasty pathologies and difficulty with correct physical responses. A truly physical character's motion can never truly match the input motion, since your controller is dramatically more nimble than even a light 2.5 pound longsword and the physical sword may be interfered with. This introduces some challenges on the side of user responsiveness and intuition, to be sure, but the physical motion also smooths out input and seems to make small scale controller imprecision a distant concern to the player. If the player can adapt to the physical indirection, they tend to allow a lot more leeway. This becomes even more valuable when trying to hide latency. If you have physically reasonable accelerations, networking overhead becomes a whole lot less objectionable.

There are definitely tradeoffs- the feeling of being a puppeteer takes some time to get used to.