You're both right! I'd point out four big categories of difficulty:
1) Physics. You need a swept collision test capable of handling very high linear and angular velocity of very thin shapes (a sword is not thick, and the tip can move pretty dang fast), and collision response capable of dealing with the results. This is doable- conservative advancement operating on distance functions will work- but working out all the details in a way that makes the final result fast is not easy. A speedy implementation alongside a speculative contact solver can work, though: https://www.youtube.com/watch?v=sfgC_eNx9M8#t=1m20s
2) User interface. A mouse and keyboard struggle to express all the degrees of freedom available to a physical handheld weapon. Swinging using the mouse works okay, but you need to do so in a way that doesn't interfere with camera control, and you need to be able to interrupt and redirect movements at any point, then add jabs, different guards, ... Expressing all of that in a single intuitive interface that doesn't require awkward mode switching is not easy. My prototype had an okay-ish approach, and I think it could still be improved further, but there's no way a mouse-keyboard user would be able to compete with someone of equal skill using VR hand controllers. There's a good reason why melee combat games are so much more common in VR.
3) Graphics. Single planar projections simply aren't enough for first person (especially outside of VR)- it feels like wearing blinders. Even third person doesn't help enough, and seeing your character gets in the way of seeing your weapon and opponent sometimes. Using a nonplanar projection like stereographic fisheye, it starts feeling okay around 160 degrees, though 200+ is nice once you adapt to the mild distortion it introduces. This requires a pretty different approach to rendering (like multiple subviews, direct projection raytracing, vertex warped rasterization, etc.). A couple of examples: https://www.youtube.com/watch?v=jQOJ3yCK8pI https://www.youtube.com/watch?v=mIax_ProQ8c#t=1m4s
4) Animation. While achieving a level of animation quality common in early 2000's titles is pretty easy- just have a set of poses for various weapon states to interpolate between, blend in some leg motion, shrug, done- achieving a very high animation quality bar is extremely difficult. Most modern AAA games use massive libraries of mocap data, and in some cases the animations directly drive motion to make everything grounded and solid. For Honor is a good example: https://www.youtube.com/watch?v=4pdcA3mhe0E. But in a game allowing total physical control with expectation of proper physical response to every interaction, the mocap coverage requirements explode. You can still do pretty well if you have enough data and put in a huge amount of effort, but at a certain point you start wishing the magical machine learning genie would swoop in and save you.
I think all of these things are solvable enough to permit the existence of a game, but they're hard enough that it's not surprising that they are still so rare.