When did you try it? Both Matrix the protocol and implementations like Element X have improved immeasurably over the last year or so.
As a user, I just need stuff like this to be standard, and work for every participant regardless of what client they use.
In terms of non-Element clients... I can't really speak for them, but I hear really good things about Cinny for folks who want a more Discord-like experience on desktop, and we livecoded an Element Call integration for it at the Matrix Conference in October (hopefully it merged). I think FluffyChat also may support the new MatrixRTC calling too.
Still, I love Matrix and hope that these issues will be resolved in time.