The only downsides I've experienced are:
1. Images and accompanying text get separated, which turn-based bots get confused by. I seem to recall there's a standard for this but it's not implemented.
2. XMPP support isn't common in claw-type agents so I've had to patch it in. Not a terrible experience overall.
I had a look at Matrix but bounced off Element, hard.