We seem to have this same dead-end conversation every few months, so indulge me as I try to break the impasse with a story instead.
When I was about 13, I worked on a really fun chat system at school called (wait for it) iNFERNO][ (it was the second attempt at a chat system called iNFERNO). It was pretty awesome, and worked like this: the school ran a large network of 68k-based Macs (LCs, Centrises, even a Quadra) networked to a single Apple Workgroup Server, which ran a Filemaker Pro db instance that the school used for its HR and record keeping. As it happened, the DB server was set up such that anyone could create DBs on it, so the school geek community spun up a few to play with.
If you've ever used Filemaker, you'll know that it's a visual DB - almost creating Forms as the first-class citizen (a bit like Hypercard) and the fact they're backed by tabular data is almost hidden. Even more excitingly, it even had basic networking so you could run network queries between DB instances - including exchanging arbitrary objects (e.g. images, videos etc), if i remember correctly.
So, what we did was to set up a centralised DB (iNFERNO itself) on the server, shared over the network, which stored a bunch of chatroom history and provided a graphical timeline view onto the room when opened up over the network from a Mac (set up to occupy the top 2/3rds of your screen).
Meanwhile, and this is the fun bit: we also wrote a DB instance called your "iNFERNO Key", which you stored on a 3.5" floppy disk and could run as a local DB off the disk on any Mac you happened to walk up to. It coughed up a window on the bottom 1/3rd of the screen, which provided your composer prompt. Everyone had their own key, and customised it to their heart's content with different themes, colour schemes, buttons, bot-functionality etc. And it stored their credentials (and we even tried some basic shared-secret e2e crypto) to make logging in trivial - you just put the disk in the drive and clicked the DB (which would in turn spawn the separate main window to view the shared DB timeline).
When you sent a message (which could be text, or arbitrary filemaker objects like images, videos etc), Filemaker let you stage the table row locally and then insert it into the shared DB instance, so you ended up with a surprisingly fun and usable server/client chat system, with arbitrary scrollback, multidevice support, file-transfer etc... using a batshit crazy architecture which involved schoolkids running around waving 3.5" disks around and forking each other's "keys" and somewhat unnecessarily authoring messages using a local DB instance when the whole thing could have been done on the serverside DB instead.
The whole thing lasted about 3 months, getting increasingly popular until the school went nuts at their Filemaker server being completely overloaded and tragically turned it all off. And whilst I still probably have my Key somewhere, we surely didn't have a backup of the central server itself :(
So, my point is: this was a chat system, with many of the features shared by XMPP, Matrix, Slack, Discord etc today. Could we have built it on top of something else? Sure, we could have built it on IRC instead or Talk or some BBS based thing. But instead we chose to experiment with a totally different architecture, which had its own weird trade-offs and benefits, and learned a bunch of stuff along the way. (I still miss the idea of literally synchronising rich graphical objects over the network - it almost felt like Croquet or one of the virtual world things).
And so, whilst you're totally right that we could have tried to layer Matrix semantics over the top of XMPP or IMAP or Filemaker(!), we similarly wanted to try a totally different architecture; one without stanzas and message passing; one without all the historical baggage of XMPP; one which only does group conversations; etc. So I'd argue this is how stuff evolves, and sometimes it's good to experiment with building on a new base. And if it doesn't work out; hopefully it at least provided a learning experience for all concerned. The same goes for COI layering chat over IMAP; almost as questionable as layering chat over a networked DB ;). So, let's just get on with building stuff and seeing what works and what doesn't. This isn't a case of re-inventing the wheel for the sake of it, but seeing how well things work if you build on an entirely different set of foundations.