I'm not sure I understand. If in fact the same underlying encryption protocol is used, why can I not retrieve the public key of a Matrix user, encrypt the message and send it? Nothing here requires the server to break E2EE.
You can exchange the private keys, but using what format? Once that's done, you can encrypt XML on one side, but Matrix has to convert it into JSON on the other (in its own, separate format with its own fields). This can be done on the server side after decryption (no more E2EE), or on the client side (after decryption). In the latter case, you're not running a bridge anymore, you're just running an ad-hoc client for XMPP inside your matrix client.
> <message type="chat"><body>Hi there!</body></message>
or raw text
> Hi there!
In Matrix, it's always encrypted JSON
> {"type": "m.room.message", "content": {"msgtype": "m.text", "body": "Hi there!"}}
It would be possible for Matrix to indicate that a room is a "raw room" and that clients should send and expect raw messages in that room - losing a few features on the go. Such raw room could be bridged over to XMPP fully end-to-end-encrypted (with small changes on either side to adjust for the differences between OMEMO and OLM).
But I doubt anyone at Element will invest time into doing it and for the XMPP side it's much more work to do it "on their own" (by implementing parsing support for the encrypted JSON of Matrix as shown above).
Having raw text in the encrypted content is by the way still the normal way how things work in XMPP world and is supported by every client doing E2EE - because it is basically free to implement (barely any additional code needed).
If you did something stateless like OpenPGP then you could do E2EE over bridges with no problem. The cost would be that each and every message would be complete in itself. So sending a single emoji would take several hundred bytes. That might be less important these days...