If FluffyChat is not having this issue, it is probably overly eager to share encryption keys instead of allowing the user fine-grained access to control keys, which is successfully hiding the complexity of the ratchet encryption but potentially exposing the user to attacks to force the sharing of keys.
Edit: I was looking around. While Matrix is well documented, Element's documentation is poor because they expect you to figure things out from popups in the UI -- fair enough, unfortunately most apps are like this, and Element's popups have gotten a lot clearer. But I did find these two pages from a university that seem to serve well as "Element's missing manual". Worth a read if you are trying Matrix for the first time, because it discusses some things that can look like bugs but are really user error.