We had difficulties "finding" each other. Even when sharing invites to groups, entering the group would fail.

I didn't set up my own server though, that was through matrix.org. And probably an intermittent issue.

That video explains the setup very well if one is inclined to run their own server. Thanks for sharing.