Curious why you chose invite-only access. Was the goal primarily spam prevention, community building, or reducing moderation requirements?
It's designed for people that already know each other who need a private space, not a large platform for anybody.
There is also some built in protection for when someone new is invited to the server. Each member needs to allow that new member access. The idea is that I, the server operator may have welcomed someone to the community, but a server member may not feel the same way, so their content is not shared with that new member until they decide too.