> Your premise is that, rather than free association, every group of people owes a kind of due process to everyone else, presumptive of their right to associate
If you "start a project and not allow [me] to contribute" then you're by definition starting a much more traditional, hierarchical, authoritarian community, with the forms of dispute resolution you would expect there. Now that's okay, but it means that you should probably be upfront about it from the onset. And it also means you should expect pushback and accusations (probably accurate ones) of bait and switch if you change directions midway. They will say "that doesn't sound very egalitarian to me" and they will probably be correct. It shouldn't be a problem if you don't want to court that segment, or power your community through their contributions. But if you do want to court that segment, then I think they will want you to play by their rules.
I also think this is why the idea of Anglo common law was brought up. It is a form of dispute resolution which is pretty amenable to egalitarian, decentralized dispute resolution. What I like about Anglo common law is that it minimizes the need for a fair authority figure in favor of a fair autonomous process operable by a majority of equal peers:
> Does it allow anonymous accusations?
> Is the accused allowed to know the charges against them, before a finding of guilt is rendered?
> Is there a presumption of innocence?
> Is the accused allowed to have a trusted third party - one who knows the rules of the game - to advocate on their behalf?
> Who, exactly, is responsible for deciding matters of fact vs matters of "law"?
> Is there an appeals process to fix possibly incorrect decisions?
All of these concerns must be addressed to create such a system. And I think that a lot of prospective contributors (such as myself) do not want to be part of a community unless it is run by such a system. I also think that such contributors will be particularly incensed about contributing to such communities that say they are run by such systems but which are actually oligarchies. In that case, I would feel like I had originally contributed to the community as an equal, but I am instead now just a cog helping build someone else's dream. This will not just discourage me from contributing to the community, but it will make me regret ever doing so in the first place, and resent the community's leaders for lying to me.
So going back to your original example, I think you are well within your rights to start a project and not allow me to contribute. But if you do that, I think you have to be honest about what kind of community you are really building. It is one thing to say you wish to build a community through benevolent dictatorship. It is another to say you wish to build a community through distributed consensus. It's not really a two-way door. Switching directions midway can cause collateral damage with an extremely high blast radius.