Put plainly, this is the freedom to form groups that govern the behavior of their members. A group could be a dinner party, a study group, a book club, a circle of friends. It can also be much more serious. It could be a corporation, a union, a church, a political party, or a religion. Or it could be an HOA, or a town, or a state. If you like one or more of those types of groups then it is important to protect the freedom of association, even if you don’t like any of the others.
Every group has rules, even the most informal. When you invite your friends out to dinner you probably don’t sit down and vote on the exact rules your dinner party will follow, but your party has rules regardless. And every group punishes violations of the rules somehow. If you get drunk and dance naked on top of a table, then you will suffer some form of sanction by the other members of the dinner party. At the very least you probably won’t be invited back. More formal groups like HOAs, corporations, and towns tend to have written rules and proscribed punishments for violations. They also have built–in ways of changing those rules.
Ultimately some HOAs are useful while others are petty and run by vindictive idiots. An HOA that was useful for 50 years and never created any drama could turn bad next year when they elect some new board members. An HOA that is bad today _could_ elect new board members and then become drama–free and stay that way for 50 years, even if it seems unlikely. People who end up in a petty HOA usually regret it, and come to see joining any HOA as a big mistake. People in a sane HOA often wonder what the fuss is about.
But here’s the truth: true freedom includes the freedom to make mistakes, even big mistakes.
The freedom of association allows us to join any group we want, but sometimes joining will be a mistake. You can’t prevent people from making those mistakes without limiting their freedom. You can’t even reliably predict which mistakes they’re going to make, so you would end up preventing them from doing things that wouldn’t be mistakes too.