What solution do you propose? Take away the right of association from one group of people to enable another group?
That seems strange to me. If you allow the government to create a precedent where you can restrict the rights of anyone, then you can restrict the rights of everyone.
That idea makes me very uncomfortable.