Take your reasons, which are really more opinions, and insert "free speech" instead of "unions" (after the appropriate changes) and you might understand why people would disagree.
- I don't want free speech because it is unfair to award lazy thinking free riding off established publications.
- I don't want free speech because it is often exploited by terrorists.
- I think people should be free to say things but I don't think it should grant them special protection from the government.
etc.
It isn't an exact comparison but that also isn't the point.