Reasonable restrictions on marriage include being under age and certain close relations.
Polygamy isn't a relaxation of a restriction of marriage, it is expanding the concept. For example, if I am in a coma, my wife can make medical decisions for me. If I have two wives, what if they disagree? If two people are married, must they both agree on bringing a third into the marriage? What if one person from a three person marriage wishes to leave?
None of these questions arise with gay marriage. The closest gay marriage comes to these problems is that it exposes the inherent sexism in child custody issues, since with two men there's not a mother involved who the court can presume is the better parent.
As a libertarian, in theory I favor getting the state out of the marriage business, but in practice, marriage is a useful bundle of rights and responsibilities. That is, it's a standard contract between two people. We should make it possible for larger groups to enter into agreements granting the kinds of rights that marriage grants, but there is nothing discriminatory about saying that the state provides a standard simple contract for two-person marriage but if you want more than that, you'll have to see your lawyer.