Relatively late in the legal evolution of civil rights jurisprudence the Commerce Clause was interpreted to not only expand pre-existing anti-discrimination powers, but to provide an independent source of power to ban discrimination. This is why Congress has the power to regulate discrimination against the disabled, for example. Nonetheless, the further away from the plain text and history of the Constitution, the more tenuous Congressional powers become. So, yes, Federal laws against disability and age discrimination have the least constitutional support and are also commonly construed the most narrowly.
Just as important, all of those textual sources grant affirmative powers to the Congress. For example, section 5 of the 14th Amendment says, "The Congress shall have power to enforce, by appropriate legislation, the provisions of this article." The 19th Amendment says, "Congress shall have power to enforce this article by appropriate legislation."
Remember, the Federal government only has enumerated powers. Nowhere is political affiliation directly discussed in the Constitution in a context that would permit it to distinguish political affiliation as a cognizable type of class, and certainly not in a context granting Congress affirmative powers to differentiate based on such class. Political affiliation would best fit with the Free Speech clause of the First Amendment, which the Federal government has powers to enforce through the so-called incorporation doctrine of the 14th Amendment. But Free Speech jurisprudence is concerned with content of speech, not classes of people, unless we're discussing journalists as a class.
Freedom of Association doesn't fit because, similar to Freedom of Speech, it doesn't imply the power to differentiate by association, per se; that is, it doesn't imply the power to recognize group A and group B and give group A special protections. Quite the contrary, it implies the inability to differentiate among them.
The Commerce Clause doesn't work. As broad as the Commerce Clause has been interpreted, political affiliation is perhaps one of the last remaining areas that isn't construed to directly implicate commerce.
Basically, the reason political affiliation isn't normally a cognizable class is because there's no direct nor even indirect source of authority that grants the federal government the power to regulate in such a manner. Plus, there's a whole host of reasons why permitting the government to differentiate people based on political affiliation would harm more fundamental prohibitions against discrimination.
EDIT: Regarding the question of gay marriage, note that Kennedy's opinion doesn't make homosexuals a protected class. Rather, the basis of that opinion is that the government lacks any power to differentiate between homosexuals and heterosexuals when granting marriage licenses. That said, there's a liberal vein of jurisprudence that construes LGBTQ discrimination as a form of sex discrimination.
I'm not being smart - this is the genuine way to approach the problem. At some point you started with no protected classes and if there was a reason that was considered good enough by enough people, then the list was extended. So if a group in society/legislative branch thinks there's a reason to make the change, the onus is on them to justify it and convince enough people to join that opinion.
So if you want a genuine discussion on the topic, you should start with a strong argument about why these views should be a protected class.
The big exception to that that I can think of is religion. One could debate why that’s a protected class and whether it should be. But in the US freedom of religion is part of our founding myth and enshrined in our constitution, which probably has something to do with it.