You're asking this question with some knowledge that there is some difference in the minds of people between the two matters. Therefore, we can draw a difference, we just may not have a philosophical grounding to express the difference rigorously. I can see how that's a problem, but it does not stop consideration of the principles. Even if the difference is arbitrary, we can't escape arbitrary reasoning in our moral and legal systems.
The difference from my perspective is that I'd be more inclined to protest a law criminalizing gay people for (using your example) having sex in private than I would be someone arguing in a public forum that Jews should be punched in the face.
Is my decision arbitrary? Maybe. Does it rely on a judiciary and legislature that agrees with my ad-hoc principle in order not to run away with it? Yes. That's not necessarily a problem. It's not much to ask people to think beyond pure abstraction of high-minded principles.
You (or some other citizen) may disagree with me and come to the opposite conclusion I have, or not. That's fine. I'll vote for my principle, and you can vote for yours.