You're presuming something that baker does not agree to. The baker views gay marriage as morally wrong or even an absurd notion. Punishing him because he has moral objections that you disagree with by framing it as a matter of "discrimination" (understood as bigoted treatment) is to ignore that his motives and perceptions may involve no bigotry at all in his view.
Soapboxy post scriptum:
This is what I dislike about these "neutrality meta" discourses. "Neutrality" is ultimately just another word for "common ground", but the less common ground you share, the less of this neutrality there is to work from. Now, in this case, your characterization of the baker's actions as "discrimination" is completely opposed to his view of his actions, but you presuppose your characterization as an element of this neutral common ground by which we can judge him. From his point of view, he may see this as an act of virtue. He may even see this as an act of love for the gay customers by communicating to them that their relationship is illicit and self-destructive. Indeed, that would be the traditional view. If someone's an alcoholic, you might kindly inform them for their own good.
So not only do I see little point in pretending to a false neutrality, but you can't both reduce everything to a mere matter of interpretation and opinion, a plurality of incommensurate views, and then turn around and elevate your view as the neutral one. Either there is a truth of the matter about which we may disagree but which can in principle be resolved reasonably or which can be tolerated, or there is no truth of the matter and it's all at bottom merely a matter of power and accident. In which case, it seems a little silly and contradictory to hold the latter and simultaneously express moral indignation at someone who is behaving in accordance with his own views. You only get to do that, maybe, if you believe there is a truth of the matter to which this other person is bound but egregiously violating.
But if you believe you're right, or you love the truth and believe that it can in principle be known, have a humble and sincere desire to know it and a willingness to discuss and consider and respectfully argue, as well as the courage to humbly accept conclusions that would put you at odds with others you would rather not be at odds with, then it should be possible to arrive at some measure of agreement. I think it is much better to start there with the hope that over time we can make progress, not through force but through reason, rather than to give up on the possibility.