That analogy doesn't work. A closer analogy would be a vegan refusing to talk to a coworker who eats meat. That would be unacceptable to me, too.
Could you explain why you think your analogy works, please?
Your right to untrammeled adherence to your philosophical beliefs ends the moment that those beliefs result in conduct affecting other people. After that point, some form of balancing occurs.