So I'm a man, and I'm married, and I'm committed to my marriage. I'm not just worried about my reputation; my marriage is a more important issue. Unfortunately, that doesn't mean I'm immune to temptation. So if I'm having to work closely with you, and you're interested in me,
and I also find you attractive, I'm going to take steps to distance myself from you. That may take the form of not working late with you. It may even take the form of asking to be assigned to a different team than you. (Note well: It's not OK for me to try to get you fired, or damage your career. It's not even OK for me to ask that
you get reassigned.)
If you want to blame me for that, saying that it's my weakness that is the problem, I can agree with you. But I'm not trying to discriminate against you, or put a glass ceiling on your career, or anything like that. I just know that I can be tempted, and if I give in, it will really mess up my life.
Unfortunately, that can still leave you blocked from your professional goals. I don't know how to fix this.