It would be more like saying "I'm not racist because my spouse is black".
Up until about the 70s or so, racists campaigned heavily against interracial marriage. Most racists likely still hold these views, though after 1967 they are no longer relevant so you don't hear about it very often.
I do not recall any sexist groups ever campaigning against heterosexual marriage. Now, sexists have campaigned against laws that would equalize the rights that people in marriages have.
Therefore "I am not sexist because I believe that my wife and I are equals and should be treated as equals by the law" would be an analogous statement.