Now, the problem that gay men seem to have with sexual abuse (while Catholic) wasn't enough to give you pause? It gives me pause. I'm even worried about allowing men (gay or straight) to be baby sitters. Because sexual abuse is an out-sized male thing. Does it bother me that men would be harmed by this restriction? No. Identity politics disgust me, even white male identity politics.
On to your argument. Why have high standards for adoption? Because there are too many people who want to adopt (white kids), and too few (white) kids to go around. Hence, we can be demanding.
Arguments to be "fair" don't go far with me.