I agree gender is irrelevant here. I'd even go so far as to agree that there may be more than two sex states on individual bases (multiple X or Y chromosomes, different levels of testosterone, hermaphordism, chimerism etc). But these are neither particularly common nor genetically transmissible and thus do not exert pressure on the nature of sexual dimorphism. Sex can be modelled quite accurately with a single binary bit even in individual cases, and entirely accurately for species in evolutionary terms. Distributions and multiple dimensions have not shown themselves to be useful. If you disagree I'd like to see references to the relevant research papers.

I don't mind recognizing any individuals uniqueness or their socially less-privileged situation for social or political purposes, but it shouldn't have any affect on the science of sexuality which should just reflect nature as it is.

Sexual dimorphism in animals can be dramatic. The male angler fish is a tiny parasite, the dramatically larger female does all the angling work. And we aren't seeing non-binary angler fishes. If none of the other species are coming up with non-binary specimens, why should we assume humans are any different?