The paper states that they used an existing net, VGG-Face, to exact facial features: https://osf.io/fk3xr/ (page 13) >VGG-Face aims at representing a given face as a vector of scores that are as unaffected as possible by facial expression, background, lighting, head orientation, image properties such as brightness or contrast, and other factors that can vary across different images of the same person. All the heavy lifting for this paper by was done logistic regression on the facial features reported by VGG-Face, they didn't train a DNN specifically for identifying sexuality. The algo wouldn't have seen clothing at all.
It's pretty disturbing how quick people are to dismiss papers that make them uncomfortable, latching onto the first excuse to dismiss it without even confirming if the excuse is valid or not. Clear Cognitive Dissonance in action.