His fame is very subjective. He's not famous in a way that is attractive to the average woman, hence his difficulty in finding an ideal mate.
I would presume that if Stallman has romantic troubles, it's due to something other than people dumping on him.