A/B testing the gnupg developers. Submitting patches as female and male pseudonyms
twitter.com
twitter.com
Unconscious bias is a real thing, and we all know that, and most of us try to ensure we are as meritocratic as possible. The problem is, it's really hard to measure your own success at that. I'd like to think that I'm going to be fair and equal, but it is always hard to inspect your own bias. How are you meant to ensure you are inspecting that code as well as you would if the author had a different name?
Of course, it's possible that there are devs that are just straight up bigoted. It seems like the person tweeting really dislikes the project - it'd be good to see more than tweets to get a feel for the full story and see the numbers. If it's 100% of a non-trivial sample size, that's pretty damning.
Second, how did they determine pseudonyms as "male" or "female" sounding? If you want to study unconscious bias, you'd need to know whether the things that supposedly trigger it actually do so.
Furthermore, did they account for their own unconscious bias? I can imagine, using the same unconscious bias theory, that they sent self-assessed lower quality patches with "female" pseudonyms unconsciously, to prove their theory correct.
Lastly, the 100% is very close to unbelievable, having an implicit assertion that identity politics are the primary driver of motivation in the developer team, way above any coding merit. To me, this is more likely to be projection from an identity politics activist(for whom, by definition, identity politics are the primary driver).
This isn't true. As long as the patches were IID (e.g. she generated patches and randomly chose which name to submit them as), it's a legitimate A/B test.
https://en.wikipedia.org/wiki/Independent_and_identically_di...
Second, my primary source for whether a name is "female- or male- sounding" is historical lists of most popular baby names for each biological sex, found through google searches.
Furthermore, I certainly have bias. Unfortunately, I have so much bias! As someone who, in grade school, was named Charlie, and had the shit beaten out of me for it, I very strongly know how it feels to be a man/boy who doesn't fit the socially-prescribed norms. However, when I am writing patches for code which determines a user's security, I do not play games; I want that code accepted, no matter what. The code submitted as a "female" was usually either rewritten by a certain gnupg developer (often to include further vulnerabilities and UX pitfalls) or later accepted as a "male".
Lastly, if you submit 10-30 patches over a decade, it's not very hard to make the acceptance/rejection rates come out to nice even numbers. I don't think there's anything as high-level as "identity politics" at play here; I'm merely suggesting that there is bias and that all of us, no matter what our gender or how equitable we are, may have it.
I would have never suspected this, but I don't think it really addresses the concerns of unconscious bias. The scheme yummyfajitas proposed seems to address both(although you would have no way to prove the randomness of your choice).
I'm sorry if I sounded too harsh in my initial criticism, it was all hypothetical without any knowledge of you or your specific situation.
Also if anyone else repeats this experiment, note that there exist crypto techniques to make it more difficult for the experimenter to do the experiment dishonestly: https://twitter.com/petertoddbtc/status/811559259590660097
(I would have said "prove the experimenters honesty", but those techniques aren't a 100% thing - you could for instance do the whole commitment/random beacon process multiple times, and only publish the one set that gets the results you want.)
edit: To be clear, submitting the same patch twice isn't a good way to do this - if I were a maintainer and learned that an identical patch had been received by two different pseudonyms, I'd certainly consider rejecting it purely on the basis that something fishy was going on; that test doesn't say much about maintainer bias in normal situations.
The process by which names get "cooties" like this is rather nasty. People seem to be uncomfortable with even the possibility of both-gender names like "Sam" and "Alex".
There almost certainly is someone in the development team for whom this is important, given the reports that identical patches would be accepted after being "laundered" through a male dev. People will go a long way to defend their biases. For example, I expect this entire discussion to be flagged off the front two pages of HN and into oblivion fairly quickly.
> meaningful study
mjg59's reply: "Remember kids: never share your lived experiences without presenting your experimental design first".
I guess you were right. As soon as I was done reading your comment I refreshed the page and noticed it got flagged. Not sure what that means exactly on HN but it's kinda worrying that you were able to predict it so easily
mjg59 has observed this happening a lot before on e.g. http://hnrankings.info/13204464/
Have I mentioned I've been A/B testing the gnupg developers over the last decade by submitting patches as both female and male pseudonyms
SPOILER ALERT: they denied 100% of the patches with female-sounding pseudonyms
the way i submit patches now is i email one of the decent devs and ask them to pretend they wrote my code and send it to werner
so far my data shows deviance is high, so it's mostly that there's the really awful projects and the ones with decent human beings
> However, women's acceptance rates are higher only when they are not identifiable as women
> For outsiders, we see evidence for gender bias: women’s acceptance rates drop by 10.2% when their gender is identifiable, compared to when it is not (2(df= 1;n= 18;540) =131;p < :001). There is a smaller 5.7% drop for men (2(df= 1;n= 659;560) = 103;p <:001). Women have a higher acceptance rate of pull requests overall (as we reported earlier),but when they are outsiders and their gender is identifiable, they have a lower acceptance rate than men
Accepting arguendo that this is clear evidence of a gender bias, its effects are not as clearly pronounced as the tweet tries to imply they are.