A/B Testing the Effect of Sender Race on Email Response Rates
blog.ezliu.com
blog.ezliu.com
Could you break down the data by name and/or gender of both the sender and the respondent? Test multiple hypotheses simultaneously? For example, in page 1002 of the referenced research article (http://faculty.chicagobooth.edu/marianne.bertrand/research/p...), they break down a whole slew of other qualities that may have affected response rate.
It's entirely possible that the name choice for your study has more of an effect than the perceived race association of those names. The "99.9% confidence" cited is really not that, and is subject to various biases which are hard to discern.
Eric left out a lot of wanted/needed details and data about his experiment. Due to the ethics questions involved, it would be best if his data was release and his experiment was evaluated, rather than lots of other people repeating the same or similar experiments.
The harm is already done, so let's try to learn from it.
Maybe I'm just being too forgiving?
Just say it like it is, without putting a slant on it - A's response rate was 0.6% higher than B's.
I don't care if the results are significantly different, if the difference between the two samples is so small.
citing an absolute difference isn't very useful when you're talking about differences in conversion rates.
* 0.6% -> 1.2% is a 0.6% bump.
* 49.4% -> 50.0% is a 0.6% bump.
it sounds like you're saying that if someone increases their landing page conversion rate by 1% (from 1%) you'd rather hear that the conversion rate increased by a point.
the business owner is probably more interested in the fact that their revenues doubled.
However, the relative difference (4.9%) is the relevant metric to be looking at as noted by "ezl" in a comment.
@dantillberg, @jcr, et al: you're right.
i heard about the original study, was curious, but not enough to think much of it. in a previous startup we had a female intern who was getting substantially better response rates than the male founders.
after the recent press about how women in senior roles correlates with startup success i became a bit more curious and wondered if i could craft the perfect "from" field for outgoing emails.
i admit this was aggressive and that I got carried away. that's no excuse.
Let's break down why it is such an atrocious idea.
Say you have a black man in a high ranking position in your company. Oh, but you want to sieze every advantage possible, and you have data suggesting that black people get lower reply rates on emails. And women get higher reply rates on emails. How do you deal with this situation?
Should he have someone else in the company send high-priority emails for him? Or maybe just set up a dummy email for him with a female hispanic name, and pretend it is his "secretary"? Or maybe he shouldn't be in the kind of position in the first place -- maybe he should be in a more internally focused position? Maybe next time you hire, you should be on the lookout for a hispanic woman for an externally focused position, to make things easier.
There are a huge number of problems with these kinds of actions, starting with the fact that you SHOULD NOT be "dealing with" the situation that you have an employee of a certain race or gender.
1. Flat out discriminatory -- forcing someone to jump through hoops because of skin color / gender.
2.a Fundamentally strikes at human dignity by treating everyone you email as racist, genderist animals.
2.b Fundamentally strikes at human dignity by telling your employees to pretend to be someone "more palatable". Or by having someone do a task, lets say, because the individual is in possession of a female hispanic name. Everyone is dehumanized.
3. Liable to twist thinking. After reading that article, and next time you're hiring someone, you brain has a good chance of (inadvertantly) jumping to the notion of how the applicant's race / gender is going to impact email reply rates. We've agreed as a society, for good and just reasons, that hiring decisions and compensation decisions should be completely blind to those (and other) factors.
4. Any actions taken to hide or shield people who seem to be less acceptable or attractive to society simple reinforces their position. The idea that there aren't successful [insert race / gender] people in [profession] is reinforced by hiding those who are.
What's scary is tacit admission that you'd rather censor scientific research than face some uncomfortable truths about human nature. What other studies about human psychology would you censor?
>How do you deal with this situation?
You don't, at least not in a liberal state like California. If I may be realistic for a moment: here, having a high-ranking employee who's hispanic-female-gay-disabled-atheist is an advantage.
>We've agreed as a society, for good and just reasons, that hiring decisions and compensation decisions should be completely blind to those (and other) factors.
This is either naive or delusional, because a lot of people seem to have missed the memo. As much as we may want to be impartial, personal preference will always play a part because we're humans instead of robots.
Moreover, there's the grey area called "team fit". I may hire Juan (Hispanic male) because he's very friendly and helpful, but has less experience than Jason (Caucasian male) who's very experienced but cold and aloof. If you look at this strictly from a technical qualifications point of view, then I discriminated against Jason. If you consider the big picture, then you see I made the right hiring decision.
What I wrote was in response to an article that carried the subtitle "Are Emily and Brendan More Employable than Lakisha and Jamal?"
And included, just above the fold, these gems:
"Question: How can I use societal prejudices to aid in startup success?"
"Race for Internet Marketing and Startups"
There are many fascinating and extremely valuable studies treating race. You are right, they are often uncomfortable. I love those studies. But this post is not that. It not only has shoddy methodology and would get blasted to pieces by peer review, but its expressed purpose is the application of racism and sexism to the world of startups. THAT is what I was writing about.