Why the Facebook Experiment Is Lousy Social Science
ischool.berkeley.edu
ischool.berkeley.edu
tl;dr: Let's summarize his complaints and my counter-objections:
1. no consent: irrelevant to whether this was good science or 'lousy social science' 2. crossed boundaries between corporations and academia: likewise irrelevant; also, welcome to the modern Internet 3. small effect size: misunderstood the statistical design of study and why it was designed & expected to have small effects 4. used LIWC with high error rate for measuring emotionality of posts: if random error, biases effect to zero and so is not an argument against statistically-significant findings 5. and LIWC may have systematic error towards positivity: apparently not an issue as negative & positive conditions agreed, and the studies he cites in support of this claim are mixed or unavailable 6. also, other methods are better than LIWC: sure. But that doesn't mean the results are wrong 7. maybe LIWC has large unknown biases applied to short social media texts: possible, but it's not like you have any real evidence for that claim 8. Facebook news posts are a biased source of mood anyway: maybe, but they still changed after random manipulation 9. experience sampling is sooooooo awesome: and also brings up its own issues of biases and I don't see how this would render the Facebook study useless anyway even if we granted it (like complaints #1, 2, 6, 7)
Now, I don't want to overstate my criticisms here. The author has failed to show the Facebook study is worthless (I'd wager much more money on the Facebook results replicating than 95% of the social science research I've read) and it would be outright harmful for Facebook to aim for large effect sizes in future studies, but he does at least raise some good points about improving the followup work: Facebook certainly should be providing some of its cutting-edge deep networks for sentiment analysis for research like this after validating them if it wants to get more reliable results, and it would be worthwhile to run experience sampling approaches to see what happens there, in addition to easier website tests (in addition, not instead of).
#1 is particularly troubling since you seem to argue it by misrepresenting his argument instead of addressing it directly. The alternative is to perform the experiment w/informed consent, not to keep the results of your unethical experiment secret. I think this is actually an issue that is important when considering whether an experiment is 'good' or 'acceptable'. Tuskegee is what I think of when people try to argue that informed consent is unimportant. Even though anyone can tell you that the stakes in this kind of experiment are incredibly small in comparison, there IS a measurable risk of harm when manipulating emotions, so ethics matter here.
#2 - you seem to make no effort to address this either. To me, it is particularly troubling if the business and academic sides of FB freely interacted here because each side has an incomplete understanding of the other side's motives and priorities and constraints. The business side is focused on revenue and actionable metrics and legal liability (that's their job), while the research side is naturally going to have different objectives and have a perspective that is more rooted in the practice of good, ethical science. There's real trouble if research ethics & principles end up subjugated to the interests of the business people (which they often do).
Seeing you utterly dismiss the first two points with 'Oh good, so the author isn't a complete idiot.' is not exactly a revelatory argumentative triumph. You can do better.
No, the alternative is to go through a long rigmarole which may or may not approve the experiment and enforce conditions which may or may not themselves constitute severe biases of their own (knowing one is in an experiment, even if the subjects have been deceived as to the content, is itself a problem).
> #2 - you seem to make no effort to address this either. To me, it is particularly troubling if the business and academic sides of FB freely interacted here because each side has an incomplete understanding of the other side's motives and priorities and constraints.
There's nothing to be addressed here. Businesses have always allied with scientists when they find common interests, from astronomers preparing tables for shipping to Gosset inventing a good chunk of modern statistics for optimizing a beer brewery. All he's done is talk vaguely about 'boundaries' and insinuate dark things.
> Seeing you utterly dismiss the first two points with 'Oh good, so the author isn't a complete idiot.' is not exactly a revelatory argumentative triumph. You can do better.
Those points are fundamentally irrelevant to the overarching claim he makes that it's bad science. I think he's wrong about the more... philosophical aspects of things, but I don't have to show he's wrong - he has to show they materially affect the truth of the results. He has to show that #1 and #2 actually matter, and he hasn't. Instead, he spent an incredible amount of verbiage on vaguely related topics and he flubbed the technical criticisms. I don't give him a pass on that and neither should you.
Having gone through some IRBs, there is a good reason and enough historic precedent to warrant this "long rigamarole". The point is that you can't simply say "but everyone on the Internet does it" and expect people who support traditional research ethics not to say "wait a minute, that shouldn't make it OK".
I don't see how this is different from the claimed 10-90% of positive posts. Similarly for negative posts. Am I misunderstanding something?
There's a reason why psychology studies (any human study really) require informed consent. There is potential for great damage and Facebook apparently doesn't care.
Now I can hear someone saying that advertisement is the same thing. Advertisers are trying to change peoples' emotions, yes, but we know and recognize that while watching an ad. People using Facebook do not expect to be manipulated like the study did.
I would be just as upset if it were Google or my local grocer.
If this is something that upsets you I suggest being upset at every commercial on the radio / newspaper / tv / billboard that you're unfortunately exposed to every day.
People have been calling out various bad practices in conversion: http://darkpatterns.org/
So it's not bad because everyone else does it?
A real-world comparison of the two different types of experiments:
Grocer wants to see the effect of a product demo's effect on sales. They have a worker offer a product sample, then recommend a pairing of products vs offering a coupon for that product. A/B testing.
Grocer wants to see the effect of a product's demo on the customer's emotions. He has the demo worker gripe about their boss, or talking positively about the community, then asks the cashiers for followup feedback on the smalltalk at their counters.
One of these is OK. The other is seedy.
It is a requirement instituted by the IRB in most universities. That is the problem being mentioned here.
Furthermore, Facebook is a different case from most companies because it is an intrinsically personal website that has the potential to affect a user's life very meaningfully. Amazon changing the design or location of a button doesn't affect me as much.
https://medium.com/@gpanger/why-the-facebook-experiment-is-l...
...where I can fix things like broken links. It was republished on my school's website, but unfortunately I don't have control over the HTML there. In the cases where stuff broke, it's because I linked to the author's personal manuscript and not the official journal page (because the former is freely available to everyone rather than behind a paywall).
I care a lot about Big Data research, especially involving social media, and think we too often ignore the conceptual leaps required to make inferences about the human experience from social media.
Here, the leap is: sentiment_analysis(what people say on social media) == how they really feel.
The point about LIWC (the sentiment analysis tool used here) is that (a) it's flawed, and perhaps not in a "random error" kind of way, (b) we don't really know how well it works because it has not been validated in social media or Facebook posts specifically, which should make most researchers nervous (but somehow doesn't), and (c) there's evidence from other data sources that suggests LIWC is biased in a way that would underrepresent the emotions of interest (namely low arousal emotions like sadness, depression, loneliness, feeling left out, etc.; these aren't picked up by LIWC as well as other negative emotions like anxiety).
See e.g.:
O’Carroll Bantum, E., & Owen, J. (2009). Evaluating the Validity of Computerized Content Analysis Programs for Identification of Emotional Expression in Cancer Narratives. Psychological Assessment, 21, 79-88.
The point isn't that using LIWC means the experiment is invalid, the point is that it should give us pause and caution us against stating the conclusions of the experiment too strongly. I think the authors do state their conclusions a bit strongly.
The other main critique is about biases inherent in social media as a datasource itself. The private, randomly-solicited emotion samples of experience sampling are more likely to capture Facebook's true emotional impact than the non-private, self-selected emotion samples of status updates. Let's just take arousal bias. If we know that people are more likely to post when they're emotionally aroused (excited, angry, fearful/anxious), but the emotional consequences we're concerned about involve low arousal emotions (sadness, depression), then there's a serious chance we'll miss exactly the emotions we're arguing don't exist. That's a big problem.
I think we're a bit too enamored with the idea that Big Data provides an unbiased window into the human experience. I think a tremendous amount of social science would argue otherwise.
Stepping back a bit, the Facebook experiment raised many interwoven issues for me, which is why they featured in the broader piece I wrote. About Facebook's culture, about ethics in corporate vs. academic research, about Facebook's emotional impact (foolish to believe there is none, I think), about how we use Big Data in research, about how we cope with Facebook's presence in our lives, for better or worse.
You may have a great experience with Facebook, and that's great. Others struggle with the medium. I mention social comparison not just because it's been the focus of research on social media, but also because the authors of the experiment bring up social comparison (as well as the "alone together" argument) in their work. Because they try strenuously to rebut the unfavorable findings about Facebook's emotional consequences, I thought it was important to point out that their study seems designed in a way that would systematically underrepresent exactly those negative emotions they're arguing against.
Certainly, some negative emotions were "contagious" through social media (anxious news reports, for example), as were some positive emotions. But is the emotion that gets retransmitted the full emotional picture? Probably not. Probably many emotions and feelings get withheld. The social science would suggest that when positive posts make us feel bad, we won't go back on Facebook and broadcast those feelings to all of our friends.
Thanks a ton for engaging with my critique and responding with your own. Writing that was a labor of love, and I learn a lot from the feedback, good and bad. Sorry it was so long.