Researcher at MIT analyzed 33K NYT articles on Palestine and Israel using ML
twitter.com
twitter.com
It seems to me (superficially) that "like-for-like" metrics such as "use of passive voice" should be fairly independent of reality; for a similar example, see the American tendency to refer to "officer-involved shootings" (but, in a clear case of bias, to never say "dog-involved bitings").
On the other hand, as you note, "use of violent terms" could be a reflection of real-world differences in violent actions, so it might be a more fraught metric.
I'm really intrigued by the idea that we can bring this kind of quantitative analysis to bear to reveal hidden bias, though; it sounds like it can be a powerful approach. But you're certainly right about potential pitfalls.
That said, the paper itself is a good read.
http://web.mit.edu/hjackson/www/The_NYT_Distorts_the_Palesti...
This is a really bold statement for any analysis and seems odd to come put of academic work that’s supervised.
This preprint is submitted to the Journal of Palestinian Studies so it will be interesting to see how their peer review checks this out.
I don’t quite understand this conclusion based on the paper as it seems like the author looked for existence of some words and for use of active vs passive voice.
I don’t understand how active vs passive voice is a sign of bias unless you control for many other factors not mentioned. Are specific authors more likely, is time period affected, are other topics similar, etc etc. I don’t get the logic that this shows a bias in general and don’t see anything presented about why these 33k articles are different.
Similarly, I’m not sure how tracking the mention of Palestinian or Israeli people just by word count means anything. Wouldn’t this be impacted by events, this isn’t addressed in the paper. Are there more deaths or attacks in different groups. This seems important to address, but I couldn’t find anything. I’m also not sure how the 33k articles were selected so whether articles were included or excluded would impact these results.
Similarly not sure if the sentiment analysis of the NLP kits she used is similar or dissimilar from NYT in general.
And the determination of violent language seems like a manual exercise with no validation and would like to know more about the methods used to create the set of violent words. I don’t see the actual training_data.csv that the author created and would be interested to see what was actually made since that file seems to be the output of a manual process to sample 500 articles and pick words.
I do think the "active vs passive voice" is an interesting metric, since it seems intuitively a bit harder to know why that would differ for the two groups.
As I noted below, in the US it's common to note that the phrase "officer-involved shooting" (i.e., passive voice) is often used to describe cases where police officers shoot someone, but "dog-involved biting" is rarely used to describe the case where a dog bites someone. This strikes me as a convincing (though perhaps not ironclad) argument for how such metrics—which are seemingly independent of the rate of mentions (which itself could be reflecting world events)—can shed light on authorial bias.
I think you could construct similar "event-independent" metrics; for example, are victims of violence more or less likely to be named when they are of one group or another? Are direct quotes from spokespeople more likely to be used from one group or another? Etc.
I generally argue against trying to make all journalism just "impartial he-said, she-said", so I'm a little uncomfortable with this exercise, but, again, the "officer-involved shooting" case is one where very clear language bias seems to affect the discourse.
Both of those seem just stated without evidence so it seems odd that the MIT supervisors would have a student write this us.
It's obvious. Any entity or agency that sides with the Zionist state, which was formed on stolen and occupied Palestinian lands, is biased. The tides finally seem to be changing, and truth is starting to dawn on people.
I think the data the paper presents doesn't warrant the conclusion.
On the contrary, being "not biased" is predefined here as it would require you to use the same amount of active/passive voice and terms for violence without regard to what is actually happening.
That can be dangerously wrong. It could also mean that Palestinians commit more violence for example. Doesn't mean the NYT is biased. You would need to quantify reality first to be able to compare it to language used.
After trying to look it up I found this article (2003):
http://itre.cis.upenn.edu/~myl/languagelog/archives/000236.h...
I thought that funny.
But I think it remains true, that you cannot infer bias by only observing language.
Also oddly, the paper didn’t include a statistical analysis of whether these differences are even significant.
Yes, we need to have a ground truth with regard to violence committed. But that data is already published elsewhere
If Israel attacks Palestine and kills a citizen, Palestine retaliates and kills 5 in return. Should they have stopped after killing just one person? It that a non-biased approach? What is the ethical approach? Kill just one? Kill nobody? Kill as much as you can?
Doesn't allow for conclusions about bias either, especially not in an armed conflict.
I don’t see any attempt in the paper to try to do this or to use your source, or others.
The number of fatalities would be hard, I think, to try to show what the violent wording would be used since it quantifies deaths (eg, 200 deaths in a single incident would be reported different than 200 events with one death each).
But perhaps trying to determine a correlation of reporting to deaths, or reporting to events. Or any attempt.
The biggest problem I have with the paper is that there’s no null hypothesis to disprove, so it “proves bias” without establishing or even estimating what a non-biased corpus looks like.
For example, I would expect to find that Fox News uses more violent descriptions of BLM protestors, and that the Washington Post uses fewer; I would expect the converse when describing the Jan 6 US Capitol riots.
This kind of comparison doesn't say whether the BLM protests are more or less violent than the Capitol riots, but it would say whether such an analysis hews to our intuition about relative bias between news sources.
However, choosing such a charged topic needs to be managed by an advisor - @h_jackson_ is about to get a lot more than she probably bargained for in terms of people arguing good faith, twisting her results to their narrative, and several arm chair scientists that are going to tear her down despite probably never actually reading what she's worked on.
Is her analysis perfect, no. Is it likely flawed, yea probably. This is where a professor can try to tone down the flames and push toward what might be next in the science of getting a right answer.
i.e. when you go in with this belief to quote her "The results of this study are now as relevant as ever. Today, the New York Times continues its legacy of Palestinian erasure."
how can one expect the results to be anything but something that reflects that. either the author would continue to tweak and tweak and tweak till they get the results desired or simply not publish something that contradicts their worldview.