> 3. RT is based on people's opinions, but news should be fact checked.
Except when RT doesn't like to hear people's opinion when it differs from the mainstream. Unfortunately it's easy to box it as "trolling" and hide it under the rug.
Except when RT doesn't like to hear people's opinion when it differs from the mainstream. Unfortunately it's easy to box it as "trolling" and hide it under the rug.
What would make a similar site for news safe from such trust abuse?
RT is a company that is currently owned by Warner Bros. (through a chain of ownership). This alone is enough for someone to raise the possibility they may not be completely unbiased in how they treat each movie despite having any evidence of such behavior. Conflict of interest, in my own experience, have never required evidence of any further problem outside of the conflict of interest itself. Since the end result is merely movie ratings which is a fully subjective experienced, it isn't worth time digging any further into.
When it comes to a news rating site that will likely be deciding things of a more factual nature (news is true or not, or at least somewhat based on true events), I think a potential conflict of interest has larger implications. Whom ever owns the site will have a conflict of interest with regards to at least some news, and the ability of that site to be used by other groups like non-profits, NGOs, or even governments will need to be limited with respect to where the conflict of interest will occur.
I don't think the difference is in the difficulty of making a RT for news, but on the downstream impacts of the conflict of interest that exists between owner and purpose.
I know a bunch of folks recently complained that they were trying to manipulate the scores for Captain Marvel, but that was also under a huge organized attack to torpedo the audience scores even before it was released for viewing. So they had a legitimate reason to go in and try to "fix" the scores. Whether the new values is "accurate" is of course up to debate (for philosophical values of "accurate") but most of the criticisms I've seen have had reasonable explanations behind them.
Like I said, with movies reviews it does not matter (at least in my opinion). The issue I have is that I don't think such a carefree attitude can be taken when it comes to rating the truthfulness or applicability of news.
As for any examples of actions, I do not have access to the sort of data to determine if any such actions were taken, nor do I have the resources nor desire necessary to even begin investigating. Also, especially in relation to recent events that touch on a more political scale, I rather not deal with the issue of evidence being trusted based on the extent it confirms existing views (a cognitive bias common to humanity that isn't particularly worse in this case, but which I have seen result in flame wars on other forums).
- For audience reviews, reviewers rate out of 5 stars, and the score is the percentage of reviews at least 3.5 stars.
The goal is to insulate scores from extreme opinions and odd voting patterns. On IMDB, which doesn't do this, a few low scores can destroy a movie's rating, and movies that invite careful analysis are effectively penalized. The Dark Knight ends up rated higher than Schindler's List, but the latter has lots of slightly-imperfect scores with incredibly positive review text.
The basic side effect is the loss of all meaningful score data. That punishes daring movies: 100 4-star reviews equals 100% while 50 3-star and 50 5-star reviews equals 50%, but I would much rather see the second movie. It also hides deep flaws: The Boondock Saints outscores 2001, but a look at the reviews shows that its "rotten" ratings are often 1 and 2-star reviews calling it garbage, while 2001 is full of 3-star reviews like "slow and confusing, but fascinating". Another problem here is that naive binary scores suffer from reviewer skew and vote-bombing campaigns. It's less bad than IMDB because extreme scores are 'softened', but still worse than systems which relate individual scores to the reviewer's other opinions. BeerAdvocate, of all things, takes on this problem with interesting meta-info like "reviewer's average distance from consensus".
- For critical reviews, each review is reduced to a binary opinion and the score is a weighted average of positive review count.
The goal there is to condense varied or absent rating schemes into an aggregation-friendly score and avoid reviewer skew. Metacritic, which doesn't do this, ends up systematically distorted by which outlets review things. Unscored reviews like Ars Technica's (on video games) are skipped, while sites with extremely skewed scores are included by some unspecified weighting system. As a result, Metacritic lists game reviews averaging 75-89 as "generally favorable", but there's a widespread understanding that any major-release product below 85 is terrible. (GameSpot once infamously fired a reviewer for scoring a game 6/10.) It's largely the same problem as Uber and Amazon ratings: does a perfect score mean "exceptional" or "nothing wrong"? RT tries to dodge the problem by saying "good" or "bad", with a weighting to accommodate a reviewer's 'signal strength'.
The side effect here is, again, the total loss of secondary data. RT sometimes gets reviews actively wrong; something like "despite lovely action sequences, this movie is a meandering waste of time" will get parsed as "lovely action sequences". That's probably an automation error, but lots of other reviews can't be clearly declared positive or negative even by humans. And within a categorization, there's a great deal more lost data. Ebert produces 'fresh' for The Avengers because "It provides its fans with exactly what they desire," but he produces 'fresh' for There Will Be Blood by declaring it "A force beyond categories."
- The reason all of this works alright for Rotten Tomatoes is that it's fundamentally answering one question: "will I enjoy watching this?" The Avengers isn't on par with There Will Be Blood, but if you like the description and it's rated fresh, you'll enjoy either one. If you see that critics hate The Boondock Saints and audiences don't, well, you probably know whether you're somebody who likes that kind of cult classic.
But what of news? People already shred Snopes whenever it tries to deal with assessments like "factually correct but misleading" or "incorrect statistic but the real number also supports the same thesis". The entire point of news reviews is to not rely on "will I enjoy this story?", so all the things RT quietly glosses over will become critical problems.
There's a lot to be said for intelligently dealing with ratings and their metadata in ways like this.
For example, the Kappa statistic: https://en.wikipedia.org/wiki/Inter-rater_reliability
Presenting crowdsourced ratings as mere averages -- or really any collapse onto a single scalar -- is a big step up from no info at all, but it's hardly the best one can do even with well-known statistical techniques.
Histograms are a screamingly obvious way of distinguishing "mediocre" from "some good some bad", which is one of the most common needs with things like Amazon products. But beyond that, there's so much more to be done. You can weight or shift scores by reviewer's average, reviewer's average distance from consensus, or a dozen other things. A one-star review from someone who uses Yelp exclusively to call out bad experiences is relevant, but a one-star review from someone who often gives 4-5 is far more interesting.
Maybe the weirdest thing is that a lot of this is done to catch fake/paid reviewers, but it's not extended to providing clearer info overall. Even the fight against fake reviews would be much achievable if it was shifted from a binary "take down or don't" to a more flexible approach to maximizing review value.