The reviewer’s fallacy: when critics aren’t critical enough

Roger Ebert used to call things "a great entertainment". The author of the article uses that term too. When Roger said it I always understood it to mean that there was nothing special about the film.... but it was entertaining.

It felt like he was identifying a separate category of films. It's not really great among film, but it gets its own job done "entertainment", and in that way it is plenty good. It leaves room for praise for what it is, but also space for the few really "great" things.

I think Ash Williams (Bruce Campbell) said it best in season 2 of Ash Vs. the Evil Dead, something along the lines of, "critics have shitty taste."

I blame mostly the mindset that a job like that can create. A critic/reviewer is often pressured to agree with pop culture in order to have their opinions validated. Movies and music are a good example of this (cough Star Wars cough). Actually look at most critics vs audience aggregations on Rotten Tomatoes or the fact that very few outfits even write music reviews anymore beyond Pitchfork.

There are very few outliers in this case and are often viewed as malicious, for example Jim Sterling.

I personally never read reviews before I ingest some entertainment I have interest in. At most I read the closing paragraph of a review which is likely to contain no spoilers and sums up the critics opinion. Scores are irrelevant to me as well, I have my own scale.

>"Theodore Sturgeon, who once observed, “It can be argued that 90% of film, literature, consumer goods, etc. is crap.” The “It can be argued” part usually isn’t quoted, and the figure is very ballpark."

Actually it sounds like the author wasn't critical enough. They believed wikipedia. Strange. But I remembered reading about this recently, that Sturgeon used the word "crud", not crap.

The rest is pasted from wikiquote:

"I repeat Sturgeon's Revelation, which was wrung out of me after twenty years of wearying defense of science fiction against attacks of people who used the worst examples of the field for ammunition, and whose conclusion was that ninety percent of it is crud.

The Revelation: Ninety percent of everything is crud.

Corollary 1: The existence of immense quantities of trash in science fiction is admitted and it is regrettable; but it is no more unnatural than the existence of trash anywhere.

Corollary 2: The best science fiction is as good as the best fiction in any field. - Venture Science Fiction (March 1958)

The original expression of this has often been declared to have been "Sure, ninety percent of science fiction is crud. That's because ninety percent of everything is crud." According to Philip Klass Sturgeon made the remark during a talk at New York University around 1951. It has also commonly appeared in variant forms such as "Ninety percent of everything is crap" and is often referred to as "Sturgeon's Law" — though he himself gave that title to another phrase:

Sturgeon's Law originally was "Nothing is always absolutely so." The other thing was known as "Sturgeon's Revelation".

Interesting point the article indirectly hits on:

- "Chummy logrollers—a perception heightened in the social media age. In a 2012 Slate piece called “Against Enthusiasm,” Jacob Silverman wrote, “if you spend time in the literary Twitter- or blogospheres, you’ll be positively besieged by amiability, by a relentless enthusiasm that might have you believing that all new books are wonderful and that every writer is every other writer’s biggest fan.”"

I've always assumed studios had simply managed to, through some means or another, 'corrupt' most reviewers in a process similar that happened to video games. I wonder how much of the problem might be that people are now just too close to the things, and in particular the people behind the things, that they're reviewing? It's far easier to be "critical" (read: truthful) about things and people you have no connection to. But in today's world of social media, reviewers and the creators of the products they're reviewing are going to be in near direct, if not indeed direct, communication. Knowing full well that speaking honestly about a product could hurt people you've formed relationships with is not something easy to do.

For megafilms like "The Last Jedi" (currently at 90% on rotten tomatoes contrasted against 49% for viewers), how many of the reviewers have at least one direct relationship to somebody involved with the project? It doesn't need to be corruption. Plain old reluctance to be honest to those close to us or do them soft favors with no expectation of direct personal benefit -- that basic aspect of human nature would go a long ways towards explaining the deterioration of honesty in reviews. Or taking it to another level, we often wear rose colored glasses towards those close to us. This entire process might be entirely subconscious.

A simpler and obvious explanation for critical complacency; 90% of critics are also crap/crud.

An interesting article, most certainly.

I suppose the immediate response is the critic sphere is heavily self selected for people who really like books / movies / comics / etc. So what to most normal people would seem like flowery platitudes and a cultural aversion to tough talk might actually be a completely genuine love of the medium itself. If you just love to read, it's not vapid pleasantness to call every book fun and a joy - it's just how you genuinely feel.

>I hate to sound like a philistine, but audience-critic discrepancies often occur when a work is less than pleasant to sit through, whether because of The Sorrow and the Pity–like length (a growing problem, pun intended) or grim subject matter. Take last year’s Best Picture winner, Moonlight, which has a 98 critics’ and 79 audience score on Rotten Tomatoes, and which I haven’t seen. The Rotten Tomatoes blurb calls it “The tender, heartbreaking story of a young man’s struggle to find himself, told across three defining chapters in his life as he experiences the ecstasy, pain, and beauty of falling in love, while grappling with his own sexuality.” I can get that at home.

I think this is another element of the critic sphere being very self selected. For whatever reason it selects for literary realist types who honestly care more about depressing stories featuring puppies with broken legs than any kind of sci-fi, fantasy, etc. genre film that's merely fun and not "meaningful". The critics aren't behaving fallaciously or malevolently, they as a group simply have genuinely different preferences for entertainment.

>Here’s the heart of the problem: The set of critics’ and audiences’ interests do not perfectly overlap but rather form a Venn diagram.

Ultimately, this is the real problem whatever is causing the divergence in interests. It makes it hard to use critics, and harder to identify the genuine diamonds in the rough for you personally.

My personal solution is simply look to the average person directly. The internet is full of forums, and many of them discuss whatever medium you happen to care about, and those forums are invariably going to have plenty of threads along the lines of "What is your favorite lost gem?", "What movie do you think flew under the radar?" "What deserved more credit than it got?".

It might be simpler to say that the interaction between the reviewer's motivation to believe that 50% of everything is crap and the fact that 90% of everything is crap has emergent effects.

The major one is that bad qualities that randomly happened to occur more commonly in products that have previously made the 50% cut become misindicators of quality, distorting the evaluation of a constantly drifting subset of products that have or lack those misindicators. This occurs asymetrically (between misindicators of positive quality and misindicators of negative quality) because there are more products misidentified as being good, therefore it's relatively far more common for particular bad qualities to randomly be more associated with works above the median then the opposite. Works that fall below the median in a world of 90% crap would have very few good qualities at all.

This isn't a completely sufficient argument, though. Surely, any person into an artform will at some point be able to recognize the difference between what is good because it is good and what is good because it is different. A good critic would be able to differentiate these two things and give a review with some nuance, but it seems incomplete to say that the critic problem is mostly the fault of seeing way too much junk. Anybody who is actually in to anything has seen a ton of junk, but not everybody has lost their ability to clearly judge things.

Barely related: Rotten Tomatoes summaries feel like they've been paid for.

Sure, if a movie is rotten, they'll trash it. But when a movie is barely fresh, they seem to accord it more praise than the critics gave themselves. I don't have a particular instance in mind so I'd be curious if anyone else has noticed this.

If there's more than 20 points between critic score and viewer score (in the viewers favor) on Rotten tomatoes, then the movie is usually good. There are exceptions to the rule, and recent inclusions in the culture wars (Justice League, Ghostbusters) can break this, but generally it's a good rule of thumb to follow. Reviewer scores come from people forced to watch hundreds of films, and as a consequence they reward poor quality idosyncrocy over good quality but predictable tropes. People crave variety, and when you review movies for a living, you lose touch of what variety is for normal people.

See also: https://en.wikipedia.org/wiki/Groupthink

From the article:

> Groupthink is a psychological phenomenon that occurs within a group of people in which the desire for harmony or conformity in the group results in an irrational or dysfunctional decision-making outcome. Group members try to minimize conflict and reach a consensus decision without critical evaluation of alternative viewpoints by actively suppressing dissenting viewpoints, and by isolating themselves from outside influences.

You have touched on one of the main reasons we decided to launch www.bad-about.com. In a world full of positive hype from reviewers and sales spin from the companies themselves, we thought it would be helpful to collect and display JUST the criticisms to help balance it all out.

Is there a service like this: rate a bunch of media, then filter for critics that largely share your preferences, then subscribe to recommendations from just those critics? If there isn't, there totally should be.