Why Wine Ratings Are Badly Flawed | Hacker News Reader

Why Wine Ratings Are Badly Flawed

online.wsj.com

32 pointstshtf43 comments

A good friend teased her husband for spending only $7 for a bottle of wine in a facebook status post. So I bet her I would eat a cork if she could consistenly tell the difference. I won.

We held a blind taste test for 12 people with 14 wines ranging from 3.99 to $50. For each variety, there was one wine that was under $7 and one that was $11-50. We bought most, but not all, of the wine at Trader Joes. Participants had to guess if each wine was 'cheap' or 'expensive.' No one got more than 8 right. Aside from guessing the price category, people rated how much they liked the wine from 1-5 so they could see what they liked after the reveal. A 50 dollar Cabernet was one of the lowest rated.

This 3.99 bottle had the 2nd highest rating:

http://cheapwineratings.com/2008/04/10/villa-cerrina-montepu...

Highest rated was a 6.99 Pinot Noir whose name escapes me at the moment. I don't claim the results necessarily prove anything about the wider world, except that my friends have been overspending on wine.

I am convinced that many wineries should sell the same wine for different prices. They could sell the same wine for $7/bottle, $30/bottle and $125/bottle. Of course the $125 should have a different label and should be marketed a lot differently but the wine itself doesn't have to change.

I personally just buy a big jug of dark red cooking wine. I can't tell the difference between it and the other more expensive wine, and I refuse to spend more money for snoberry (note: I have nothing against others paying that much if it makes them feel good/better).

Well, there's a big difference between the issue of wine rating silliness and the wide variety of flavor you'll find in wine. A $10 bottle of gabbiano tastes almost nothing like a $10 bottle of monticello rioja and neither will taste anything like your jug of wine. Saying they taste the same is like to saying orange juice tastes the same as lemon juice, and that both taste the same as grapefruit juice.

I was saying that I couldn't taste the difference and according to the article, many other people can't either. They are essentially tasting the label and the marketing, which is why the marketing and the label should look different in both cases.

They are people more sophisticated and better at wine tasting, but there are a lot of people who just follow irrational marketing cues.

"I was saying that I couldn't taste the difference and according to the article, many other people can't either."

This isn't what the article is talking about. The article is arguing that subjective, linear wine ranking is faulty, that preferences are not objective or static, that it's difficult to distinguish between similar wines and that identifying individual flavors and scents (eg, "lavender, roasted herbs") are tied to the taster's imagination.

If you can tell the difference between grapefruit juice and orange juice, you can tell the difference between a gabbiano, a monticello rioja and a generic merlot. They taste dramatically different and you can't give a merlot to someone who knows what chianti tastes like and have them believe it's a chianti. Different varietals have very distinct flavors.

And when it comes to quality, there is a much, much more significant difference between a poorly stored, very low-quality wine and a mainstream ~$10/bottle wine than there is between a $20/bottle wine and a $100/bottle wine. After all, a number of the pricer wines would be simply $20/bottle if it weren't for the ratings systems critiqued by the article.

You don't taste any oxidation after a few days?

I'd suggest going with boxed wines if that's your preference. Target makes a really good one, and it stays good longer.

Yes the boxed wine tastes better. I also keep it in the refrigerator and take some out some time before drinking it and let it sit. I think that's called "decantering" but I din't know when I started doing it, it just made the wine taste better.

That might be good fun at Thanksgiving or a holiday party. I think it would be important to restrict by type, though (at least for the whites, not so knowledgeable about the reds).

Also, a comment for the OP: If you get the print version of the WSJ, this article appeared just two pages before the regular wine column by the husband/wife team. Sure enough, lots of references to "taste of cranberries", "fruity" etc.

I think a better comparison is to have a wide range of prices for a single varietal. For example, a white is going to taste different than a red. Does one of them taste better?

High-end reds cost more than high-end whites. That could be because they do taste better, are more costly to produce or are more rare.

That said, thumb through a Wine Spectator and you'll see a bunch of high-priced wines that don't rate as well as their price suggests. I usually have good luck with buying bottles where the wine steward has added a special "yummy" sticker or writeup.

Maybe another fun game would be to bring in a favorite art object and have everyone guess what it cost. Just because you or I wouldn't pay $100 for a Kinkade doesn't mean that a lot of people will.

Perhaps this is the sort of thing that happens when we make judgments in areas where we lack expertise.

If I'm not mistaken, part of it is that reds age better (due to the tannins).

You missed an independent variable in this test. What were the relative levels of tasting expertise among the tasters? I think you can only conclude that wine price doesn't matter much for the lowest common denominator there.

I think my last sentence agreed with your point.

For anyone interested, only one person, who was originally from Sonoma Valley, claimed to be a connoisseur of any kind. He didn't do much better than average--I scored higher and make no claim to any expertise.

A similar contest for 5 brands of Tequila (plus a shot of vodka, for the distracted) was done at the Telluride Neuromorphic Engineering Workshop. The outcome was also surprising: second-cheapest Tequila was rated highest by the audience.

Mr. Hodgson said he wrote up his findings each year and asked the board for permission to publish the results; each year, they said no. Finally, the board relented -- according to Mr. Hodgson, on a close vote -- and the study appeared in January in the Journal of Wine Economics.

I love it. He knew from his findings that the wine review board's decision would vary from year to year.

Thank you. It so rare that a statistics joke makes me actually laugh.

When I lived in Cincinnati, the "Everybody's Cooking" radio talk show was like my "Car Talk" for foodies. One of their soapbox things was wine ratings. They thought it was way too complicated. "C'mon, are you really going to taste the difference between an 88 and an 89!?"

Those two chefs proposed the "Siskel and Ebert" system. Thumbs up/thumbs down. "Do you like it? Do you not like it? It's only fermented grape juice!" They advocated ditching the mysticism and just enjoying it with dinner.

The style / grape type is probably more important than some number. "I like really crisp whites", "I like really thick, fruity (but not too tart) reds", etc. will narrow down a search better than just looking for high ratings.

If you really like steak, you're probably better off with a cabernet sauvignon than a riesling, rating nonwithstanding.

Probably the biggest problem is the packaging. With produce, you can go to the store, and feel and sniff the wares. If you don't like it, you go to a different store. With wine, you have to wait until you get home with the bottle. And even if you find a good wine, you have no idea if it will be the same year to year, or even a few months later. You have no idea when marketdroids are going to convince the executives to cash in on the name. (Same happens to beer.)

Whole Foods does right by its customers by handing out samples of chosen wines. I like being able to taste before I buy!

Well, with beer, it's much cheaper to get a variety of singles. After my wife and I did that a couple times, we had a very clear idea what styles we like. (And that we prefer beer to wine.)

Incidentally, West Michigan is one of the best places in the US for craft ales. There's Founders (http://www.foundersbrewing.com/founders/), Bell's (http://www.bellsbeer.com/), and the Hopcat (http://hopcatgr.com/main/About.aspx), for starters. :)

The Pacific Northwest was marvelous for great, innovative microbrew when I lived there in the early 90's.

Cool, those guys are still around! http://maritimebrewery.ypguides.net/

I agree with the excellence of Founders. Many truly unique beers. I'm waiting for their wares to become available in Canada. Until then, I always search them out when visiting within their distribution area. Last found a some in St. Louis.

Something I was reading lately* noted that those numeric ratings are usually reached tasting wine alone, rather than with food. That can bias ratings toward wines that stand out after tasting several similar cabs (or whatever) in a row, including wines that utterly lack subtlety. "This tastes like getting whacked with a giant oak plank."

* One of the Culinary Institute of America texts, I forget which.

I find that the taste of the second glass of a red is often more rounded and enjoyably palatable than the first glass. I don't think this is inebriation but something possibly to do with how the taste buds themselves work. Temperature will alter the flavour a lot, as will the food being consumed alongside (as you noted).

Also don't underestimate the influence of oxygen.

This is one of my pet peeves. I can't figure out any way to find good wine outside of personally sampling it. Ratings don't work, and price only has a rough correlation to quality. My current system is to buy a random sampling of low-mid priced wines and when I find a good one, buy a case.

Reminds me of the experiment run by a german tv program where they let some people compare a real, but mediocre wine with a liquid composed from single substances to match the gas chromatograph readings of a good wine. The artificial wine won. http://www.wdr.de/tv/quarks/sendungsbeitraege/2009/1110/002_...

Fascinating. I don't know much about wine, but I've noticed that my own appreciation of the same type of wine will vary greatly. Sometimes it seems highly dependent on what I was doing earlier in the day and what I've recently eaten. If I was cleaning, for example, perhaps some trace residue of ammonia on my hands might interfere with the taste/smell. Similarly, if I had spicy food, an otherwise enjoyable red might taste strange to me. And of course, there's a fair bit of variability between bottles for various reasons, making the whole process even more uncertain.

The strategy I've always employed is to pay no more than $6-9 unless it's a special varietal. Some of the best wines I've ever had were South American bought as one-offs for $10-15 though much of the enjoyment came from the novelty. Much better than honing your pallet to the taste of expensive wine is to learn to pair wines with food. Learn what varietals you like and then play with foods spicy, sweet, and sour. The most enjoyment from wine comes from learning how the tastes combine and change wildly.

Pairing wine is a game of combinatorics. The tricks tend to be to maximize the number of other flavors you can try with your wine. In this way, you can transform a single bottle into 3 or more different flavors. Some tricks:

Bread, cheese, and wine is a common thing but don't go overboard on the bread. Much of the time the heavy starch is going to wipe out the flavors. Invest in more variety in the wines and cheeses.

Spice comes in many, many varieties (from peppercorn to bright red ground cayenne, to Chinese sichuan pepper) and each one often has a different reaction with wines.

Steak and wine is a great combination because well-prepared meats have tons of interesting flavors thanks to the "maillard reaction". Sausages are a particularly good choice.

The Economist recently had a article about the scientific basis of the "only white wine with seafood rule": http://www.economist.com/sciencetechnology/displaystory.cfm?.... Apparently, the key factor is the level of iron in the wine -- too much iron in the wine/seafood combination results in a "fishy" taste. It would be interesting to see more research into the chemistry of wine/food pairings.

From a chemistry standpoint, the biggest factor is the acidity of the wine. Acidic wines (particularly whites) can be very crisp and palate-cleansing, while less acidic wines taste softer and smoother.

I don't have my copy of Harold McGee's _On Food and Cooking_ with me, but that's the first place I'd look for more info.

I'm not so sure that applying statistics to wine tasting isn't like applying physics to haircuts: yes there's probably no relationship between the money you paid and the way you feel about your haircut, but if you thought there was you probably were missing the point.

Wine is an immensely subjective and varied experience. That's one of the reasons people like it so much. There's one feeling you get when you're dining out and a friend turns you on to a 10-dollar red that tastes great and another feeling you get when you go to the cellar for a special occasion to get a hundred-dollar bottle -- even if the wines are statistically indistinguishable.

What can I say? It's a great product. All of human experience goes into the process: peer pressure, prior experience, atmosphere, cost, anticipated pleasure, etc.

When I started trying to learn how to taste wine I went through Wine Spectator and only ordered wines of the very highest rating. I understand this is just an anecdote, but I found that about 1 in 3 were actually distinguishable by me from grocery-store wine. I took that to mean that yes, there are differences, but the amount of signal is vastly overstated.

But that misses the point: it's emotional, not statistical.

it's emotional, not statistical

You just said a mouthful :-)

When I started making my own wine a friend sent me a review of a Barolo -- the variety I was making -- and I mentioned that the wine review seemed awfully similar to high end audio equipment reviews (e.g., $15k CD transports + another $10k for the DAC and preamp!) and sent him a few links.

He was absolutely astounded (and amused): both kinds of reviews used language so similar that in some cases they were almost interchangeable.

What's my point? Both products can be very expensive with little objective differentiation leaving the buyer with a great deal of incentive to defend his purchase vigorously to anyone who asks the question, "why the hell did you spend so much money on that"

lol. Your are correct. The less actual substance and more emotion there is, the more tenaciously people will defend their positions. Again it's a feature, not a bug: one of the reasons I decided to look into wine collecting was observing a couple nerd friends of mine argue (discuss) for 30 minutes what kind of wine to order at a fancy restaurant.

Both products can be very expensive with little objective differentiation leaving the buyer with a great deal of incentive to defend his purchase vigorously to anyone who asks the question, "why the hell did you spend so much money on that"

To which the appropriate reply is: "I didn't spend my money on that, I spent my money on me (or you, or our dinner, or your graduation, etc)

It's like weighing the steak at a fancy restaurant and comparing it to the price-per-pound down at Costco

It reminds me of my step-father's view on haircuts: for a certain amount of money, you deserve to get at least so much hair removed. :)

Having said that, I've got a bottle of 98 Donnoff Riesling that has my name on it for Christmas Eve. And I've been saving a 1996 Opus One for gosh, I don't know, perhaps the end of the world or my first million :) And I'm thinking that I've sat on those 4 bottles of 97 Dow's Port long enough.

To my mind, I'll take these wines and statistically match them up anywhere. But who cares if I'm wrong and it's just me? I've enjoyed collecting them, and I'll enjoy the special feeling of consuming them when I think the time is right.

But damn it, I'm right. (grin)

Vodka is an even worse when it comes to peoples perception of quality. The higher the price, the better it must be! Twice I've had bartenders will up shot glasses with well vodka and Grey Goose, then number the napkins - my friends have got it wrong both times.

I've been to Mt. Hood distillery, they import 180 proof ethyl alcohol by train from the midwest where its distilled from grain, then the use it to make many different popular brands of vodka, gin, liqueurs, and others. The Vodka bought in America is Ethyl alcohol cut with water!

Similarly: Judgment of Paris: California vs. France and the Historic 1976 Paris Tasting That Revolutionized Wine

http://www.amazon.com/gp/product/0743247515/ref=cm_rdp_produ...

the idea of wine ratings seems weird to me.

Everyone has different tastes, different taste buds, how come we don't have any rating systems for candy? or steak? or eggs? or icecream?

This Snickers bar, really brings out the taste of Artificial Red #503

Is it really any weirder than a rating on films or books or music? That is also largely about taste. And there are plenty of competitions and ratings on most kinds of foodstuff (including candy and ice cream), so wine is in no way unique in that respect.

In books and films you have a little more information. An even then the only realistic scale is one or two thumbs up.

Suppose the only thing you knew about a film was it's director (the grape) the studio (vineyard) and a score out of 100

I'd also know who gave it the score. If a director that I love makes a film that a critic that I respect gives a high score I'd be happy to see that film knowing nothing else.

To paraphrase and invert Tolstoy: Rotgut wines are all alike; Every really good wine is good in its own unique way.

Different kinds of wines pair well with different kinds of food. Thus the intricate details of notes, nose, et cetera. It's usually only something the sommelier of a nice restaurant need worry about - pairing with the entrees well. Most of us just should just wonder if it tastes good or not :)

I don;t know how to rate wine at all. The way I rate wine is by the burn. If there is no burn and goes down smooth it is good wine.

The burn comes from the alcohol. Smoothness could be due to lack of tannins in the red wine. (Tannins cause the feeling of astringency in your mouth and inner cheeks.)

Reply on news.ycombinator.com