Online recipe ratings are broken
blog.recipepuppy.com
blog.recipepuppy.com
YouTube noticed a similar issue. Getting users to rate on a star rating just isn't effective. You may have more luck with a simple 'yes/no' option. Would you make this again? And offer the opportunity for comments.
I find the comments in the reviews on epicurious to be very influential in how I follow the recipe, and if I cook it at all.
I think this applies to almost all online rating systems, in particular ones where it's just pick a number out of five or ten. Most will be either very high (perfect, or perfect with a slight problem), or the lowest (completely shit). Barely anybody will mark something as 2/5 because it was utterly terrible, but this one feature was good. They'll remember the bad parts. No, no data at the moment and it's just a hunch, but one I hope to be able to measure soon.
I'm hoping to apply this to webhosting, rather than simply rate out of ten and leave a 2 sentence review, it'll be much more indepth, with multiple options (support, uptime, features etc). I'm betting that the more effort that you go through to leave a thorough rating will be balanced by the fact you won't have to rate that often (at most maybe once a year once you find a good host).
Oh well, still in the idea stage but I think people should reconsider just a simple rating out of five for anything non-trivial (like rating videos). Anything important (major purchase decision) should be more detailed and have more information for the prospective user.
Videos are a bit different, because people review as they are watching. However with something like hosting, you need to give considerable thought to how you get people to report positive or average experiences equally as often as negative ones.
That's just my initial reaction.
It's definitely something to think about and it's a pretty difficult balance I'll have to achieve but I'm confident I can figure something out.
Users may have better luck finding quality recipes by going to curated sites Cooks Illustrated rather than mob-rule sites like RecipeZaar. As with social sites (Digg, Reddit, HN), the chemistry of the community and the influence of the admins can have a powerful effect on the quality of the content. It also depends on what you're looking for. If all you want is cheesy enchiladas, recipes from the popular sites will work just fine.
There are many sites (not to mention books and periodicals) which have thought and care put into their recipes, often with supplementary material which helps you understand what you're doing and become a better cook, rather than just an ingredient-assembling automaton.
Another aspect to consider is that many people like to read (or watch) things about food but don't actually cook much. That's why the Food Network has become so popular (ever notice how all their commercials are for ready-made products?). A pet peeve of mine is to see 5-star reviews with a comment like, "Looks delicious! Will try making it sometime." Pretty annoying, but some people just like to watch. Maybe that's why recipes with pictures get higher ratings.
If you're looking for something on how to make a dish (like I did recently for baguettes) and find several recipes to choose from, I find a few things make it easy to choose from:
1. Real, individual ingredients are better than composite ingredients (that is, flour, baking soda, oil, etc. rather than Bisquick™). 2. Simpler recipes are usually better than complex recipes, especially if you've never made the dish before. There are exceptions, of course. (I'm certain that the King Arthur baguette recipe that I found would be better than the one that I tried, but it requires making a sourdough starter some fourteen hours in advance. I will try it some day.) 3. Well-written descriptions suggest better recipes. This is a bit like code comments, though—there is such a thing as too much description. 4. Recipes that depend on exact timing of multiple pots and pans are probably not a good idea to experiment with if you're not already good at making similar dishes.
That said, I'd pay attention to recipe ratings if there were only two yes/no questions: I made this; I'd make it again. Report the resulting value as a simple ratio of (repeat)/(made once). Seriously, I don't make recipes more than once if I wasn't enthusiastic about the results or weren't convinced that I could make it better by changing the ingredients or cooking instructions better. I don't know anyone who does. Star ratings are a waste of time.
A good example is dishes containing coriander, a large number of people hate it with a passion while others love it. People who hate it won't make the dish and won't review it. Those who love coriander will make it and most likely enjoy it.
Applying statistical analysis with the goal of getting a bell curve isn't the correct direction to reason in.
It requires more of a slider approach though, which is something you have to spend more time teaching users about.
It really is hard to find a native New Mexican, other than those raised in Mexican households, who actually prefer or even like cilantro compared to the other ingredients we have available.
I guess I'm just furthering your point but some people may prepare those dishes with no Idea that coriander tastes the way it does and just don't have so much hate to decide to review it (or maybe they just think they lack the cooking skills to make a dish).
The number of people that come back and rate a recipe is probably small. And how many people who had a middling experience with the recipe would make the effort to come back and rate it? Or even a good experience, for that matter? A bad one? You bet they're going to rate it.
So right there, you have all kinds of bias problems with the data. And just to make it more fun, how many of those people rating the recipes prepared it properly? How many substituted ingredients that just completely ruined the dish? You see this all the time on these sites - "This recipe is horrible. I vomited for 3 hours straight. BTW I didn't have any red onions so I used pickled cocktail cherries instead."
Somehow you have to find a more indirect way of evaluating a recipe's worth. Maybe it could be a measure of how many times a person comes back to look at it (on the theory that each hit is presumably them coming back to look up that recipe again)? Or perhaps how many people chose to make it.
Additionally, I think the mentioned websites ignore an important fact about recipes: they are not static. Most recipes have variations in technique or ingredients. Many cooks, as you stated, substitute ingredients that subtly, or not so subtly, change a recipe. If a mechanism exists to capture those variations (wiki style, maybe?), those people might have an incentive to come back and share their experience rather than their impressions.
So someone might post a bad recipe and have other members alter it and rate it highly after their own modifications. You can only figure this out by reading the comments.
So some kind of yes / no on whether you have modified the recipe is pretty essential.
When a restaurant serves one customer unsatisfactorily or a product is delivered in poor condition, the rating suffers substantially. This rating no longer represents the quality of the thing itself but also the individual's experience.
The same is true with a cook's experience preparing a recipe. Depending on the reviewer's bias, a recipe rating might describe circumstance and culinary skill better than the quality of the instructions.
As other folks here have mentioned, there's also the issue of average reviewers being not-so-great at representing their true opinions with linear rating systems, but that's a whole other topic.
CookThing seems similar I guess, but not quite the same.
Of course, with any crowd sourced system it is good to take the time to really read through who you are getting your trusted advice from. I still remember reading a review of a Korean restaurant on yelp that had two or three one star reviews because "they did not have sushi".
Their rating system is not broken to the degree of these other websites.
This is a natural desire to conform, and it isn't always conscious. If a user perceives that most people have rated this item X, he will naturally want to rate it X, and will even re-evaluate his judgment based upon the ratings of others.
Of course, for an online-rating system with instant feedback, this gives the earliest ratings the highest weight, immediately skewing all the results which follow.
But what effect does the average rating have on a person who disagrees? They will naturally tend toward the opposite extreme. They want to make their disagreement obvious. If the average rating is high, the negative rater will gravitate toward the lowest ratings, when in fact he might have gone 2 or 3 stars had he had no such prior feedback.
This is a rather intractable problem, since about the only way to prevent this sort of feedback would be to hide the rating until the person has actually tried the item in question and rated it themselves. But that, of course, makes a rating system useless.