Ask HN: How much transparency for an online reputation system? | Hacker News Reader

Ask HN: How much transparency for an online reputation system?

14 pointssachinag8 comments

In light of the new orange usernames, it made me think about something that's been bothering me for a while. Please bear with me - this makes more sense as a blog post, but I'm not going to get the comments there as I will here. :)

At Dawdle, we have a substantively different approach to how we evaluate sellers versus eBay and Amazon. At first, we didn't want to rate sellers at all, given that we have a 100% money back guarantee like StubHub. However, it became clear that users just weren't comfortable with no seller ratings given the type of merchandise on Dawdle - new and used video games, systems, and accessories.

So when we decided that we needed to have a seller rating, we tried to figure out the best approach. We thought that eBay/Amazon-style ratings aren't the greatest because it's all about accumulation of points (+/- 1, similar to Karma here on HN), where each additional point has diminishing value. So we decided just to start with buyer feedback, but layer a ton of other things on top of that. Thing is, we don't say what all those layers are: http://www.dawdle.com/help/index.php/sell-feedback/

We wanted to make sure that people weren't setting up shill accounts and inflating their ratings, and we like how Google gives guidance but not the exact algorithm for their rankings. The general idea is to encourage good behavior and to allow us some leeway to fix things as we see issues.

So what do you think? Should we tell people how we calculate it so that everyone knows it's on the up and up and we're not secretly favoring people? Or can we get people to try us out knowing that they're on a much more level playing field as a new user versus older users? (We're going to make it even easier to pick what seller you're buying from - you couldn't in our beta, just like StubHub doesn't tell you who the ultimate seller is - and these sorts of things are very important to both buyers and sellers.)

These Clay Shirky articles might interest you. In fact one of them might have been linked to from HN recently, I'm not totally certain:

http://is.gd/iv84

http://is.gd/NZu

Awesome, reading now. There's a lot of stuff about implicit rankings that we've thought about (see my standard presentation on this: http://www.slideshare.net/dawdledotcom/actually-useful-trust...), but having written words helps a lot.

I think there's a fine line to walk: tell enough about the principles that you use for contributors to earn reputations, but not with too precise a detail. Not necessarily to keep people from trying to "work the system" - but because you may need to tweak how things are weighted without always having to make a formal commitment to weights, etc.

Besides: telling people which factors will affect reputation ratings is a powerful statement about the principles and values you want your site to support. That make sense?

Thanks; do we do enough to tell people what principles and values we support?

I'm not sure that I can answer that (not familiar enough with the gaming population). So I certainly wasn't trying to suggest that you are expressing it enough. For that matter, I think that sometimes such expressions get laid on too heavily/preachy. That's part of why I think it's powerful (for sits on whose opinions I depend) to "show not tell" And the criteria a site uses for karma/reputation is a strong way of "showing."

> so that everyone knows it's on the up and up and we're not secretly favoring people?

Trust has to start somewhere. The people that think the above will in the most part either a) not be using your site or b) not convinced by your "open algorithms" anyway.

On the other hand opening them up fully makes it easier to game the system: unless it is a bulletproof system I would avoid full disclosure.

reminds me how i prefer rating schemes over point accumulation ones. especially if the rating average is based on latest actions more than older ones. point accumulation with exponential experience levels (think rpg) is better than linear point accumulation.

i vote for you to keep the exact algorithm secret. make it work. that's more important than transparency. your real goal is trustworthiness, which i think you can accomplish in other ways. (i will never get used to voting on the quality of a post rather than in agreement/disagreement, which is what voting is historically intended for)

Everyone should know the rating system. People tend to ignore rating systems that are hidden or not easy to understand. An example of a rating system that suffers from many hidden features is the "credit score" system; of course it is not designed to be useful to consumers, it is designed to be useful to the companies that know it's hidden factors. An example of a scoring system that is defined, but generally hated for being opaque and complecated, is the BCS college bowl system.

You should make it so it is not easy to game; the way to go about this, is to make it so that the easiest way to "game" the system, is to do something that you want to encourage anyway. So instead of making a fake account and making fake transactions back and forth to up your score, give sellers a bonus for allowing new, unrated buyers to pay after receiving the item, for example.

Think about how the standardized tests for private pilot licenses and etc work. They publish thousands of example questions with the correct answers, and then for your test they select a few at random. If you "game" the system by studying all the questions, you achieve their goal anyway, which is to learn the material as well as they can reasonably measure.

Now, you could take out the reputation and knowing the seller part of it, but I think then you would have to step in and guarantee things for the buyer.

I would do an ebay-like accumulative rating system, but have points be weighted with age, so that a good rating from a week ago is worth more than one from a year ago. You might fade them linearly to zero, but I think your formula should give people who have had an account for a long time some weight just for that.

I am experimenting with a scoring / rating system now. it is for my site slackerfactor.com, which is mostly functional. The site allows you to keep track of who paid which costs among roommates or a similar group; if someone is a slacker, and doesn't pay his fair share for a while, his "slackerfactor" score shows it, and the people who are doing his share similarly have a better score.

I decided to make the score fade with age, linearly to 3 years, after three years it doesn't matter any more. Since recent debts / credits matter more, someone who is up for re-evaluation as a housemate at the end of a lease or whatever, and has been slacking all year, has an incentive to clean up their act and pay a bunch of bills or buy groceries or whatever. The main site doesn't display the slackerfactor yet, only the beta site, because the formula probably still needs to be tweaked according to how people will use it.

Another approach you might want to put in your bag of tricks is per-user modifications of the score. On Slashdot you can say you want to ignore "offtopic" moderations when deciding what comments to show, or to rank "troll" as a plus instead of a negative. Similarly, you might offer users the ability to have some checkboxes such as "ignore rankings more than a year old" or "negative rankings count as double if the person making the negative ranking has a score above 50" and so on. Giving people the ability to tweak knobs often quiets the most persistent complainers.

Reply on news.ycombinator.com