Hacker News Reader

Comment by edw519 | Hacker News Reader

I love it and I hate it.

Why I love it: It's precise. It's elegant. It's rigorous. It's based upon solid, proven science & theory. It's a perfect application for a computer. And most of all, it does what's intended: it works.

Why I hate it: What human can understand it?

I used to implement the first manufacturing and distribution systems that used thinking like this. They figured, "We finally have the horsepower to apply complex logic to everyday problems." Things like safety stock, economic order quantities, reorder points, make/buy decisions, etc.

But the designers of these systems overlooked one critical issue: these systems included humans. And as soon as humans saw that decision making formulas were too complex to understand, they relieved themselves of responsibility for those decisions. "Why didn't we place an order?" "Because the computer decided not to and I have no idea why."

I suppose the optimal solution is somewhere in between: a formula sophisticated enough to solve 95% of the problem but simple enough for any human's reptile brain to "get it". This isn't it.

View on HN

>What human can understand it?

Lets start with Wilson's midpoint, since that's just high school math.

     def mid(upvotes:Int, downvotes:Int) = {
      val total = upvotes+downvotes+0.0
    
      val up = upvotes/total
      val half = 0.5
    
      val a = total/(4+total)
      val b = 4/(4+total)
    
      a * up + b * half
     }

So there are two weights a and b. Using these weights, the midpoint is a weighted average of half and the proportion of upvotes.

It should be very clear that if the total becomes large, a goes to 1 and b to zero. At that point you end up using the proportion of upvotes, just like Amazon.

Now, lets bring in the entire confidence interval.

     def wilson(upvotes:Int, downvotes:Int) = {
      val z = 1.96
      val n = upvotes+downvotes+0.0d
      val phat = upvotes/n

      val lower = (phat + z*z/(2*n) - z * sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n)
      val upper = (phat + z*z/(2*n) + z * sqrt((phat*(1-phat)+z*z/(4*n))/n))/(1+z*z/n)
      (lower,upper)
      }

Using the author's data points, the results look like this:

     val itemVotes = List((600,400),(5500,4500),(2,0), (100,1))
     itemVotes.foreach( x => {
      val s = mid(x._1,x._2)
      val w = wilson(x._1,x._2)
      printf("Up:%5d\tDown:%5d\tMid:%.3f\tW:[%.3f,%.3f]\n", 
      x._1, x._2,s,w._1,w._2)
      })

       scala> Up:  600	Down:  400	Mid:0.600	W:[0.569,0.630]
        Up: 5500	Down: 4500	Mid:0.550	W:[0.540,0.560]
        Up:    2	Down:    0	Mid:0.667	W:[0.342,1.000]
        Up:  100	Down:    1	Mid:0.971	W:[0.946,0.998]

Basically, we prefer the lower bound of the confidence interval instead of the midpoint of that same interval.

I think the original post still stands - try explaining the above to an irate customer.

I've dealt with people who have wanted a retraction of negative comments to their business on the web, I explained it didn't work that way as best I could, but there isn't much you can do. Same with this, if they are irate about a low rating, you play nice, but you don't expect them to learn the ins and outs.

Yeah, most merchants only really want to participate in rating schemes when it puts them above their competitors.

The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it.

Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably correct. However, they're still going to field questions from merchants saying "My product has a 5-star average, but this competitor's product only has a 4.5-star average. Why the hell is my product shown further down in the list?"

Similarly, you may work in the acquisitions department of BigCorp and you need to purchase some equipment. When you prepare your report for Big Boss of your recommendations, how do you explain that you are recommending something which has a 4.5-star average rating instead of the 5-star average rated one?

The ratings may be 100% mathematically sound, but the users of the rating system are not the same as the producers of the rating system. So it's largely irrelevant if the producers made it perfect if the users don't understand what "perfect" means.

the article also misses the human factor. And intent.

for amazon, is it better to show the best product first or to help sell a few of the lacking-rating ones to generate more ratings and work out the uncertainty with real world data instead of crazy math?

I don't think it is a good idea to misuse your customers for that kind of testing.

They'll only have to field those questions if they display the plain averages despite not sorting on them. If we've established that average rating is the wrong metric, why would you even show it to the user/merchant/whoever? Display the same metric that you sort by, and then the merchants will be perfectly happy with higher-rated items showing up higher on the list, and Big Boss will be happy that you've chosen the #1-recommended option.

I really don't think that solves the problem; you look at product X, it has 0 reviews, you give it a 5 star, now it shows 1 rating and 3.7 stars?

So you let people rate it with 5 stars, but then show it as 3.7 magic points. Magic points can be your proprietary ranking system.

You don't need to let them rate with stars and also show it as stars.

If I saw that happen, I would probably assume that the item was so popular that a bunch of other people had rated it concurrently. Unless the number of ratings were displayed simultaneously. There's a bit of a comprehension barrier in letting people know that the ordering score is not the same as the thing they're inputting, and what the relationship between them is. I don't know how to solve that problem (if I had to, I'd probably just not show the ranking numbers at all- the user already sees the sorting order, what do they need the raw numbers for?), but that's OK, 'cause I prefer to sidestep it- just don't use a graded rating scale like "5 stars". At least not unless you have a really good justification and explanation of what precisely the scale means. 5-star rating systems tend to result in large numbers of 5s and large numbers of 1s anyway, so you might as well just go with a binary like/dislike system. The fact that a ranking is a real number and your input is an "up"/"down" enumeration makes it clear that the ranking you see is not the same thing as the data that you put in. It could also be useful from a defusing confusion standpoint to make the ranking more complicated by mixing in an age factor (like HN does) or some other extra band of information. Then if a user doesn't immediately understand the relation between their rating and the final ranking, they can easily write it off as being due to some other factor that's not in their control anyway.

You can do all stars but if you say you need at least X people to rate before showing any. That's what many sites do.

> "My product has a 5-star average, but this competitor's product only has a 4.5-star average. Why the hell is my product shown further down in the list?"

'Because you have fewer votes than your competitor'

I don't think it's that hard to understand.

Right. If you look at most-helpful ratios for product reviews on Amazon, you will see this. 500/600 helpful floats far above 10/10 helpful.

I disagree that people won't be able to understand it - I propose they can't be bothered to understand it. Regular people do not give a shit about algorithms - either the computer/software works or it doesn't. Tech geeks regularly forget this simple fact.

Most people have other things they care about and all they want is to press a button and have an answer come out.

If its instantly clear how the answer is generated, they think about it for a very short while, agree and move on with their lives.

If it's not instantly clear how the answer is generated, they assume the computer is right and move on with their lives.

It's neither precise, nor elegant, and certainly not rigorous. On this approach, an item with 1 upvote and 2 downvotes will be ranked below an item with 1000 upvotes and 2000 downvotes, as one commenter pointed out; and in general, bright new items will almost never be presented to anyone. Search on "Bayesian" in the comments for the precise, elegant, rigorous solution below. I won't claim that it's easy to understand, but it's a lot easier to understand than the frequentist ad-hoc version.

I'm not sure that's a "critical issue". 99% of Google users don't care that PageRank is complicated; they just marvel at how good the results are. Just like most redditors simply talk about how great the comments are, and not the math that makes them so.

Amusingly the proposed solution in this blog post is almost line-for-line exactly the method Reddit uses to rank comments - proof, I think, that the method isn't "too complicated."

When you're working with a system managing orders, where there's a direct human interaction with the algorithm (like the top-level poster is), simplicity is somewhat important. When you're working with a system that's designed to look like magic to the end user, making the system more magical is often better, because it makes it less intuitive to game.

The math doesn't make the comments good. The math makes good comments easier to find, and buries comments with inadequate favor or confidence.

Besides, the math made no effort to write the sentences. ;)

I think you're overlooking a critical component of the Google equation: the supply.

Webmasters * constantly* complain about the impenetrability of the 'algorithm' and how it constantly changes for secret reasons.

So you're right in a way---the demand side doesn't care how you manage to curate supply, but supply is intimately aware of your curation and worries about being unfairly demerited.

I actually find issues with the "best" comment sorting, in some cases. I think it generally is the "best" sort, but in very popular posts it tends to break down.I often find comments with 5x or more upvotes buried down below the top few comments.

I think as comments get higher to the top, people start voting them down more, which might not be the cases for comments which are rising (theory: people reading further down in the comment section are possibly more thoughtful and more likely to upvote, while at the top the ADHD crowd might be more likely to knee jerk downvote?).

In any case, i've recently changed my default sort to "top" and feel it's an improvement.

what?

There's nothing wrong with the math. It just requires better educators to explain it, with analogies and metaphor. Why would you compromise the data/outcome in a hope to simplify the problem?

Further: wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not?

Hint: insurance/loan underwriters regularly end up in situations where the human component of a transaction may look different than the data, and react accordingly...

> It just requires better educators to explain it, with analogies and metaphor.

So... anyone want to take a stab at explaining that equation to those of us who don't really get it?

If a comment has one upvote and zero downvotes, it has a 100% upvote rate, but since there's not very much data, the system will keep it near the bottom. But if it has 10 upvotes and only 1 downvote, the system might have enough confidence to place it above something with 40 upvotes and 20 downvotes -- figuring that by the time it's also gotten 40 upvotes, it's almost certain it will have fewer than 20 downvotes. And the best part is that if it's wrong (which it is 5% of the time), it will quickly get more data, since the comment with less data is near the top -- and when it gets that data, it will quickly correct the comment's position. The bottom line is that this system means good comments will jump quickly to the top and stay there, and bad comments will hover near the bottom.

[1] http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...

That much was fairly clear on the site. I'm talking about the math itself.

With zero knowledge of statistics, here's how to think about it - Wilson wants to construct some score. That score can be as low as 0 and as high as 1. So he wants some interval [x,y]. The center of that interval would obviously be c = (x+y)/2. Wilson first decides what c to pick. So Wilson says, the center c must be decided by the proportion of upvotes. But then he thinks, c must also be close to a half. So he says, okay, lets figure out c using some sort of an average. So he chooses two weights a and b. The weighted average would then be a times half plus b times the upvote proportion. He chooses those weights in such a fashion that if you have lots of data, one of the weights vanish & the other becomes unity. So the weights matter only if you have too few upvotes & downvotes.

Having figured out the midpoint c, Wilson has to actually figure out the lower bound x and upper bound y. Now he draws a distribution centered at c....ok so at this point you would need to know what is a distribution, and why you would need one, whether that distribution has a skew & whether its homoskedastic & so on...which is stats 101, so I won't go there. But if you've gotten this far, you should be able to atleast see the intuition behind Wilson's procedure.

Wilson's 1927 paper is freely available [1]. I can't say I have brushed up on my statistics enough this decade to verify the math but the basics as I read them are as follows:

- You have a normal distribution (bell curve) of data points, in this case quality scores.

- You wish to sort these points based on their respective vote totals. Any given data point has pos positive votes out of n total votes for that item.

- You have a confidence interval, e.g. 95%. This confidence is expressed in terms of the bell curve, so a 95% confidence is within 1.96 standard deviations of the mean [2].

- You have a Ruby function accepting the aforementioned n, pos, and confidence variables and returning a decimal value representing the normalized confidence_interval_lower_bound, that is the quality score that our input data point has a 95% chance of meeting or exceeding.

- Given a set of data points, evaluate the ci_lower_bound for each, and then sort them accordingly. The results will give you a best-guess sorting that accounts for the fact that some data points will have more votes cast for/against them than others.

[1] http://www.med.mcgill.ca/epidemiology/hanley/tmp/Proportion/...

[2] http://en.wikipedia.org/wiki/1.96

Say the 'acutal' rating for an item is p. (ie, if you got an infinite number of people voting, the ratio of upvotes to total votes is p).

Now say your users vote, and they upvote with that probability p. The number of upvotes k you get out of n votes will follow a binomial distribution B(k;n,p). The binomial distribution has mean np and stdev sqrt(n p(1-p)), and is very close to gaussian in shape. Since the stdev is a rough measure of the 'width' of the distribution, common way to describe the error is (mean +/- lambda*stdev), where you can tune lambda to your desire. If you increase lambda you get a wider confidence interval, and therefore more certainty that a measurement will be within that confidence interval.

Now, say you measure k upvotes out of n votes. You can divide by n to get p0, your estimated rating based on those votes.

An easy estimate for the error of this measurement is to assume that p0 is approximately correct and equal to p. Then the expected number of upvotes would be n p0 with stdev(n p0(1-p0)). Divide this by n to get the fraction of upvotes, to give a final estimate for p of p0 +- lambda sqrt(p0(1-p0)/n)

Now, your estimate (p0) of p is not quite right, and therefore your estimate of the error (which depends on p0) is not quite right either. The wilson score attempts to correct for that. We don't know p, but imagine if we did, we would expect any measurement p0 to be in the range p +- lambda sqrt(p(1-p)/n). That is, we expect

        abs(p - p0) < lambda sqrt(p (1-p)/n)

If you solve this equation for p in terms of p0, you get a formula given in the article, ie, the confidence limits for p given p0.

There's nothing wrong with the math.

I never said there was. In fact, I praised it as an elegant solution.

It just requires better educators to explain it, with analogies and metaphor.

You're right. In theory. In practice, no one does this, mainly because they can't afford it. You're implementing technology that costs $200,000 to save $100,000.

Why would you compromise the data/outcome in a hope to simplify the problem?

Actually, simplying the problem actually reduces the compromise when humans are involved. So, no.

...wouldn't you look to hire (and train) the best people who understand the domain they are in, thereby being able to judge whether the outcome of an equation is valid or not?

Only if it made economic sense to do so. I am not going to replace 800 workers earning $12/hour with "the best people who understand the domain" because the computer suddenly has formulas that people don't understand. Experience has shown repeatedly that workers, at any level, simply stop caring when they feel powerless by "solutions" like OP's Wilson formula. That abducation of responsibility almost always far outweighs any incremental benefit that a more sophisticated but incomprehensible formula introduces.

Good points, nice discussion, but please, for the sake of this community, don't reply with words like "what?", "Further:", or "Hint:". You made your point without the snarkiness.

Workers who feel powerless by sophisticated solutions need to learn the solutions, and not be intimidated by them.

Yes, assembly line workers need simple steps to do a job. But that of course isn't who this is referring to. In your example managing inventory isn't an assembly line, checklist job.

You're not replacing 800 workers with the best. In your example, you wouldn't have 800 people managing inventory. You replace your inventory manager with someone who understands the math (if you want to manage your inventory that way)

Sufficiently advanced sophistication is indistinguishable from obfuscation. And obfuscation is a problem that's not ideally solved by hiring smarter people.

I think that's the point edw519 was trying to make.

re the what/further/hint: good point - pre-coffee grumpiness. :) sorry.

I'd imagine the right way to handle this would be Reddit style: Implement the naive solutions (i.e. sorted by average score) alongside the top-performing solutions.

Add in a UX that makes it obvious that there's a choice to be made and allows the user to easily evaluate the results of the naive solution alongside the optimized solution, and end users who are themselves evaluated on overall performance metrics will tend to choose the more effective solution whether they understand it or not.

The addition of metrics to allow for side by side evaluation of the simple strategy versus the optimized strategy is key - someone choosing the unintuitive yet superior strategy will likely have to explain themselves to management at one time or another.

Aan interesting parallel is NFL onfield playcalling decisions. There are plenty of instances where coaches make choices that are statistically unsound but they reinforce the image of a tough, responsible coach. Conversely there are coaches who play by the odds and get lambasted when their decision doesn't pan out.

I thought he was talking about the end user. They may not understand why something with 209 thumbs up and 100 thumbs down ranks better than one with 5 thumbs up (100% positive).

I hate it exclusively. It's clearly better than (1) and (2), but...

I talk about this with some of my med school friends that are interested in/want to create/hate/fear automated diagnosis. At one level, having appropriate statistics to make use of a wealth of prior experience, worldwide prevalence, epidemiological data, &c is basically a requirement for the future of proper healthcare.

It's also an obvious terrible mistake to ever convince people that these statistics are anything besides a decision making tool.

I think the right usage of statistics is to enlighten and confuse simultaneously, not to "answer". They should provide analytical depth to a decision, never an escape route.

For this reason, I think proper statistical application is an intersection between mathematics, computer science, and UI design.

No matter how much statistical data you reveal, you still need to present the results in some particular order. And this formula is the best ordering function I know of for things that are rated.

I think the situation is much more informatively modeled by a partial order though. You can induce a total order on a partial order and you can use this formula to do so conservatively, but maybe the real way to solve the problem is to create an intuitive way for users to appreciate partial orders.

> I think the right usage of statistics is to enlighten and confuse simultaneously, not to "answer". They should provide analytical depth to a decision, never an escape route.

You're right, but I would go further. I would argue that when statistics are presented as "the answer," in many cases they are being abused. A good scientist recognizes the limits of his dataset. Like how a computer program can only do what it is written to do, statistics can only tell you what is in the data, they can't tell you what is _not_.

Operations research and hard systems modeling are neat. The only gripe I have is that often times the models make simplifying assumptions which hold tenuously at best. One example of this is the Gaussian copula used in VAR. In actuality, neither the marginals nor the copula are Gaussian in the large majority of cases; they ARE stable distributions, but the tails can be absolutely HUGE. The Gaussian distribution is attractive because it is well behaved, and in most cases we don't have enough observations to properly understand the full tail structure of the underlying distribution. As a result, you get quants that make a model and call a small subset of points (which are completely valid) "outliers". Then when we observe the tails of the distribution, things fall apart because over-leveraged bankers made decisions based on the notion that the model accurately reflected reality.

You have overlooked a critical point: no one has to mess with this (otherwise quite simple) formula anymore, except the programmer, and even he already got a working piece of Ruby code.

»The computer decided not to and I have no idea why

This just means you need a better notification system, not a dumber model. I am fairly sure Netflix and Pandora use complicated recommendation algorithms - we don't complain because the complexity is abstracted away behind labels like "dark cerebral thrillers" or "songs with rhythmic gyrations like Pretty Lights".

Right, the solution here is that if you're going to sort by Bayesian average or Wilson score confidence interval or something besides what we're familiar with, reframe it to avoid my "something's wrong" detector. Hence the fact that people accept "trending" or "popular" but if I sort by "rating" and the results don't make sense because I expected an average, I don't think "I should probably understand this algorithm". I think "this site is broken and dumb".

And as soon as humans saw that decision making formulas were too complex to understand, they relieved themselves of responsibility for those decisions. "Why didn't we place an order?" "Because the computer decided not to and I have no idea why."

And somehow the solution isn't to have the computer output a detailed description explaining why it arrived at the decision that it did?

Math is hard, let's go shopping!

If you think this is too difficult then maybe you shouldn't try to design scoring/voting systems.

> "Why didn't we place an order?" "Because the computer decided not to and I have no idea why."

This may or may not be a problem. Remember, human intuition evolved to be able to give a fast answer to any question in the face of insufficient data, not an optimal answer to specific questions with sufficient data. In many domains, once you have appropriate data and the means to statistically analyze it, an algorithm using such analysis can reliably outperform human experts - and in some cases, the problem was the political one of how to get the human experts to accept this, take their hands off the wheel and stop second-guessing the algorithm.

In other words, if you are dealing with such a domain, you may be better off to count your blessings, let the computer do its job and save the humans for tasks that can't be handled by a formula.

To get a general idea, you can compare this formula with the simple version, that is * mu - 3 sigma* (be careful when the number of samples is small), where mu is the average and sigma is the standard variation, and 3 is a parameter that depends on the confidence (~93%). When the number of samples (n) is big, then the two formulas give similar results.

The way I think about it is this: comprehensibility is a feature. You have to weigh it against accuracy, effectiveness, etc., but it should be taken into account.

Put this code in a class or a library and let a smarter programmer maintain it.

Meh. "Why did you return the list of reviews in this order?" - "Here's the link to Wikipedia explaining it".

The formulas aren't "too complex to understand", the people are being too lazy to take the time and understand them. No reason to dumb it down and ruin it for the rest of us.

Reply on news.ycombinator.com