How to solve the problem that the topmost comments get all upvotes
debiki.com
debiki.com
The real meat is here: http://www.evanmiller.org/how-not-to-sort-by-average-rating....
Even if it looks hairy it's fairly simple to implement in e.g. Python (from Reddit's code, rewritten from PyRex): https://gist.github.com/amix/5230165
Maybe pg could try this out for a few days and we could see what the results are!
If you change this to an 80% confidence interval, it'd become narrower and might actually somewhat favor new comments, with only a few votes (upvotes)? So this might be configurable?
> whereas an opinion with 1000 upvotes and 500 downvotes is probably more interesting to read than one that only gathered 50 upvotes within the same interval
Isn't it more likely that the 1000 upvotes and 500 downvotes is a cute kitten photo? Or something similar (a very short but strong and popular comment)? :-) And that the one that gathered 50 upvotes and no downvotes is the truly interesting read?
However! If you're able to estimate how many people actually read the comment that got 50 upvotes — then you'll know if it's truly interesting, but too long to read — only 50 people have read it. Or if it's boring — 1500 people have read it.
Perhaps a good sorting algorithm would be: Lower bound of confidence interval for:
(upvotes - downvotes) / num-people-who-read-the-comment
Only time this would be bad is if there is something important, such as a correction or a key reply by the author or the person they were calling out etc where it makes sense that everyone should read first.
It'd be cool if websites could use eye tracking, then we could easily tell what got read. Maybe in the future.
Cascade Model (one of the original click models): http://www.wsdm2009.org/wsdm2008.org/WSDM2008-papers/p87.pdf
Dynamic bayesian network model (a more generalizable Cascade Model): http://www2009.eprints.org/1/1/p1.pdf
DBN model with scroll and hover interactions (kind of like your example 2): http://jeffhuang.com/Final_CursorModel_SIGIR12.pdf
Anyway I updated the article I wrote with a section "This problem elsewhere, and solved?" that lists your links (and said thank you to you).
May I ask, how come you knew about how search engines function? (For example, do you work with developing search engines or have you studied them at University?)
Indeed!
tags.push("ressources")
In general, I find that comment which links to a PDFs tend to be good. Especially when the linked-to PDF is formatted in two columns ;)
> In the two examples above: When you upvote a comment, the computer thinks that the other comments you have read (the blue ones) but did not upvote, are not terribly interesting.
If you upvote a comment as interesting and the not the ones leading up to it, then you are punishing that comment because you're punishing its predecessors in the threaded chain, so it will be less-seen because you upvoted it.
Any other assumptions of which comments are read will probably not end well. Some people may skip to comments to get someone's personal opinion, other people may only read shorter comments.
~~~
It's worth noting that HN has an interesting algorithm that does not exactly sort by comment score. I have a very high average karma, and when I post on a topic, even if the topic already have 60 comments, my comment begins at the top and stays there even if I'm not upvoted. I suspect that the sorting algorithm is doing something like implicitly adding average karma to the karma of each post when determining order. Therefore it doesn't sort by "best", but by "well regarded".
This is good and bad. It's good because if a person who usually makes high quality comments comes along late to the game, their voice will be heard. It's bad because it creates its own rich-get-richer problem.
I think a better solution to the problem is to simply:
1) Hide upvote numbers like HN is already doing[1]
2) Dither the comments between "new" and "well regarded". First comment is the newest, second is the most well regarded, third comment is the next newest, etc. Perhaps randomize which category comes first.
~~~
[1] Of course hiding upvote numbers has its own major problem because sometimes the hard number of "I agree" upvotes is important. When you want to "Ask HN", say, what particular frameworks people can vouch for, you have no real way of evaluating the comment responses in terms of people that agree. Since they aren't even sorted from best to worst, the information you have is somewhat dim. This could be remedied by giving submitters the ability to turn on comment scores for only their thread. (Or the ability for mods to do it)
Description: http://amix.dk/blog/post/19574
Comment sorting: https://github.com/nex3/arc/blob/b78c7f/lib/news.arc#L2028-L...
It must be more than just a function of time and score, otherwise any of the new 1-point comments would have superseded mine, but they didn't.
Source: experience :)
The topic isn't so much poll position but giving every comment a chance at being noticed so that the better options rise naturally to the top and create a better conversation.
I don't think this is the "I don't care about silly internet points" discussion you think it is.
If you could collapse the top level comments, this problem would evaporate.
1 http://braythwayt.com/2012/03/29/a-womans-story.html 2 https://news.ycombinator.com/item?id=3772292
I think HN would have to increase the total number of comments per page before allowing collapsable chains.
This is also bad because, if true, it creates a strong disincentive to comment on less popular stories because, regardless of the quality of your comments, doing so would lower your average karma.
I think the issues you mention can be addressed. I've outlined how, below:
1: > """If you upvote a comment as interesting and the not the ones leading up to it, then you are punishing that comment because you're punishing its predecessors in the threaded chain, so it will be less-seen because you upvoted it."""
That's a good point. I'm not sure if it is an issue, however. Not upvoting a comment would have very little impact — it wouldn't weight as much as a downvote, for example. So I think the effects wouldn't be that bad. (A comments score could be: `(upvotes - downvotes) / number_of_people_who_read_but_didn't_upvote` and adding +1 to `number_of_people_who_read_but_didn't_upvote` will have only a tiny tiny effect.)
Anyway I think the issue can be addressed like so:
If there is a thread with very interesting comments somewhere in it, prioritize that thread a little bit — so it'll get a score that is a little bit better than the very first comment in it (the comment that starts the thread).
I think SlashDot does something reminiscent of this? — Sometimes, [a mediocre comment that starts a new thread] is collapsed, but [comments deeper inside the thread but with many upvotes] are shown in full.
Alternatively: Modify the algorithm: Don't punish ancestors, only siblings.
2: > """Any other assumptions of which comments are read will probably not end well. Some people may skip to comments to get someone's personal opinion, other people may only read shorter comments."""
I think it's okay that the algorithm makes mistakes sometimes. It only needs to work well on the whole, given data about many visitors. — On mobile phones, however, where only 1 comment is shown at a time, it'd be easier to know what the reader is reading.
Also keep in mind that the examples in the article were intended as examples — if there are issues, the algorithms can perhaps be modified to take them into account. For example, your example visitor that reads only short comments, would be more likely to upvote only short comments. So the algorithm could then "realize" that "oh, this visitor only reads short comments. I'll take that into account" :-)
However, the order is a probability distribution derived from the votes on all comments.
When there are few votes on the comments, the distribution is chaotic, because the algorithm doesn't really know anything about the comparitive quality. After enough comments, the sort order gets more and more deterministic...
I think it would also have to be logarithmic, because being the top comment is MUCH better than being the third comment, while being the 6th comment is only marginally better than being the 8th comment.
Coupled with the stochastic model, it could make it a far more level game.
I also think it's easier to calibrate [an algorithm that estimates which comments people have read], than to calibrate [how much more importance to give to a vote that happens far away from the original post].
In fact, the latter is impossible? Because you have nothing to calibrate against? You don't know what the correct result is.
But here's a reasonable (?) calibration for an algorithm that estimates which comments people have read: If you upvote comment X, then you have probably read its 3 earlier siblings and its parent and grandparent. This doesn't need much calibration, and would probably (I think) work fairly well on the whole? — It's somewhat possible to validate and tweak this calibration, by observing how people actually do, when they read forum & blog comments.
http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...
I seem to recall they also randomly premute comment order a bit as well now.
The one you link considers the upvotes and downvotes as the sample population then constructs the probability that you will upvote it and ranks on that. This allows late posts to take over early posts if they have a better ratio, even if they have significantly less votes, but only if it is "enough" better to make up for its lack of confidence.
This fails to address a few things here:
* It completely ignores people that read the post but didn't vote. There really is no way to get a perfect count of this no matter how much scroll logging, but you could approximate it and then include it in the calculation.
* On HN many people can't downvote.
* As others have pointed HN doesn't necessarily start with a base assumption of all commenters being equal, nor should they.
For example: You would be better off also randomly displaying a comment that requires more voting to establish good confidence bounds on the first page to each user somewhere.
This enables you to get better results quicker over the entire group, and gives you better results than taking into account how many people read but didn't vote.
It fits well with how people act and it fits well with what people complain about in the comments. Hard to measure though. Although if something as rough as scroll logging can still make an improvement I'm sure you could find something rough along these lines that would.
(It's so fun figuring out how to make our filter bubbles more harmful isn't it?)
Would it help if there was an incentive to vote? You could, for example, increase a user's score by a point for each vote given but weight the vote with more points. This way user who write good comments and active readers could benefit.
But then again, this could tempt some users to abuse the voting system. I can't predict which of these two reactions would prevail.
Solution: Disable voting on the top comment(s). This would allow the trailing comments to catch up until they are the top comments and their voting is disabled. The top comment can't race far ahead of the others.
Now I'm oversimplifying, but the effect would be that comment 1 and 2 swapped place over and over again. Or the first N + 1 comments, if you restrict voting on the first N comments. (And people might be confused, perhaps annoyed, when they cannot vote on the first N comments?)
This would not help promoting [a really good but forgotten comment] that is located far away at position 10 or 20.
The submission ranking, I assume, takes more into account than just upvotes (age, number and quality of sub-comments, etc). The top-comment problem feels like the same problem, so might be an opportunity for code consolidation.
To reliably determine if someone has actually read a comment, this means the comment has to be hidden. You cannot assume that simply because something is viewable that the user is viewing it. Since you have to hide a comment, this requires a user interaction event to change its state. This is a broken model since comments are primarily a passive, not interactive, experience.
Search engines have a similar problem: They estimate how useful a search result link is, by counting the number of people that click that link. To do this, they need to take into account that people tend to click the topmost search results only.
But they are handling it just fine, without hiding any search results or something like that. Instead they simply count clicks on the search result link, and apply some mathematics related to the probability that you click that link, when it's so and so far away from the top. And this is similar to the approach I suggested in example 1, which relies on clicks on the vote up/down button (instead of search result link).
(If people don't interact with the page (don't upvote anything at all), I think an algorithm could simply disregard those people.)
There are theoretically sound solutions. Typically, I recommend [1]. Very briefly the solution is to yield an unbiased click estimate (or for what matters here, an unbiased number of upvotes). Look at the position and cascade models as well as the solution in [1]. An approximation of this solution at the end of the article is very easy to implement while theoretically sound.
Side note: another 'solution' would be to randomly generate a slightly different order for every reader (e.g. sampling based on the current number of upvotes for every comment). The readers would then spread their upvotes across more comments overall.
[1] O. Chapelle and Y. Zhang. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th International World Wide Web Conference (WWW), 2009. http://olivier.chapelle.cc/pub/DBN_www2009.pdf
The approximative solution at the end of the article seems really interesting :-) It'd feel better to implement something that is theoretically sound. I suppose the algorithm could need some tweaking, since a discussion is a tree or a graph, but a search result listing is... a list.
Generating a slightly different order for every reader also seems like a good idea, and fairly easy to implement. — So as not to make people confused when they reload the page (by shuffling comments around), perhaps one could use the user ID or IP number as random seed.
(May I ask, how come you know how search engines solve this problem?)
http://www.evanmiller.org/how-not-to-sort-by-average-rating....
My solutions is just throw the idea of "order" out the window and instead use a "sorting hat" process. Sure, by default it may appear in chronological order or "most votes" but based on why you - the reader - came upon that content in the first place, you'll be able to more easily sift for what you're looking for.
Whether you're in the mood for a pun thread, some criticism, or some "deep thoughts", you'll be able to pluck those out from the greater total conversation and then work forward or backwards in the context from those points. Even if a pun thread was the first 100 upvoted comments, you'd be able to discard those with one click and get to the first "serious" response.
Basically, it's because I know what it's like to browse r/science.
I like this initiative! In the video, I like the idea that people be able to discuss only a part of a legislation proposal. Actually I've been experimenting with something related, namely inline comments, http://www.debiki.com/-81101-future-features#inline-comments.
Re: "based on why you - the reader - came upon that content in the first place, you'll be able to more easily sift for what you're looking for" — do you sort comments based on some information you have on the visitor?
Re: "and then work forward or backwards in the context from those points" This somewhat reminds me of considering the discussion being a graph of comments, in which you can navigate freely back and forth? And you can quickly bypass subthreads (e.g. replies to the pun thread?)?
Also, I like what you've done with your inline comments. For my project we don't just use it for "improvements" but for suggestions, general comments, questions, etc...a whole range of taggable purposes for which you'd be highlighting that section in the legislation. So you'd highlight first, name a purpose second, then type your content.
We wouldn't collect any info on the visitor other than what they provide ("I don't want to see _____" would remove posts marked as such).
the number of upvotes is limited.
the karma is actually useful.
the level you browse at is configurable.
the algorithm is simple
thus, top comments don't get all upvotes, AND you actually get a lot more useful comments. I like it. (the article submission and approval process however is far too slow for today's fast paced news systems, this is where HN excels)
That's just something you can assign your own score to; it doesn't affect anything else.
I do find some high ranked posts to be "not so wise" on any forum, but I don't find "obvious trolls" to be highly rated (be it reddit, hn or /.)
the difference that i see with /. however is that I don't get all the high score posts "lost" somewhere because 3 posts have 10292938 points and the others "didn't get voted on recently" (where recently can be the past 5 minutes really)
Also, if you have threaded discussions, a good post in response to another comment may need the other post(s) to provide context, so will simply highlighting that indivdual post make sense without the others?
For example: voting for the top comment only adds 0.1 to the comment_score but voting for the bottom comment adds 2.0 to its comment_score, etc.
I think it'd be hard to assign the 0.1, ..., 2.0 wheights "correctly"? How would one know if the current choice of weights make things better, or perhaps even worse (if you overdo it).
This would also fix the browsing experience on app stores, where apps near the top of an alphabetical or chronological list of search results are seen by a disproportionate number of users.
And if you don't use columns, but show the most recent comment in a box above other comments? Then a variant of the original problem appears? — The most recent comment is visible at the top of the comment section (in the most-recent-comments box) and gets most upvotes / attention. (If you cannot upvote it, people will feel annoyed?)
Also, showing the most recent comments first, rather than genuinely-interesting-comments, somewhat wastes people's time?
Anyway I think it's a good idea (although hard to implement?) and I've thought about something similar a bit too.
It really didn't consider me thinking about a topic, more just if I was reading them all quickly, what I might get to.
One thing that might be an interesting add is where I am clicking my mouse. Especially on high text ratio websites, I click where I am reading to help guide my eyes. Do others do that? That could help the when read algo.
I suppose the-blue-boxes-approach would work only with mobile phones, where only one comment is shown at a time.
(I haven't thought much about it, but I think I tend to keep the mouse pointer just anywhere.)
Often I find myself reading from the bottom up, so I'm more likely to promote overlooked comments. The comments on top don't need my vote, so I seldom give it to them. As long as I'm not the only person doing this, everything should be fine.
If this is a big problem: I've heard people mention it, and I think I've encountered it sometimes. But I guess it is a rather serious problem, because people are so terribly lazy. I mean, they are short of time and have to prioritize.
If people cannot upvote top comments, then the problems mentioned in my reply to this comment should apply: https://news.ycombinator.com/item?id=5430699
Also, forbidding votes on top comments doesn't help a comment at position 10 or 20 to surface to the top. (So really useful comments posted rather late, would still be forgotten forever)
(My 2nd reply) Yes apparently it is, I feel fairly sure now.
Search this page for "Search engines have to solve this problem". — They have the same problem, and "have" to solve it. So I think it does matter fairly much. (Much enough to be worth solving :-))
And read this article from Reddit: http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s... It's about an even worse version of the problem, when one uses a naive (but prevalent!) approach to comment sorting. Anyway it exemplifies how comments posted later on has no chance to reach the top of the page, no matter how useful/interesting they are.
http://camdp.com/blogs/how-sort-comments-intelligently-reddi...
Sort by up-votes, down-votes, newest, oldest, author, poster karma, various auto-sort algos and, of course, random. Pick a few options and giv e the reader control over the post discovery process.
I think that could be very interesting.
Alternatively, gray out the button more based on how a comment ranks (higher = harder to see)
I'd guess people would eventually find the button anyway, even if it's grayed out? I mean, they know where it's located, and notice that they can click on various shades of gray... I'd guess they'd feel somewhat upset about the odd UI, with clickable "disabled" buttons :-)
You could also make the hitbox of the button increasingly smaller - to where the no.2 comment has a 1x1 px box.
tl:dr make it frustrating to upvote top comments -- if they really 'deserve it' people will go out of their way to do it
I suspect you were joking, but I don't think this will work because everyone knows where the upvote button is.
This did however lead me to the idea of hiding the upvote button in more difficult to find places the higher upvoted a comment is. And now I'm laughing about this when taken to a surreal conclusion so thanks! :)
(e.g. "This top comment is brilliant, where's the upvote button? How did HN get it inside my fishtank?!" etc.)
You don't have to assume a person has read all ancestors, just because he votes on a comment. — If you assume s/he has read only the closest 3 ancestors (which I think is more reasonable), the problem you describe is largely gone.
If you do, however, assume all ancestors have been read, I think there would be a tendency that the most popular comments cycled through the topmost positions (up, down, up, down, between position 1, 2, 3 perhaps). But things would not be sorted by time.
Adding average karma is certainly an interesting system as somebody else mentioned.
"The really interesting comments, however, remain forgotten somewhere below, because too few people take the time to scroll down, find them and read them."
The solution proposed:
"This should solve the above-mentioned problem:
"The computer counts how many people have read each comment, and takes this into account, when it sorts all comments."
Ladies and gentlemen, please check my reading comprehension. Do you see what I see here? The person posting says "too few people take the time to scroll down, find them and read them" and then says "The computer counts how many people have read each comment, and takes this into account, when it sorts all comments." How does this give any more prominence to comments that few people are reading (as compared to comments that more people are reading) than any other way of sorting comments? If the problem is that some people aren't reading certain comments, how can how often those comments are read be used to draw more attention to those comments?
Perhaps I am too tired after a weekend day of teaching followed by research to understand what is being proposed here, but I don't think this makes sense.
Anyway, in threads here on Hacker News, there are other ways to find good comments. First of all, there is the bestcomments view of the community,
https://news.ycombinator.com/bestcomments
which, while not a perfect technical solution either, sometimes does promote sub-sub-subcomments to visibility far greater than the visibility of the original greatgrandparent comment in the same thread. Some readers of Hacker News also follow people who post good comments by looking up the links to their comments from their user profiles, for example:
https://news.ycombinator.com/threads?id=patio11
https://news.ycombinator.com/threads?id=raganwald
https://news.ycombinator.com/threads?id=jgrahamc
We can also use HN search to search up comment threads by keyword, and evaluate them for ourselves rather than by how they are placed in a thread. Anyhow, I don't worry about this. Sometimes I think the top comment in a thread is the most interesting and informative, by far, and other times I read far down into a thread to find the comments I like best and need most. Either way, there is plenty of good stuff here. The best way to bring about more good stuff here is to read a lot of the comments thoughtfully, and to upvote all the good stuff you find. Emphasize the positive, and upvote early and often.
The solution is to sort instead by (upvotes/total views). That way, the comment at the top that 1000 people have viewed and 500 people have upvoted falls back behind one that 50 have viewed and that 40 have upvoted, a comment that typically would go unseen by the masses, but which is signficantly more likely to be upvoted (or enjoyed) by any given person.
How quickly late answers gather upvotes seems to be a big factor. I'm also fairly certain the relationship between the upvoter and upvotee plays a big role. For example if you frequently upvote a persons answers, that upvote carries much less weight than if you upvote some random noob. And if you downvote somebody whom you frequently uvpote, that seems to count as a super downvote.
Also, not all answerers are treated equally. Answers from a person with an algorithmically good (or bad) track record start out higher (or lower) than for others. This happens to me on a couple of topics where I've had popular answers. I can make a stupid one line answer and it'll instantly leapfrog answers from randoms with up to ~10 upvotes each, sometimes more.