What Percent of the Top-Voted Comments in Reddit Threads Were Also First Comment?

I would add that while it's plausible, it's by no means obvious that Reddit is leaving business value on the table. Imagine that they seek to minimize the bounce-rate on any thread such that they maximize overall time on site for each user. In this case, the important part could be that the top comment is 'not wrong' rather than 'right'. If the first thing users see is too factually incorrect, controversial, or emotionally upsetting, people could be driven people away from the site. Essentially this is the same mechanism that helps cause the Facebook news bubble effect -- people leave if you tell them what they don't want to hear, so you write an algorithm that underweights anything that might do so.

To put it into economic terms, if users have high loss aversion (from being upset, etc.) relative to their thirst for knowledge gained from great comments, you would expect exploitation to be overweighted vs exploration. The business value maximizing solution would then be that once an appropriate top comment is found, the thread should tend to stay in that equilibrium.

"I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking. "

The traditional solution to this problem is wilson score intervals.

The real problem there is that most sites that use them only use one part of the bound, instead of using both. Ideally, you use the lower bound to scale how good something is, and the upper bound to determine with what priority you show it order to get more votes on it to determine better bounds to be more confident of your lower bound.

This is what Google Moderator did, and pretty much never had these problems as a result.

(Is it perfect? No. You need a good place/way to show things for voting. But assuming you get that, it works very well in practice).

If you only use the lower bound, you get systems where you only ever become more confident in the scores of the things that show up first (like is happening here).

That said, it's still biased. Years ago, HN admins/mods marked my account as "low-quality content" so now my comments always quickly fall down to the bottom of the parent, no matter how many points they have. They admitted it in an email recently, and refused to take it off my account, despite the quality of my comments no longer being trollish. When you have human intervention like this, there's danger if it heavily biasing the entire forum for the worse.

This search for a better way to rank comments still, in my opinion, makes a large incorrect assumption; namely that it is desirable (and possible) to display comments in order from "best" to "worst". If you think of comments only from the prespective a viewer then yes ordering comments from best to worst makes sense, the viewer wants to get as much information as quickly as possible. But if you are looking at establishing a robust meaningful dicussion, part of that is that all comments should have a chance of getting responded to that is proportional to their value, not their rank. Realistically the top three comments in a large thread are about equally good, but the top comment is going the vast majority of views and replies. One system the facilities dicussion might effectivelly shard it's users into separate threads of managble size so they can have a real dicussion, though I'm fuzzy on the details.

Given all the worry we have about about how our online discussions work I think their is real societal (if not extrable monetary) value in discovering a better solution.

Yes, this non-linear idea seems very interesting. I have read several discussions on various boards with much hand-wringing about the best way of sorting comments - by time, by votes, by user rep, by some sort of weighted average of the above? Suddenly things get rather complicated as people try to decide how the weights should be balanced. But this idea of showing multiple different feeds side-by-side is very refreshing. Two (or even three?) feeds with different weightings and (and relative sizes!) could provide a better balance than what we are used to.

(As an aside, it would be entertaining to take it to an extreme - seeing the top voted comments also shows you the bottom voted, most controversial also shows least, etc. A sort of anti-echo-chamber measure.)

I like the simplicity of this solution, but it might be a good idea to combine it with other relatively simple measures that dampen the "first comment effect," such as randomizing which comments show up at the top, as others have suggested in other threads here.

Can you comment on how comments could benefit from NLP related advances? E.g. detecting entities and using those as entry points into discussion, detecting similar topics discussed in different threads, detecting comments which are low effort and don't add much value. I hate the fact that first/top comment sets the tone for discussion overall and lot of people end up expending energy on that topic alone. People end up discussing something tangential just because the first commenter had "interesting" observation. Also I hate the pun threads which plagues reddit.

The effect is very strong on HN. If you want to be read it's much better to piggyback on it than start a new one.

Randomizing the top three comments.

Have three categories: New, Best (up:down ratio is high), Controversial (up:down ratio of 1:1).

Pick one from each category and then randomly make them the top three. For larger threads you can salt them into the top 10 comments.

New comments that are truly good have a chance of being picked up to Best or Hot, and Controversial comments spur discussion and moderate the self-reinforcing circle-jerk that threads descend into.

I've always wondered why the karma of the upvoter isn't weighed into the equation.

If someone like tptacek (with 259K karma) votes my comment up, I think that's a lot more relevant than someone with 300 karma.

Well, the trend is so smooth that it would be very simple to apply a modifier to comments to counter it. Effectively, votes would slowly expire. In order to remain relevant, a comment would have to be constantly upvoted.

I actually think this would be an interesting experiment to try. Who's hacker friendly and has a well-used comment system that we know?

I don't know if this would change the ranking but I've always wondered if it would be good if threads were ranked by the waited sum of all replies to it. In otherwords the original comment might not have a high vote but if a reply did, even a deep one, the thread might appear at the top. Or does it already work that way?

I am guessing that HN will have a lower percentage than reddit. New comments get a bit of time at the top and have a chance to gain votes if they are worthy.

This is just a "con" of nested threads. A flat, traditional forum/message board would fix this but introduces its own set of negatives.

Some weighted random sorting of top level comments (at each stage in a tree) or sorting by "hot/rising" would go a long way, I think.

If two commenters want to say the same thing, the first arrival will post it, and the second will upvote it (and maybe reply with additional information). So it's not surprising that top comments are weighted towards early arrivals. The question needs to be are there later comments that are objectively better that don't end up sorted higher when sorting by "best".

The "best" sort, by the way, does not use time as a factor, but rather generates a confidence score. Newer comments have less votes and thus less confidence (which ends up giving them a slight boost).

What if you consider it as an optimization problem for the late commenter? You have to choose where to insert your comment, most likely there are multiple options.

A root-level comment when there are already dozens is unlikely to be noticed; there is significant momentum behind the top comments, usually for good reason - whether jokes, clever insight, or a popular viewpoint.

Instead, if you reply near the top in a context where it makes some sense, you skip the root-level graveyard and are almost guaranteed good placement with the downside of "overhead" due to the upwards chain always appearing first.

Consider also that the penalty for being the 2nd, 3rd, or worse comment at root-level gradually increases due to the increasingly nested replies to each highly rated comment. If you assume root-level comments express opinions, minority opinions are also more likely to maintain good position in a reply comment such as the one you posited.

Lastly, "trickle down karma" could be a factor. A quality reply to a root-level comment could enhance the parent comment's value and result in additional karma for them. Such situations could include biases like assuming comments which merited replies are interesting due to peer validation and thus reading them, and more fundamental added value such as uncited sources being given.

Not all subreddits behave that way, and is one of the reason why I looked at 100 different subreddits.

The subreddits not posted as exceptions in the article typically followed the global trend.

The reason is most likely that the top and bottom of the page are easiest to find so the comments at the bottom have a little more visibility than those in the middle.

The HN algorithm changed very recently, which was one of the reasons I had the idea to take a closer look at the Reddit data. HN, within the past few months, will highlight new comments within the first 2 slots even on busy threads.

I would perform the same analysis on HN data to confirm if I could. (HN does not expose comment scores)

That's the problem with threaded views, as opposed to the flat-view style of other forums. Granted with the latter system, the first few comments on a new page will always get more exposure, too.

For posterity, the tl;dr concern is one of the reasons I include the relevant chart at the beginning of the post.