Hacking the Hivemind: Predicting Comment Karma on Internet Forums (2014) [pdf] | Hacker News Reader

Hacking the Hivemind: Predicting Comment Karma on Internet Forums (2014) [pdf]

cs229.stanford.edu

70 pointslainon49 comments

I think karma mechanisms have had a net negative effect on online discussion. I don't want to have preconceived notion on how I should feel about a post - I want to make my own decisions. There's no escaping it now though, which is sad.

Given how much content is generated on online forums, it is natural to have a mechanism to surface better/relevant content (different forums define relevance differently). Otherwise you have to wade through a sea of memes and low quality content to get to something that is worth consuming.

>Otherwise you have to wade through a sea of memes and low quality content to get to something that is worth consuming.

Except now you have to deal with two definitions of quality - what you want to read and what gets voted up, and those don't always correlate.

Given how much content is generated on online forums, it is natural to have a mechanism to surface better/relevant content (different forums define relevance differently).

My key objection is that what the majority of users think is "better/relevant" should have no influence on what I think is better and relevant. I can decide for myself, I don't need the 'hivemind'.

What's wrong with just a "deleting memes" policy?

bcherny is saying that karma's primary function is content curation. sure, you might like a few downvoted submissions from time to time, but for most content karma scores will be saving you time sifting through a sea of content. removing content by hand requires an active and vigilant moderation team, user votes help mitigate that burden. Also, the power to 'delete' is most likely a slippery slope...

I see a lot of great content here greyed out. Including some of my own!

At the end of the day, I'm not happy to follow the collective judgment of other users.

With your suggestion, you wouldn't be able to see the content at all. It would have been deleted.

Rarely happens here, in my experience. Significantly downvoted content almost always has severe issues, either in form, tone, or quality. We sometimes might happen to agree with some controversial opinion that has been downvoted a good bit, but more often than not the tone was just not conducive to a good discussion. There's probably a few exceptions I guess, but overall the false positive rate is low enough for me.

The greying out of downvoted comments seems to me to be a separate issue to karma - and particularly egregious given that (thanks to pg) there are no standards at all for voting.

It would be nice if HN would allow sorting of threads by various filters, such as time of last reply, and/or decide whether or not low-sorted comments get greyed out.

But I'm very glad they grey out comments, including some of your own. So what do we do now ?

There's already a mechanism in HN for not seeing comments you don't want to see. You can collapse them.

The logical solution isn't to ignore or remove the karma system, but to use it better.

For example, with sufficient data to analyse, it shouldn't be tremendously difficult to find relationships between users who tend to agree or disagree, and provide each user with a customised view, taking these preferences into account.

(Of course, slashdot took a slant at making it cleverer - by allowing upvoting for different reasons - e.g. funny vs. insightful.)

The logical solution isn't to ignore or remove the karma system, but to use it better.

I much preferred online discussion pre-karma.

True but the incentive system which gives people points for content tends to create an environment where people try and say what is popular or will get high karma instead of what is insightful and avoid what is unpopular or will get downvotes even when it is true and insightful.

Tying a system for surfacing high quality content to individual points seems like a recipe for disaster. I like facebooks system, where points and algorithms work together to try and surface content that users will like, but there is no actual running point system to try and game.

And I just upvoted your vote because I do agree... or it was because your post was top-voted?

I'd love to see an automated karma system that scored posts on correct grammar, spelling and readability. I guess that would discourage non-native speakers of a language from participating though.

My personal threshold for "how good should I be at a language before I start arguing online in it" is very, very low.

I don't think we should be overly PC. If someone doesn't understand the English language well enough to not write all of their thoughts in a single, unpunctuated wall of text, and doesn't rely on the red squiggly lines to indicate that something was misspelt... How exactly will they contribute to the discussion in a way that will be constructive and positive?

I'm not saying that their ideas aren't worthwhile, but simply that the medium won't allow for those ideas to be presented in an engaging manner.

There's a huge difference between the wall-of-text full of typos and misspellings (which seem to become the weapon of choice of native speakers as well), and the misuse of pronouns, or forgetting which word is an irregular preterit. I, for one, would be in favour of seeing the former ranked lower.

Better to have feedback to the writer in the editor, so they are given a chance to learn and improve!

you mean automated censorship system.

>I don't want to have preconceived notion on how I should feel about a post...

Then don't! Or, if you mean you don't want to know how anyone else felt about a post, and you use the Stylish browser extension, you can use this style I made which makes all comment text the same vivid #000000 --

https://userstyles.org/styles/141705/that-s-just-like-your-o...

I never look at the karma (consciously at least). However, I like the filtering effect that it has for the most part, if coupled with good moderation (r/askscience,r/askhistorians)

Well, there's always 4chan.

That get rids of user names as well.

That article reminds me of The Auditors in Terry Pratchett's "The Thief of Time" who try to analyse a painting by reducing it to its component molecules.

There is no mention of tailoring comments and replies to the target audience. Once I adopted the mindset of a senior high school or college student and posted accordingly, I was able to increase my karma on Reddit from 1,000 per year to 1,000 per month.

I still haven't managed to crack Hacker News, though. Tough audience.

> I still haven't managed to crack Hacker News, though. Tough audience.

Just link a bunch of studies to anything you say.[^1] It doesn't actually matter if they're related.[^2] No one really checks the sources anyway.[^3]

[^1]: https://arxiv.org/abs/1502.02876

[^2]: https://arxiv.org/abs/1610.02728

[^3]: https://arxiv.org/abs/1611.03799

This hasn't been true for me.

My top 3 posts [0, 1, 2] are all fairly trite. All three of these were rated much higher than posts with extensive citations.

[0] https://news.ycombinator.com/item?id=13538068

[1] https://news.ycombinator.com/item?id=13441744

[2] https://news.ycombinator.com/item?id=13594765

Edited:

This post already has a higher score than https://news.ycombinator.com/item?id=14180432 which has five citations.

Compare this low-effort comment:

https://news.ycombinator.com/item?id=14124411

to these:

https://news.ycombinator.com/item?id=14149200

https://news.ycombinator.com/item?id=14016438

Getting "in" when a post has few comments and is going to attract a lot of comment readership quickly or stay on the frontpage for a while is probably more important than pure comment quality.

Absolutely true, and especially problematic because by their nature high-quality comments will take more time to formulate and therefore always be late to the party and get less visibility.

Haha no way. My highest rated posts have all been some vapid feel-good thing that I posted shortly after an article was submitted to HN. My detailed explanations that link to studies rarely get more than 10 points (same phenomenon occurs on Quora as well actually...)

I always wonder how much these factors affect a comment's score:

* Proximity to the top of the page

* Time of day posted

* Number of sibling comments and the score of those comments

On reddit, for example, it seems impossible to get noticed once there are already a few highly-upvoted replies to the comment you're replying to.

I've been kicking myself that I didn't take sociology in university to study conversational dynamics on threaded internet forums. It continues to fascinate me

> I still haven't managed to crack Hacker News, though. Tough audience.

I can't tell for sure it's intended, but this sounds Machiavellian. And also following your own stated advice. Compliment the audience on being tough -- tell us what we want to hear -- in order to be rewarded with upvotes. Nice work.

I've written many, many articles/analysis about the karma systems of Reddit/Hacker News.

Karma aggregates are meaningless internet points. Neither Reddit nor Hacker News, nor their users, give major impact to how much user a user has (aside for low karma aggregate thresholds for downvoting on HN), nor is there any correlation between the amount of karma and the quality of content that user outputs.

Sure, in aggregate they're meaningless internet points. As a speaker/writer, though, you can use the score on an individual comment you've made, as a rough approximation of how well it managed to communicate its intent: if a comment you've made scores really highly, it can do well to stare at it and attempt to figure out what you did that made it work for people.

Saying that a 20pt comment is better than a 5pt comment because solely of its intrinsic quality is confirmation bias. There are many factors which help determine the score a comment receives, some of which are noted in the paper (time-of-day comment was made, time comment was made relative to the time original post was submitted, the score of the parent, etc.)

For example, this comment made today, which is literally just a quotation, is at 13pts currently: https://news.ycombinator.com/item?id=14178760

But it was a quotation in response to a top rated comment in a top rated thread.

Oh, sure, a factor of 1-10x is noise.

But if you normally write comments that earn 1-5 points, and then one comment earns 300pts, there might be something to the way you wrote it. Or might not. It might just be that you wrote the rebuttal everyone wanted to the stupid comment in a very popular thread.

But you should try to figure that out, because if—after eliminating sources of error—you're still left with an anomalously-high score, then the cause was probably you.

More than just with highly-positive scores, though, I also tend to analyze the type of writing I put into comments that ended up with negative scores. More often than not, it seems that my own problem is that I didn't sprinkle in enough "hedging words" (like the 'more often than not' in this sentence) and then someone came along with a single counterexample and a downvote storm ensued. I wouldn't have figured out that people had that reaction if the points weren't there.

Hedging words that show awareness of the difference between opinion and fact in particular seem [1] to be pretty key here.

I think[2] analysing which comments get up voted can be very helpful if you're doing so when looking at writing style. but when approached with an eye towards content, it seems[3] like it would lead to the echo chamber effect that often gets raised as an issue.

[1] there goes one of those hedging words now

[2] oops, there's another.

[3] and a third. It would be interesting to see the ratio of filler/hedging/weasel words to votes in HN vs elsewhere.

A point score normalized according to the total number of points in the thread could at least make it possible to compensate for the 'general popularity' variation of different articles.

How about slashdot with 'insightful' / 'informative' / 'interesting' / 'funny', upvotes?

Does a system where the more karma you have the more power your upvotes have work?

Is there a better system?

Just post something about React Fiber.

Does HN even respect karma? My guess is no and is just there as a red herring(not necessarily intentionally) that keeps people from cracking the real metrics it's using.

What do you mean by respect karma? Obviously the ordering of sibling comments is determined in some way by upvotes and downvotes.

Just ask everyone who offers an anecdote for citations, and tell everyone which named bias their point of view is.

> Once I adopted the mindset of a senior high school or college student and posted accordingly, I was able to increase my karma on Reddit from 1,000 per year to 1,000 per month.

Why, though?

"one could optimize karma by posting a long, pessimistic reply to a brand-new article."

I knew it! Everybody on the internet is an asshole!

This seems like a really fun project to try and reproduce!

I wrote a blog post along similar lines around when the orignal paper was published: http://minimaxir.com/2014/10/hn-comments-about-comments/

Unfortunately, since comment scores are still omitted from the API, it would be hard to modernize this analysis. (and the algorithms have definitely changed over the years)

Needs a [2014] in the title.

Reply on news.ycombinator.com