Content moderation at scale is impossible to do well (2019)
techdirt.com
techdirt.com
Score Voting (= Range Voting) is where candidates are given a score from, say zero to 99, and an average is taken. Effectively the same style as in figure skating. Yes, a once-every-four-years global aesthetic athletic event is the only place to get voting right.*
*what do you mean by "right?" Well, there is something called Bayesian regret and you can minimize it by having Score Voting (=Range Voting) or by having the magical harry potter wizard hat choose the perfect candidate. Other voting systems have more Bayesian Regret and the current First-Past-the-Post voting style has quite a lot, actually. https://rangevoting.org/BayRegsFig.html
Usually only notable popular accounts get coverage over their censorship (or lack of). The number of such accounts is a few orders of magnitude smaller than the general userbase. Any network would want to protect these accounts from arbitrary bans and they probably do.
Most big stories about social media censorship often include comments from company reps who stand by their decision. Stories ending in "Oops we made a mistake" tend to be much less common as far as I can tell.
As such I'd say people are actually pretty good at identifying the cases that aren't just a "bad call" but a contentious matter.
But it is possible to give everyone tools to modereate for themeselves https://adecentralizedworld.com/2020/06/a-trust-and-moderati...
> There is the argument (that I regularly advocate) that pushing out the moderation to the ends of the network (i.e., giving more controls to the end users) is better, but that also has some complications in that it puts the burden on end users, and they have neither the time nor inclination to continually tweak their own settings.
This can be somewhat mitigated with the ability to set a trust filter level to a low level, even in the negatives, to see posts and opinions one may not usually be exposed to. This again is up to the end user if they wish to expose themselves to differing opinions or not.
It is up to individuals to determine what’s best for them. Of course, one would hope they make good choices, but it is not society’s job to save people from themselves. It’s far better to have this freedom and the power to use or misuse it, than to allow companies to build the echo chamber behind closed doors.
That's a nice dogma - which breaks down as soon as one of the cults/echo chambers starts harrassing or doxxing or otherwise hurting someone they declared an enemy.
This whole concept is based on an assumption that if someone bothers you online, this is always a personal issue that can be resolved by you pretending they didn't exist. By now we have enough experience with online communities to know that things don't work that way.
His first two points ("moderation is likely to end up pissing off those who are moderated" and "content moderation is always going to rely on judgment calls") are true of all moderation and certainly HN. They have nothing to do with scale though.
His third point is more interesting: at scale, even 99.9% accuracy is going to leave hundreds of thousands of mistakes a day. No one will care about the 99.9% you got right, only the things you missed. Reporters will call public attention to these.
That's a good point and not widely understood. But he stops there. All that shows is that content moderation is impossible to do perfectly, which we already know because of points 1 and 2. More interesting would be the part he didn't write: given that a large number of misses is inevitable and reporters will call attention to them, what follows next? Probably that the public image will be that moderation sucks. That's rough, but it doesn't necessarily follow that content moderation is impossible to do well.
Consistency is probably a red herring. Moderation will never be perceived as consistent, at any scale. Different mods will always make different calls, and even the same mod will make different-seeming calls because humans are complicated.
If players can trust a ref to be not perfect, but fair (that is, screwing up equally for both teams), the game will flow much more than if it is continually stopped so everyone can argue and check the replay.
I don't follow sports that do instant replays but I'd expect that people don't agree about them anyway.
The best way to signal fairness is to find a thread like a butterfly option where you can scold one of each side. Readers are alert to that and they love it. Everyone feels reassured that things are fair. That would be analogous to a ref giving a penalty to each side. It's not always possible, of course, and when the scolding is one-sided people will automatically assume that you secretly favor the other team.
Any other programming fourms with a different set of mods?
Using a statistical consensus, and correction of outliers, but doing so through the userbase such that as the community grows so too does the base of moderators.
Making it statistical means that no one moderator can have undue influence (other than site owners).
HN does some 2nd order stuff, flags/vouches from trusted users i think are weighted (?) but doesn't go to the next level of then decreasing the weight of votes for those whose votes were meta-moderated as 'wrong' (ie contrary to community consensus).
In theory that should handle brigading, for example, because you don't know what you'll moderate before you say yes, and you can't choose to moderate - the site chooses you.
Slashdot also had several dimensions to it's moderation and allowed users to choose to consume content contrary to the community norm - eg you can choose to see more humour, even if the community in general doesn't like to view humour.
Still the best system, on paper (I have no idea if they have to constantly fix things!?!) that I've seen.
One thing I do like about Slashdot is that they limit the total amount of score a post can get and how much each person can moderate at once. This makes it much much harder to brigade content, which is the usual problem with community based moderation.
Self-moderation is basically what Stackoverflow is and a lot of users are not happy with the outcome. Most of the votes that close questions as "duplicates" and "not constructive" are done by fellow users. I tried to explain this previously[0] but unhappy Stackoverflow users still insist there's an "us vs them".
And consider that Stackoverflow's userbase is a much smaller scale than Facebook.
The community provides a default position (duplicates closed), provides the information ('this is duplicate'), but under a Slashdot style system you can choose that content that others 'hide'.
Definitely a bigger brush than HN.
This is indeed the crux. The .1% failures are quite inevitable. They represent the most valuable training data, and that deserves attention. The breakdown that I see is that it becomes a popularity contest: if you have enough friends, followers, or you can bend the ear of somebody who does, then and only then do these cases receive human attention. In a sense, this is outsourcing the human-level moderation to the public, a level of bad PR determines the escalation. This will fail to capture a large portion of the false positives, fail to feed back into the training set, and the number of failures will continue to scale with the system. The goal should be to eradicate these mistakes, and I can't see how that can be done without investing in human metamoderation -- even if they catch 1% of the .1%, that could provide an opportunity to continuously improve the system. If your profits scale, then so must your investments, or as you say, public confidence will collapse.
How would you convince someone to increase budget for moderation without bad PR?
Sure. With a maximum bandwidth of what, dozens a month?
Nope. It works well on discord.
I can think of few reasons why:
1. The reliance on advertising model pushes them to retain users on their platform for longer and increase engagement in unnatural ways. You can see with how they push you to use the app or throw big subs everywhere.
2. The front-page has limited slots and there is no way to divide the community into smaller parts in itself. On discord, you can do this with channels and roles.
3. There is no voting system and realtime communication is more human like. You can get feedback naturally which you won't on reddit.
Small groups self-selecting rapidly give rise to highly idiosyncratic community behaviours (groupthink, cults, circle-jerks, mutual admiration societies). Large groups tend to vastly overemphasize first impressions and surface appeal, not long-term quality, depth, or complexity. See Dwight MacDonald writing on mass culture in the 1950s. Specialist fields have both legitimate barriers to entry (talent and expertise) and self-imposed ones (in-group, grants funders). Any remunerated content must appeal to the compensation mechanism, dynamics, or authority (this article was linked from Techdirt's announcement it had been banned by Google Adsense, currently on HN). Who pays the piper calls the tune.
Focusing on who moderates misses other factors such as how (mechanisms) and criteria (what is elevated / dropped). Those are also worth considering.
Most moderation tools are very blunt instruments: show or not. I've been quite frustrated with options on nu. By considering other factors -- prominence or obscurity, degree of exposure, how highly-rated though deep comments are presented, say, "virality", placement and frequency of countering or corrective messages, strength or lack of central control, frequency of posting or participation, and contributer reputation, just to name a few, there are numerous other controls which might be adjusted. Randomness is also underapreciated.
Most of all is what the moderation strives for: quality, popularity, engagement, conversions, advertising impact, artistic novelty, legal prohibitions or mandates, political objectives, humour, ... Few discussions of moderation even acknowledge this.
(This submission itself went through HN's 2nd chance queue.)
HN operates largely on human-induced nudges, some firmer than others. It's not elegent, and may not scale much beyond present activity. But it's been remarkably effective and consistent for going on two decades.
Any reason why this was killed twice?
I vouched again.
If so, is the crux of the problem that 1. we don't know what is legal, 2. We can't perfectly allow it, and or 3. Something else.
If 3, is it that websites don't really fear legal repercussions, but fear bad PR, thus alienating users/revenue? If so, doesn't it follow that the level of content moderation will track the projected cost of losing the projected users/revenue?
Not every online community wants to be a 4chan or Voat, which is what that sort of hands off moderation inevitably leads to.
Something like show dead posts on hn.
Although, why host content that isn't profitable?
>
> Something like show dead posts on hn.
Or a killfile in a news client. While usenet was to some extent full of troll posts and spam, with the right filters, the signal to noise ratio was very high.
That works to a degree, but it isn't foolproof because it still requires people to see that content to begin with. It isn't proactive.
That option works poorly against targeted harassment or brigading.
Most pointedly where one jurisdiction's mandated content is another's prohibited. Maps exhibit this frequently, though there are other examples.
I've addressed considerations other than law already in this thread.
So maybe you try to say "if it's legal to a majority of our users then it is legal on the site", but then it becomes illegal to discuss Tienanmen Square on your supposedly global website.
Websites that ignore the law get shut down or shut out. Declaring yourself above the law because you're on the web never works once law enforcement is involved.
No. Many websites would want to be environments with different standards than the minimal legal one. In the same way that you see different types of discussion and language in church than in school than at work than at home. Of course, none of those places have to try to do it at scale... but should anyone try to do it at scale?
Do we need that scale?
Again and again, industries operated by rational agents self-regulate with the goal of preventing the government from getting involved. (With many obvious exceptions, like businesses seeking regulatory capture, producers of commodities with razor thin margins, operators who don't expect to be in business long enough for regulation to take effect, etc.)
Once you get the government involved, you have not only the risk of public backlash whenever you make a mistake, you have the risk of fines or worse. Now your legal council has to get involved in the operation of your moderation system. You potentially need multiple people to sign off on moderation decisions to ensure compliance as well as a system to maintain a paper trail in case government auditors come sniffing around. Your operational costs begin to balloon.
A PR person who says, "If you don't like our behavior, change the law" is basically self-destructive.
let everyone rate content on multiple relevant criteria. Eg, quality, grammar, agreement, sexuality, violence, hate, etc.
Make some default filters that choose societally acceptable settings for each filter, and hides new content that is unrated.
Enable individuals to tweak the filters to any setting they wish, for a totally custom view, including viewing unrated content.
Enable individuals to share their filters with others.
Voila! A system that allows any content to be posted, but shows only "safe" content by default and allows individuals to bypass the filters if they wish.
The closest thing to that model is something like the fediverse where you're choosing to federate with instances based on their moderation rules and your users choose to stay on your instance based on yours. Even that has problems with chains of trust and discovery though. At scale you can't assume good faith anymore
It was super controversial at the time, as I remember (I was new, so I didn't care that much).
Because part of the purpose of voting/karma is operant conditioning - visually marking certain comments as "low quality" and others as "high quality" and to encourage commenters to prefer creating and engaging with the latter over the former through the constant feedback loop of reward and punishment.
> If I’d had to hazard a guess, a solid 50% of downvotes are people blindly ramming the downvote button because that comment already has a negative vote value.
That's the system operating as intended, ruthlessly separating the signal from the noise. I don't particularly like it either but Hacker News isn't about free discussion so much as curated discussion.