Is content moderation a dead end?
ben-evans.com
ben-evans.com
There's no objective definition of right or wrong in content moderation. Right and wrong is subjective, especially across cultures, and moderation should be subjective too.
I believe end users should have the choice to adopt blocklists, Adblock style. Those lists could contain single posts, accounts, or even specific words. A lot of content (like flashing images or spoilers) does not merit deleting, but there are users with good reasons not to see it. They should be given such an option.
There should be a few universal, built-in blocklists for obvious spam, phishing, child porn etc, but all the rest should be moderated subjectively.
A Clubhouse-stule invite system (with unlimited invites) would also be a good idea. It would make it much harder for spammers, cammers and troll farms to make new social media accounts.
Putting the moderation burden on people is also not a solution, it's duct tape.
I don't think GP is saying we should put the moderation burden on people. When you accept that there is no objective definition of right or wrong, you begin to see that perhaps there are ways for people to self-organize on the internet according to their values rather than being shoved into the same box like we often do today. Many people are looking for ways to efficiently and effectively organize on the internet in a more sustainable manner.
If you leave it up to each individual to decide what is appropriate or inappropriate, and provide them with the tools to block content they consider inappropriate, that's a burden on them, because you're not taking care of it at the community level.
And if the community's strength comes from shared values, and you leave that up to each individual to decide, what's shared, and what sort of "community" is actually offered?
That's exactly the point GP was trying to make. That people should be able to organize in groups of shared values and context. Rather than there being a rather large rough mono-culture of moderation policies.
You are describing "well managed communities". HN is arguably one of those. Many topic-specific forums, IRC channels, mailing lists, and communities on platformy things that seek to reinvent those are as well. They tend to be centered around a topic or purpose and have rules, guidelines, and social norms that facilitate that purpose.
I think the toplevel comment is talking about global many-to-many networks where people connect based on both pre-existing social relationships and shared interests (often with strangers). Those require a different model, and centralized moderation based on a single set of rules is probably not the best one.
ie Twitter is huge, but there are sub-communities that exist within it (some well managed, others not so much), and Twitter is building community features to that point.
Twitter can certainly do a better job of moderating its sub-communities, taking into account shared values and context, but I still don't see how the solution is to have users deal with content moderation.
The assumptions behind having individuals decide what's appropriate or not are:
1. The majority of the community would make the same choices
2. Those posting what most feel is inappropriate content would not get any responses because no one would see it
3. Because of the lack of response, those posting inappropriate material would move on elsewhere and no longer post.
And if I'm viewing a conversation/thread/topic with something blocked in it, where I have blocked it but other members of the community have not, what does that experience look like to me?
And if a spammer posts a dozen links to their website, but I haven't blocked that content yet in my content filtering setup, that'll be fun.
This just sounds something like Parler. You can post anything you want there, nothing is blocked, and you're free to set up browser-based filters to hide the content you don't wish to see. In effect, you end up with a community biased towards those who are comfortable with vitriol, and as such it's filled with terrible content.
Typically, that involved using an existing application to connect to the service and read posts. As you spend more time reading through current and past threads, you would get an idea what you want to filter and what you want to see. In other words, you would set up your filters over time as opposed to doing everything at the start.
> And if I'm viewing a conversation/thread/topic with something blocked in it, where I have blocked it but other members of the community have not, what does that experience look like to me?
If no one responds to the post, you simply wouldn't see it. Otherwise, you would see responses to a post/comment that you couldn't see. The software could also allow you to hide any responses to that post if you had no interest in the subthread(s).
> And if a spammer posts a dozen links to their website, but I haven't blocked that content yet in my content filtering setup, that'll be fun.
I guess that depends if your client is configured to display linked content inline rather than just the urls. I don't use clients that display content inline and then just add a filter for the spam post.
> This just sounds something like Parler. You can post anything you want there, nothing is blocked, and you're free to set up browser-based filters to hide the content you don't wish to see. In effect, you end up with a community biased towards those who are comfortable with vitriol, and as such it's filled with terrible content.
It's how usenet worked and there were plenty of vibrant communities spread across many groups that lasted decades, if not longer.
not necessarily. Taking out the more heated factors for a moment, think of a concept as simple as spoilers. spoiling others are widely considered to be an impolite move, but a community formed around talking about a work wouldn't want to have to blur every image or paragraph in order to have a conversation on what may be years old points. There's no objectively perfect solution, even if there may be an almost objective factor to address.
In instances like this, leaving options is valuable. Some may want to keep all screenshots unblurred, while others may want to take minimal risks.
>and you leave that up to each individual to decide, what's shared, and what sort of "community" is actually offered?
I don't think anyone is proposing an anarchy. At the end of the day, there may be some rulesets made by consensus, but it is beholden to the sub-community moderator and their personal whims, who is beholden to the community owner and their personal whims, who is beholden to some loose sets of laws based on their country. So this structure is impossible outside of some sort of decentralized p2p server setup (which sounds like a mess to communicate in).
When I say based on shared values and contexts, I mean at the community level.
So for a community where people discuss Harry Potter books, many members are not going to want to be careful about everything they say for fear of spoiling someone else's experience. But some people might want to experience reading the books for the first time with other people in the same boat as them.
If this community was well managed, they might solve this by:
- Writing into the community guidelines that spoiler tags only need to be used when discussing something that's NEW.
- Users have the ability to have spoiler content hidden/shown by default. The key here is that the definition of what a spoiler is, is still defined by the community, not the individual.
- Creating a separate area within their community for new readers to chat about the books, where discussion about events in future books is not allowed.
By joining the community, and agreeing to the community guidelines, you are agreeing to abide by an objective definition for what's appropriate within that community.
If you don't agree with those guidelines, you probably don't want to join that community. But without any shared guidelines, you do trend towards anarchy. Certain topic/niches will approach it faster than others.
Objective in what sense? Consensus != Truth.
>>If you leave it up to each individual to decide what is appropriate or inappropriate, and provide them with the tools to block content they consider inappropriate, that's a burden on them, because you're not taking care of it at the community level.
You can't take care of anyone at the community level because a community as an indivisible unit does not exist. The word "community" is merely a label, an abstraction for the ideas and events that characters interactions between two or more different people.
>>And if the community's strength comes from shared values, and you leave that up to each individual to decide, what's shared, and what sort of "community" is actually offered?
Either this is a loaded question or you've really put the cart before the horse, but I'll bite. A community's value comes the achievements of its individual constituents. Shared values are not a required prerequisite. They naturally arise from personal observations and discovery, real or perceived. That's it. Every culture, superstition, law, religion was a consequence of personal examination by one person. What made these so-called shared values "shared" was war and trade.
That means the burden is not on "people" in the sense of individuals. You can expect a certain content and tone coming to HN because the moderators ensure that. Yes, they're people too, but not in the sense PP and GP used it.
That was clearly an "each individual user should..." statement - and that's likely unsustainable for large user groups.
In political discourse, and in debates on big platforms like Twitter, it's the opposite- most of the discussion is about people's or groups' conduct and substance takes a back seat. Because a heterogeneous group with different values is involved.
So for social media and online forums, the question is, how big and diverse can the audience get while still supporting civil, substance focused discussion? HN does a pretty good job, and also has some obvious biases, scared cows, and weak spots. Online newspaper article comments probably have one of the lowest quality of discourse for a given participant size. What forum is best, I'm not sure, but its instructive to look at it this way, because it reflects politics generally, if we want to address real issues while maximizing participation.
We tend to pick our online friends according to mostly shared values, the same as real-life friends. Enragement ML then proceeds to amplify the differences instead of letting you experience that you're mostly similar, radicalizing both partners in the long term.
A chronological order minimizes that experience simply by the probabilities - if most of your values overlap for everybody in the feed, the chances of you seeing a large chunk you do not want to see is small. You're occasionally reminded that you don't always agree, but it's in a context of lots of other posts that don't violate your values.
(Part two of the problem are searchable feeds, where groups of people just wait to keyword-snipe)
Clubhouse is not just private phone calls. It's a social network.
Two people chatting fits this criteria. You can have a private chat on Clubhouse between two people.
Yes, you can have a private chat on Clubhouse. You can also have a private chat on Facebook or Twitter. They are all social networks.
I don't think trying to find edge cases for what constitutes a social network is helpful to this conversation.
I like the methods this site uses over others quite frankly and that seems to be quite a bit of human moderation to get to it. But i think scale also has something to do with it. Reddit was much like this site in its earlier days.
But I think moderation was more important. However much pg wrote about being curious or “intellectual honesty” or whatever, it was the change in moderation (to dang) that saw changes. But that sort of thing doesn’t fit all communities.
For one thing this one is basically public (there’s no shared-private things like subgroups or DMs) and tends to have reasonably highly educated members. For another, the volume is low enough that dang can deal with it all with only some automated tools, giving a reasonably consistent approach to moderation. Also I think he is paid much more like a software engineer in Silicon Valley than a typical social media moderator.
There are certainly some other more general things though, like the anti-flame mechanisms, the lack of more inflammatory media like pictures or videos, the lack of printed comment score and downvote limits, and the generally public moderation trail (though moderator may privately email users too).
The problem with most communities is that there is no culture being enforced — whatever grows, grows haphazardly and randomly.
It’s also why good communities tend to start small and grow slowly — and why eternal September is so deadly. If you have too many people enter the system too quickly, that culture can’t be enforced and trained before the next group of people come in. Rather than having the “elders” teach their ways to the “young”, it becomes the young teaching the young.
As an additional influence, those elders tend to leave and establish new communities, making it such that the problem can never be corrected.
You don't have to just ignore or exclude minority viewpoints to be a community. Most of the more welcoming communities typically embrace those as a matter of fact, and try to educate or evolve based on challenging their viewpoints.
Communities aren't only exclusive, they can be anywhere on a gradient between exclusive and inclusive.
An inclusive community may have a central focus, but would allow outsiders to participate.
Each of your examples have both inclusive and exclusive communities.
Um I like what you wrote, reminding: //wiki/Genitive;
'Indicating an attributive relationship of one noun to the other noun', 'serving purpose indicating relationships', 'may feature arguments', 'a head noun, in a construction'
...a shift, yes -but that may be way too offtopic (-;
There are plenty that do. Maybe some of the fundies dont...But the entire point of Christianity is to welcome non-believers and convert them. And plenty of Protestants and even most Roman Catholic congregations I have seen are very much that way.
Especially when a school is involved.
> In a Dungeons and Dragons community you cant call everyone a nerd and tell them to play Call of Duty.
Well of course not. That would be like...the opposite of my point.
>In the Black community you can't be a white woman with hair extensions and claim be to be the victim of discrimination.
Of course not. But having grown up relatively poor and in a minority of my neighborhood at that time, if you were to ask anyone from my neighborhood if i was a part of THAT community I bet they would say yes. And they absolutely were willing to fight with me/defend me when i was singled out for my, get this, race. I know plenty of members of the black community that are fine to sit down and listen to my opinions on some certain topics, again its all about respect and approach. I know plenty that would hear me out and dismiss me, and plenty that would shut it down (and i wouldnt even bother voicing in the first place).
It is like if you bought a doughnut from a shop, some will say it loud it is wonderful, if you have complains you probably will make sure you say it loud and clear, the majority people happy with it will just walk away silently.
This way, the voice you hear on internet are mostly from the two extreme of spectrum, nothing in the middle. It would be nice if we have some adjust somehow.
Even if it’s 3%, the algorithms can promote not just the polar opposite opinion (which is currently done to create “sticky” polarity), but the things that let people say “huh, I never thought of it that way”, or “I never knew that there were people in that situation” because those lead to empathy, understanding, and recognizing the humanity of people in other “groups”.
I recommend this solution because I think this is generally how human societies have traditionally worked. Humans have always been free to say whatever they want (meaning, it's physically possible to say any words), but there has often been a penalty for promoting ludicrous or hateful ideas: You weren't respected and were often ignored. If you gathered followers around you who threatened to become violent, you were removed. I don't see why the same system couldn't work for content moderation.
Of course, the fundamental problem is that controversial things trigger emotional responses which trigger engagement, and that's what social media's business model relies on.
Sometimes this went wrong, see yellow journalism. One thing that is different now is that you no longer have a say in who edits your newsfeed. You can't very easily switch news-feed.
I feel like the variation in opinions in modern-day editors is much smaller than the variation in opinions in society. Or maybe it isn't but the editing, being done more implicitly, is not convincing the wider populace that this is the way. That is, content moderators have much less authority (in the sense of respect) than old newspaper editors.
Many publications have ripped out their Disqus, Facebook commenting plug-ins that used to accompany their articles.
Internet moderation is too expensive. They'd be happy having Reddit and Twitter (users) do the moderation for them.
Those who are working for smaller shops, or who are freelancing are in a very different place than people working within or dependent on the big media machine.
What beat you cover matters too. Some topics come with more permissive norms all around. Contrast social issues with economics and foreign policy. How much agency one has to both report in an investigative sense and be questioned varies widely.
There is wrong, as in not adhering to the orthodoxy and or established narrative, and there is wrong in a factual sense.
There are also taboos.
How many stories have you seen written from the labor point of view, or that question the growing Washington Economic Consensus, or even recognize it, for example?
How often do you see major league corrections? Where are they placed, why?
If you are reading overseas, indie, freelance work, you have seen some, and often the work is clear, solid, easy to understand.
Side note: Investigative work is particularly troublesome right now, and where it is unorthodox, is being actively suppressed to an increasing degree.
And I want to be clear here!
This is not saying people in journalism are bad somehow, nefarious. No, they are working in a highly consolidated media environment, or they are out in the cold working hard to do the job, despite considerable pressure.
Secondly, the entities they work for really do not want those questions at all. This may well be the more significant force in play.
Doing real journalism these days is hard work that does not pay well. While that has generally been true, consolidation, which just got permission to contract even further, has made all this more difficult.
Access journalism is a real thing too. Ask the wrong question, despite it being one very large numbers of people want asked, and boom. Gone. No access. Gonna need a new gig and potentially have it covering a new beat.
Often, we see steganography more than journalism because of these things. People pointing that out in comments gets conflated with people being idiots, trolls and asses and it all gets shut down.
Finally, old media is more like broadcast than it is Reddit, or even Substack.
In broadcast, nobody actually cares to ask what people want, nor is there any real two way dialog. It is broadcast, focus grouped, researched all to the nines, very little actually being done ad hoc, or even outside advertisers interests, which are often the media entity interest, which makes it the journalists interest, or conflict of interest depending on where they are at personally.
There are facts and there are opinions as to what those facts mean.
Journalists who enjoy sufficient agency see it like you say and have no problem with critics.
Many do not have that agency, and the entities who employ them much prefer broadcast rules and norms where their authority comes from size and position not so much performance. And those people will do what they can, but will also admit it often isn't much.
In Japan, "access journalism" is also a serious problem with political reporting. But to be fair, so is the United States, but more subtle / less obvious. Example: Can Fox News really get a long-form interview with President Biden? (I am not trolling.)
User generated content moderation probably should include the step of "picking a context and a set of values" but it is a lot more than that. I can very easily see why someone who is great as an editor is horrible at user generated content moderation.
My point was that we used to have an easy way to pick (and switch) the "context and values" against which the media we consume are evaluated. But these days, 90% of people use the same set (as chosen by FAANG) and really, this "context and values" story is not part of media choice, nor is it a large part of media identity. In fact, I believe google and twitter try really hard to not identify as choosing what is true.
Why would spam, phishing, child porn be the 'universal' ones?
If you are making an argument that it should all be opt in...then it should all be opt in. Otherwise, this is the same drawing of a moral line that we all tend to do where we call ours obvious and others subjective. Maybe some people want the spam? Shouldn't the spammers have the ability to share it in case people want it?
My point isn't to argue for those things...its to say if we just accept that content moderation is subjective...we can't then label some things as subjective and some things as not, the framework should just be - laws of the state/country/equivalent structure. Those provide mechanisms (theoretically but more soundly) for feedback that corporations do not and frankly should not on the boundaries of acceptable and unacceptable content.
C.f., Nazi imagery in most of Europe.
However, even what child porn "is" sometimes depends of the culture or even the political climate. Like with the Japanese "lolis" (erotic drawings of fictional minor-looking girls), illegal in some countries but fine in others (so far).
The ship has, in a word, sailed.
If you create a vast, global, instant, high-fidelity, interactive content dissemination system with a strongly aspirational and appealing audience, then parasitic opportunists who seek only to serve their own rather than others' interests will flock to it. And have, in spades.
And I've seen this game play out repeatedly: in print, on radio, television, Usenet, email, and since approximately 1997, on the Web and mobile Internet.
There are parties already making truth determinations, and their doing a demonstrably horrible job at it from a common weal perspective. And that is the problem.
"Just don't use Facebook" doesn't work for two principle reasons:
1. Facebook is increasingly central to, or required for, numerous real-world interactions.
2. Even if, as I do, you don't use the service, you live in the world it creates. Facebook has massive negative externalities. Like, oh, say, civil war and genocide in Myanmar, to mention only one aspect.
(There are others closer to home for most readers here. I'm hoping HN won't lose its collective mind if I don't mention these.)
Another element that factors in is that Big Lie propaganda is reliant on disseminating the Big Lie. Scale is directly the problem, and offering unrestricted access to, choose the amplification metaphor of your choice, the printing press, microphone, camera, TV/Radio station, etc., has risks. Especially for those who would see the tools themselves burned down along with all else.
In which case, drastically curtailing the spread of any content from identified actors and their associated networks responsible for spreading obvious and notable disinformation ... is highly defensible.
I'm well aware of numerous arguments, predicated on or observing exceptions to free speech, which typically follow such statements. I find both the free-speech absolutist and the private property / private actor restriction privilege arguments tired and uninspired. The reality is complex, and I don't have either simple solutions or any which are conformant simultaneously with "free speech" or "property rights" positions.
Both, to my mind, exist in a nexus and network of overlapping interests and rights. I've suggested "information autonomy" as an alternative to "free speech", though the expansion of that notion quickly points out internal conflicts. Common weal might offer one path out.
You get lots of messages from Nigerian scammers, but the solution was not to prevent people from writing freeform emails. The solution was to build powerful spam detection algorithms, make it easy for people to classify emails to help strengthen the training set, and the problem is basically solved.
There's no easy answer to content moderation. There's no one size fits all solution, nor is there is some weird hack that's going to fix it. It's a part of your product. If you treat it as such, you're better off.
If you treat it as a separate problem that just needs money thrown at it or duct tape wrapped around it, you're never going to stop throwing money and tape at it.
Everyone wants an easy way out. You need everyone on your team in the room brainstorming solutions.
I'm curious why the people falling for them need a spam filter to recognize them as scams. I still see an occasional one slip through my email provider's spam filter, and I've never had any problem figuring out that they were scams.
Spam detection works well to eliminate email you don't want to see but nobody seems to have figured out how to apply it to social media content. Part of the problem (I guess) is that most of us agree what spam is whereas objectionable content is very much in the eye of the beholder.
- Individuals are subject to a fraud risk.
- Trust in the medium as a whole is decreased.
- Risk of false-positive spam flagging increases, decreasing legitimate message transmission reliability.
- Fraud will continue so long as costs of operation are lower than rewards.
Economic spam, in a word, follows an economic model, and is subject to economic attacks. The notion of some form of "email postage" is one ultimate form (thus far both impractical and widely rejected), but other "increasing the costs of business" have had ... significant if not perfect ... results in limiting the activity.
The calculus for political propaganda is vastly different, with the rewards of even a marginal success being tremendous, and the resources of antagonists being prodigious. It is very much like war, that most costly of economic activities.
It is war.
It is information warfare, rather than physical warfare. The weapons are messages, messengers, channels, informants, and audiences, rather than missiles and knives and fire and water and chariots (or tanks or helicopters or bombers or ...).
But Sun Tzu wrote of both types 2,500 years ago.
Unfortunately, the fundamental architecture and premises of a major contingent of participants fundamentally enables the attackes presently being conducted. (This is of course something of a tautology: if a terrain is fundamentally favourable to a specific mode of attack, that attack will tend to be employed, successfully, on that terrain.)
This is also the basic problem of growing communities, where you want to attract new people while still providing value to your core group, while still managing both attrition and predators. What content moderation problems have proven is that even with absolute omniscient control of an electronic platform, this is still Hard. It's also yields some information about what Evil is, which is that it seems to emerge as a consequence of incentives more than anything else.
In the hundreds of forums I've used over decades, the best ones were moderated by starting with a high'ish bar to entry. You have to be able to signal at least this level of "goodness," and it's on you to meet it, not the moderators to explain themselves. There is a "be excellent to each other" rule which gives very reasonable blanket principle powers to moderators, and it's pretty easy to check. It also helped to take a broken windows approach to penalizing laziness and other stupidity so that everyone sees examples of the rules.
Platform moderation is only hard relative to a standard of purity as well, and the value of the community is based not on its alignment, but on its mix. If you are trying to solve the optimization problem of "No Evil," you aren't indexed on the growth problem of "More Enjoyable." However, I don't worry too much about it because the communities in the former category won't grow and survive long enough to register.
I've had the same experience. And at the other end of the spectrum, the reason Facebook, Twitter, etc. have such problems with moderation is that there is no bar to entry--anyone can sign up and post. With what results, we see.
I actually think that Facebook's business model was premised on the Steve Carell movie, "Dinner for Schmucks," where a tool originally designed to signal you went to an ivy school got that cohort on board, and then invited everyone else for as kind of a cruel joke where they didn't know they were only the entertainment. Now the party's long over and the only people left are loud drunks yelling at each other and couples acting out some very public domestics.
Imo, there is no solution to those platforms' moderation problems, only management into something even more banal and worthy of disruption.
easy to check, but most sites tend to have almost zero accountability for moderators "staying excellent" and most moderators are often unpaid, normal human beings with their own personal beliefs. They won't be let go unless they frustrate the site owners.
And to begin with, people's perceptions of if "laziness" and especially "stupidity" should be aligned with "Evil" is a question for discussion in and of itself. Some people may just want to throw around some jokes and memes and not be in some pristine forum full of essays for response. It's not my personal preference, but I also don't think it's worthy of a blanket internet ban, nor do I necessarily want to say forums for those people are of "lesser quality" (even if you can argue that it attracts more truly "Evil" people).
I guess it's a tiptoe as moderation as always. I've also see some forums lean too deeply on quality and end up feeling gatekeep-y as a result (and not even for sensitive topics where it may be needed, but some "casual" forums, as advertised). You turn a community into work and people will look elsewhere.
It's going to be a lot like your credit score, your criminal record, etc.
At the end of the day companies want advertisement money and if you scare the adds away the same networks that control those adds are going to end up keeping track of you to keep you away.
Once that anonymity is completely gone, the internet will be just like real life. If you are a problematic assh*le, you'll get a record saying just that that employers, land lords, schools, etc. are going to check, just like they do now with credit scores, criminal records and school records and if you don't behave, you are going to be a marginal banned from polite society like it happens in real life outside the internet.
Thus giving full control of people to private entities that are increasingly not held accountable by anything or anyone.
Why would we want a private entity, not elected by the people, to decide our morality? Enforcing morality is dangerous and arguably immoral since it uses force to align people's thinking, which treats everyone like children that are learning instead of adults with freewill.
GP is talking about a world where you can't shop on Amazon because you committed wrongthink on Twitter.
Is it necessary?
Well - for many people literally the only way they communicate with their friends is in group chats on social media platforms. Losing that leads to social isolation.
Is it equivalent to the real world?
Of course it isn't equivalent - as in "the same as" the real world. For many people it is much much more important since most of their life is lived via online mechanisms.
My point: They can "check" all they like. In many places, various personal background issues are protected by labour laws.
Also, your post makes it sounds like being denied a rental apartment contract is unreasonable for someone with poor credit history. Would you rent to that person? Again: Some jurisdictions do not allow credit checks or do not allow /discrimination/ due to credit checks when renting.
I agree: Many items in the background check are ridiculous. Many years ago, I worked at a company where many of the Big Shots were regular, recreational cocaine users (confirmed by juniors who went bar hopping with them once a week). All new employees were required to pass drug tests. It was absurd. In that era, even marijuana was enough to have a job offer withdrawn.
i got the impression from the separation of church and state that we don't generally want elected officials deciding what is or is not moral.
merely what is legal.
We also want that enforcement to apply no matter what religion you have.
At least, I do.
i look forward to your support for making the tithe mandatory.
People say "it's not worth potentially getting shot or otherwise attacked to try to get that asshole to put on a mask"
https://geekfeminism.wikia.org/wiki/Who_is_harmed_by_a_%22Re...
And a nightmare.
Can you imagine the social cooling if this was in place?
Is the solution in the article? Do we simply need to recognise that as all society is online we now need online police? Online Community support officers (UK police adjuncts (think teaching assistants).
I suspect there is an overlap with the "defund the police" movement and the notion that we need to take away a lot of policing functions that are not actually violence / crime related - eg mental health. Social work is .. a lot of work.
Edit: It's worth noting that there are ~24 million or more police officers whose job description simply did not exist before Sir Robert Peel. That's a bigger number than i imagined !
wow: https://www.worldatlas.com/articles/list-of-countries-by-num...
And that brings up the issue of how, exactly, are we supposed to allow a literal “thought police”.
This will play out over time - call it 30 years - as we try to find out how to do lots of new things
- Handle mental health better. Partly we need medical breakthroughs, but social acceptance will be a huge improvement, as will standardised approaches, early years interventions and detection. It will take huge amounts of parent training - and the recognition that has genuine costs (how many start ups did not start because the parent decided to put their energy into supporting a child. And woe betide anyone suggesting that is another hurdle to be overcome with go getting attitude)
So we need huge investment in dealing with chronic mental health, not just medical, but social support, education etc.
- then sorting out acute. Look at any UK prison. It's basically young men with some mix of drug / mental health / abuse issues. Want to reduce the prison population - start 20 years ago. Don't define the police - simply introduce interventions so that in 20 years they won't need to do the social work job they do now
There is probably a dozen brilliant papers on this clearly showing what we should do and already modelled in a few enlightened communities globally. But it's going to take a decade of mistakes before those percolate up.
Let me know if you spot the me early.
After that it's social norms. We are trying to find a set of behaviours that are acceptable in the new online spaces. Public urination is frowned upon IRL - trolling is the online equivalent I guess. One can imagine things like loss of anonymity being the first part. Then slowly people develop tools to use humans inbuilt social mechanisms - so for example some asshat is intolerable, a record of their conversation is sent to the mother-in-law, 4 grandparents, all their wife's bridesmaids. (this may not work but you get the idea). We know this sort of thing works because every so often we all find that great viral thread where someone gets a comeuppance.
All of this does demand content moderation as ben says - and yes I do think all of this is too much. My take is social media will die back to manageable levels - if we are looking at a mass crowd situation, a swirling football crowd and asking, how can this crowd ever become a manageable city. Well crowds don't - crowds become cities, they disperse.
- There are too many forms of social media - each of us has one main form, and keep up with rest - just as we would have one friend in a crowd but interact with others. This will just die down. I mean how much value do we get from social media versus the constant worry / mind back that makes us check all the time. as humans develop social media innoculations this will die back
Secondly ad dollars will help die back social media - really it's amazing no one has noticed it's a scam and waste money f money.
At some point regulations and advertising will drive to a point where it is simply easier to dump the algorithm, stop driving for engagement and just supply the limited feed of friends posts, interspersed with billboard ads. Influencers will still influence, ads will still pop up, but as we are choosing who to follow we won't care. Just follow fewer people.
Agents - the other big silent one. I can run a refer over my emails but the facebook client won't let me. This is the other big change likely to occur - software agents acting in my best interest filtering shit for me.
Anyway bed time
International issues would get in the way. As much as it sucks, most police won't act on a death threat made from some 12 year old in Brazil towards a normal US citizen. It's a lot of resources for no benefit on their end.
it wouldn't be enforceable unless websites are enforced to only be used by citizens within that country (and worldwide punished for not abiding), with specific embassies for international conversation. Some cases may even divide it within states/providences.
I see that as the worst case of a medium that was created with the goal to bring the world together.
I am not convinced "online police" are the right solution, but if we want to avoid walled gardens everywhere, if we want public spaces, we need public peacekeepers.
In most cases, a quiet word from a police officer, who s enough to move the situation on, usually enough to prevent it escalating anywhere.
The rest is likely to be slowly dealt with as part of a society wide system of mental health.
A quick example - a long time ago I got a note from @dang saying my 2am comment was "racewar" (turned out he mean the comment above me) but it contains most of the issues we discuss here - a dumb comment, a quiet word from an official peacekeeper, a process of escalation / complaint and a just resolution - certainly left me thinking harder about my 2am comment habit.
that seems to be what moderators/admins do currently. I know many focus on the worst, most toxic communities, but for many a simple warning will diffuse some small quibble. It seems this article and other individuals argue that this isn't sufficient, however.
I get it, the platforms are his bread-and-butter, but at some point he needs to stop pooh-poohing the solutions others are putting up and, you know, actually suggest a solution himself.
In this article he presents the possiblity that moderation does not work in scale, and that we should stop pushing for it. Then what? Well, Benedict doesn't get into that. He suggests that change might be necessary and that it might be a complete change to a different model of interaction - but he stops short of suggesting that is might be up to the platforms (or the government) to enact that change. If he were to turn around and say "Twitter, you need to fix your shit. Stop allowing unlimited retweets of some poor shmuck's bad day tweet" then he would actually have a position worth considering.
Edit: Changed the last line because I got distracted when writing the original and forgot to finish it.
The analogy might be that there is no solution to the problem of the mortality, but there are quite a few ways you can work to keep from dying of a heart attack in your 40s.
This then somewhat disincentivises bad behaviour and, in particular, repeated or automated bad behaviour. Surely this would harm growth and it doesn’t feel like you’d have success charging a smaller fee in poorer countries. It could also be a recipient for being very unpopular with your users whenever they get punished. And I would worry that it would just incentivise spammers to steal credentials.
So it's not a foolproof system, but at-least it can help keep the lights on for people who are terminally online (unless they do CC chargebacks).
Also we're finding that some of the features that enhance user privacy could help reduce bad behavior, like defaulting to smaller, closed groups of people.
But it's a fascinating challenge and there's so many facets to it – glad people are starting to think about different ways to approach the problems with social!
(Sneaky pitch: if this sounds interesting to you maybe let's work together! https://frenlo.com/)
You basically create a feudal vouching system, were highly engaged community members, vouch for others, who again vouch for others. If people in this "Dharma-tree" accumulate problematic behaviour points, the structure at first, bubbles better behaved members to the top. If the bad behaviour continues, by single members or sub-groups, the higher echelon of members will cut that branch loose, or loose their own social standing.
Reapplying members, would have to apply at the lowest ranks (distributing the work) and those would risk dharma loss if they vouched for sb unworthy in the trial out phase.
We never solved the bad tree problem though. What if there exists a whole tree of bad apples vouching for another? You can not rely on the other feudal "houses" indicating this correctly, due to ingame rivalry.
That's when you add top moderation, so the algorithm becomes a way to scale the moderators, not a full moderation solution.
You can't create an algorithm that solves moderation, unless you create a fully featured AI with a value system.
i still haven't seen a moderation system better than slashdot, which community-sourced its moderation/meta-moderation semi-randomly. though it still had issues with gaming and spam, it seems like a good base to build from. and yet we ended up with twitter, facebook, reddit, yelp, etc. that optimize for (ad) views, not quality.
It's true that this will lead to echo chambers, but by looking at vouching relationships rather than the contents of posts, it should be easier to detect the echo chambers and give people the opportunity to expand their horizons.
As long as people could choose different communities that had moderation policies which suited their members, it would allow those who wanted to hear other perspectives to do so, and might make people aware of what sort of echo chamber they are in.
Bad actors form massive communities, and quickly. You can look at the rise of various networks, such as T_D on Reddit, or the rapid growth (and often fast decline) of other such forums elsewhere. I find it very difficult to believe that such growth is organic.
(Commercial and brand accounts / pages / platforms often follow similar trends, and for similar reasons. Advertising and propaganda are one and the same.)
At worst you lose imaginary internet points if you say something that the group doesn't agree with.
It is something of a collective filter bubble, and there is a pervasive criticism from those who largely don't participate here of typically HN behaviours. I agree with a fair bit of that. There are certainly topics HN doesn't seem to be able to reasonably discuss. (This seems to be a frustration of the mods as well. They're certainly aware of the issue.)
My own (in my view at least) largely contrarian voice seems to have been reasonably well tolerated here, though.
I mean, it has though.
When I want to buy something new, I find Amazon reviews to be far more helpful than anything else that has ever existed. Obviously you can't only look at ratings or only read the first review, but it's pretty easy to find the signal amid the noise.
Similarly, TripAdvisor has given me far better recommendations of sights to see while traveling when compared to Lonely Planet. Yelp is eons better for finding great restaurants than Zagat ever was. And so on.
I don't understand how you think these systems are "doomed to fail" when they already exist, are used by hundreds of millions of people, and are better than what they replaced?
The most troublesome users were often the most prolific posters. The people who had a lot of free time on their hands to post every single day were often the most disgruntled, seizing every issue as a chance to stir up more controversy.
It was tough enough to reel in difficult users when they had no power. Giving extra power to people who posted the most would have only made the problem worse, not better.
The most valuable content came from users who didn't post that often, but posted valuable and well-received content occasionally. I'm not sure they would have much interest in moderating content, though, because they would rather produce content than deal with troublesome content from other people.
Content moderation is a pain. The trolls always have infinitely more free time than you do.
I think empowering the "power users" like this inevitably leads to Stack Overflow-style communities, where arrogant responses are the norm, the taste of a few regulates the many, and the culture of the community ossifies because new contributors are not welcomed.
I'm lost on how to do this within an internet community, however. You don't want burners being made to report over a disagreement in opinion and impact them. Or worse, power users tearing each other down in retaliation. I don't think "election periods" would work well either because few people really care to be a moderator to begin with. I can only see it being done by having professional moderators and having a manager review any potential user complaints. Which I'm sure has other problems.
More posts if people rate your posts highly, to some useful maximum.
Less posts if they rate your posts poorly, down to a set minimum that every starts at.
Then all the fancy IP detecting, fingerprinting etc tech could filter duplicate accounts and be a boon for once.
An early form of this appeared as an observation on Hal Varian's page at UC Berkeley (he's an economist, was with the School of Information Management Science, and now works for Google).
On HN it's interesting to look at what kind of riff-raff appear high on the leaderboard, vs. revered names in the field who may have posted only a few times.
Neither truth nor quality are popularity (or volume) contests.
But the challenge is still the same one you describe -- what do you do with competing "groups" or subcommunities that flag radically different things. What do you do when both supporters of each side of a civil war in a country consider the other side's social media posts to be misinformation and flags them? Or even just in a polarized political climate?
I still think (hope) there would have to be some kind of behavioral signal that could be used to handle this -- such as identifying users who are "broadly" trustworthy across a range of topics/contexts and rely primarily on their judgments, while identifying "rings" or communities that are internally consistent but not broadly representative, and so discount that "false trustworthiness".
But that means a quite sophisticated algorithm able to identify these rings/clusters and the probability that a given piece of content belongs to one, and I'm not aware of any algorithm anyone's come up with for that yet. (There are sites like HN which successfully detect small voting rings, but that's a far simpler task.)
Even just doing it very naively and choosing, say, a fifth of your users for each post and only giving them the option to flag it might help significantly. It would probably make it more difficult to motivate the members of these problematic groups to actually coordinate if the average expected result was the inability to participate.
And you could do it in more sophisticated ways too, and form flag-capable subsets of users for each post based on estimates about their similarity, as well as any other metrics you come up with - such as selecting more "trustworthy" users more often. This would help gather a range of dissimilar opinions. If lots of dissimilar users are flagging some content, that seems like it should be a strong signal.
The main tool is to drop the idea of a global definition of trustworthiness: You have seed IDs to find people who supply CAPTCHAs, but otherwise all trust is computed locally.
If you want to try this, there are two steps:
- First the current state in Freenet: https://github.com/xor-freenet/plugin-WebOfTrust/blob/master...
- Then the optimizations needed so this scales to arbitrary size: https://www.draketo.de/english/freenet/deterministic-load-de...
Here’s some data if you want to test algorithms: https://figshare.com/articles/dataset/The_Freenet_social_tru...
And some starting code of a more generic prototype for faster testing: https://hg.sr.ht/~arnebab/wispwot/
(if you prefer a slightly larger writeup that might expand as time moves: https://www.draketo.de/software/decentralized-moderation.htm... )
If you keep seeing content you like (through retweets), you can follow that person directly to get more. If you see content you dislike, you can unfollow the person who brought it into your timeline (by retweeting it).
Of course this would work a bit better if there was a way for accounts to categorize posts they author or retweet. You might follow me for tech-related content but not care much about my French politics content, which I would be happy to categorize as I post/retweet but have no way to do on current Twitter.
Besides - these are evolutionary games being played between grazers (content consumers) and predators (alarmingly large group).
As long as there is signal in a forum, there will come to be a method to subvert it.
Honestly the question I would ask people is how do you measure bad behavior on a forum.
Any technical idea, such as your tree, is doomed to eventual obsolescence. The question is how long it would take, and how effective it would be, and how you would measure it.
When you say highly engaged community members I hear people who have no life who derive self importance via imaginary internet points and social position not by doing things but by running their mouths. While it claims to encourage community it discourages it by potentially punishing association. It would make people afraid of being associated with ideas others consider bad which sounds great if you imagine communities run by people that are largely or mostly good and intelligent when in fact people are largely bad selfish and stupid.
It would be ruthlessly gamed by individuals whose status would be based on their efforts to stir up drama that sounds fantastic when its directed at people like Epstein or Harvey Weinstein less so when you realize that this would be effective at silencing people regardless of guilt because people would need to as you say cut the branch loose.
I have literally never heard a worse system of meta moderation proposed.
I've been applying a somewhat similar notion to collecting and managing reading material and suggestions, what I call BOTI, or Best of the Interval. It's a round-robin style system (think "43 Folders" from GTD), where I compile a list of references over a period of time (monthly to annually seems to be most appropriate for me, though hour / day / week / month / year / decade / ... could be applied). At the end of an interval, some limited number of items is carried forward.
This is one way of addressing the "firehose of content" nature of information, recognising that in any given time period, you only have so much personal bandwidth to dedicate.
With the federated model, content federating is itself subject to assessment, and is effectively re-vetted. Note that different communities might favour different content: cooking, kittens, Kabbalah, canoodling, cypherpunk, core meltdowns, classic cars, concerthalls. Vetted / re-vetted streams themselves might be of interest.
Both community- and time-based elements of this could get interesting.
Another lever is related - decide how much you want to limit the kind of content. A like button is easier to moderate than upvote/downvote, which is easier than a poll response, which is easier than restricted markup, which is easier than allowing unsanitized html/js/sql. (I think there's a lot of unexplored territory between "poll response" and "restricted markup", in terms of allowing people to participate with the generation of content.)
Then there is distributing the moderation abilities themselves. Can users become moderators or only admins? Is there a reputation system? I miss the kuro5hin reputation system and would like to see more experiments along those lines.
And then finally you get to the hard stuff, the arguments about post-modernism and what truth is, creating codes of conduct, dealing with spirit-vs-letter and bad faith arguments, etc. Basically the "smart jerk" problem. I hate that stuff. I want to believe something simple like, as soon as you have a smart jerk causing problems, it means you've given them too much opportunity and should scale back, but I think it's not that simple.
For those unfamiliar, Kuro5hin, inspired by Slashdot, was an online discussion site where individuals would submit items, along with a (hopefully) brief description, or an original essay. There was no editorial queue (as Slashdot had and has). Both posts and comments could be rated on a five-point scale by individuals, and the ultimate score landed in the bound 1-5 (later extended to 0 or -1 with flags). Moderation converged on a mean with time, at least in theory. Users "mojo" (karma) would follow similarly.
This avoids both the runaway endless votes problem of sites such as Reddit (or Hacker News), and the behaviour of Slashdot in which moderation would rise or fall but never converge and was artificially capped at 5 points. On Kuro5hin, the 100th moderation score was weighted as 1/100. On Slashdot, the 100th moderation score had a weight of 1.
If this sounds something like a star-based review system such as Rotten Tomatoes, or Amazon ... it's because it is. And ultimately the same problems apply:
- Sybil attacks in which sockpuppet or shill accounts are created to sway moderation / review scores are quite possible.
- If the site fails to attract (or motivate) consciencious moderators, the moderation quality still falls.
- Information on divergent (strongly bimodal) rating is lost. 10 moderations split between 5 '1's and 5 '5's look the same as 10 '3' ratings. The information contained is not the same. (Summarising complex data is, it turns out, complex.)
- There's still no distinction between popular and truthful content. Or between "this is true" and "I disagree with this" (or don't like it, or the submitter looks funny, or whatever).
The K5 system was interesting, but ultimately not a magic bullet. The site itself failed, though for complex reasons.
The original concept didn't have that.
Another approach was tried by Advogato, which had a "trust root" (ultimately Raph Levien). It was based around individuals rather than content and ... that also has issues. Nothing pathologically bad in the case of Advogato, though the site largely languished.
For Levien's concept, see http://www.levien.com/thesis/compact.pdf
Should all forums rules conform to HN?
In the sense that we all value curiosity, yes.
We are all also very different. When political posts go up you see it; value, perspective, economic, and educational differences are all highlighted simultaneously.
I think what keeps us from destroying each other is that the binds that bond are curiosity, and those bonds are strong enough for now. dang also sacrifices his sanity going around and nudging people back in the right direction.
Those bonds are nothing to be understated though. I've tolerated some edgy opinions on this website, and probably given some too, but I also come here to learn about different perspectives and genuinely do enjoy them even if some offend or hurt me. Other people see those differences and talk about making lists.
Yes, exactly! I think you nailed the core of it. The communal drive to pursue curiosity is, for now, stronger than the communal drive to make lists; in comparison, there are many large subreddits where you couldn't expect the same thing.
It absolutely is, which is the main reason dang is constantly having to talk people down off of ledges. This site has old-school hackers and new school VC startup hackers, anarchists, communists, socialists, capitalists, libertarians, Libertarians, atheists and believers, alt-right white supremacists, BLM supporters, QAnons, scientists, "scientists", skeptics, shitposting memelords, furries, pro Stallmanite free software zealots, anti-Stallmanite free software zealots, Stallman agnostic free software zealots, FAANG millionaires and people living in their cars.
Name a political position, no matter how repellent someone will come along to advocate it, sometimes just for kicks. Name a programming language, you'll find someone who loves it, someone who hates it, someone who's never heard of it, and probably the person who invented it. That diversity is one of the good things about this place, and one of the worst, because it brings out the worst in people.
There's a very small domain in which discussion on Hacker News is merely contrarian and contentious, but stray outside of that tranquil oasis and you're suddenly in the Thunderdome being chased around a gladiatorial arena by an overclocked sea-lion and a troll with a sniper rifle. But the troll is civil, and that makes them better than the trolls on Reddit.
It's honestly a wonder this place hasn't burned down yet. It's made me want to burn it down a few times.
Very different demographics then world at large.
You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
On the other side there is unmoderated Phoronix, which has the worst comment section that I've ever seen.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...
When the content is more specific, like PC master race, or people that drive VW bugs, then the community identifies itself as a single tribe, and they tend to treat each other well.
The crux of the issue is that there's no "cost" to be bad. If there was a "cost" then bad actors would go away for quickly. Any "cost" you impose will be diametrically opposed to popularity - but a low volume/unpopular site is unlikely to be abused to begin with.
Automating that away is the only path to being on equal footing. If you introduce any human element, not only will it be a bottleneck, but the cost could be large enough to bankrupt even the largest companies.
In my opinion, if you can't hire enough people to moderate without going bankrupt, then bankrupt you go! Would this mean we can't have social media? Maybe. Probably for the best.
But you can't moderate then you go out of business. That's the way it should be. Probably then people would find a way to moderate and stay afloat.
I think the solution to this is smaller, more isolated groups -- with limited edges between. Back to email threads and Mumble servers, imo. The downside is we'll all be living in filter bubbles, but I think any shared community with a common value (like a video game) is better than some ginormous platform like Facebook.
Facebook has even killed craigslist. Or at least greatly reduced the usefulness.
That already seems to be the case.
the vast majority of websites operate perfectly fine without a comment section (and some already have removed them). This isn't as far off a reality as you jest.
In any case, it's ultimately a cultural choice, like any other social element. Maybe America somehow bans comments without ties to real world ID and all the major american sites pivot into little more than coporate news aggregators. It wouldn't stop other countries and their respective forums that users seeking discussion would run to. And I doubt there would be some strong enforcements to make sure Americans aren't commening on WeChat or Weibo or some other non-American service.
In my own experience, it's the trolls that are rather confounding.
Go to any twitter poster with even a slightly political bent. Look for the first shitty person. Look at their posting history. It'll nearly always be 100 post/day of shittiness telling you just what you need to hear. Unless the evil Russians are extremely clever, it all appears to be grassroots poor behavior.
I guess you can view social media as a giant laboratory showing the behavior of people when they are not nose-to-nose with you in a bar. It's all super disappointing.
Maybe there's a place for highly curated social media.
If you are found to be a fake person you forfeit the bond. Now it costs $100,0000 to create 10k fake people and you can lose it all tomorrow.
Is this doable? Or maybe these requirements together are contradictory?
Essentially registrar and major sites would coordinate to get bot accounts revoked your registrar wouldn't want to cancel user accounts erroneously. In most cases logically the registrar could be your email provider which would just be selling a verified email address but a third party could offer the same thing.
So yeah, I agree with TFA in the sense that these are problems to be solved largely at the system level. For example, compared to Twitter (where anyone can reply-to, quote, RT, and @user anyone), Twitch and Tiktok seem to do well at permitting individual creators to have their own space with their own exclusive authority over what is and isn't okay in the space. And they have (or at least enable to exist) lots of tools for exerting that authority— witness things like "bye trolls" scripts on Twitch that do have to be set up in advance, but then can be used at the drop of a hat in response to brigading to immediately close the stream to new followers, and disallow posts from non-followers, plus delete chat posts from anyone who joined the stream in the last X minutes.
with the utopic assumption of infinite moderation, I don't see a problem here. Different houses will have different rules and levels of moderation (a meme community will have different standards from a more sensitive topic). As long as the moderation informs newcomers of the rules, they behave under that sub-community if they choose to participate (while respecting global community rules, and of course server country laws).
This is all under the assumption that "sub-community" is distinctly defined. For something truly monolithic like Twitter, the only choice seems to be to leave control to the individual, and perhaps allow private groups to be formed.
I've always maintained Facebook made a mistake when they took responsibility for misinformation posted on their platform. Now, four years later, they're continuing to double down on this stance, and forming "ministries of truth".
Freedom of speech is a powerful concept, and does not like to be stifled by types that argue it should only apply to the government. When you fight against that principle, you win in the short term, and lose in the long term. We are now starting to see those ugly realities of the long term losses, four years later.
The author is touching on something prescient here, but I disagree with some of his observations. For example, that the solution to "virus scanners" playing whack a mole was to move to cloud computing. (The solution was clearly to improve software, with things like memory safe programming langauges. Moving to cloud computing reduces freedom, not enhances it, and centralizes all the valuables in a single location a la Tower of Babel.)
If you remove the "algorithmic feed" mechanic, much of the abuse vanishes instantly. I am shown what came latest. Not this weird algorithmic mash of content that has been gamified for my attention. RSS is the way to go.
I don't remember this being such a huge deal, and there was always the alt groups.
Also, Usenet was so niche that state actors weren’t running troll armies for propaganda purposes, but this is something any modern social network has to deal with.
There was certainly less of it; all of Usenet fit in a box of magtapes. But it could still have some pretty big tempests in that teapot.
It never even got close to solving spam, which exploded after Eternal September. It took AI-esque systems (and enormous heaps of data to feed them) to reduce spam to a manageable level. Trolling is a harder problem than spam.
I did some analysis comparing Slashdot (with distributed moderation) traffic volumes to Usenet (centralized moderation) in the late 90s.
If I recall, the peak volume of posts to the largest (moderated?) Usenet newsgroup was about 3000 posts/day. I seem to recall it was alt.fan.rushlimbaugh or something. I don't know about the quality of moderation but I took that to be the "peak traffic a single/paired human moderator" could handle before we had databases and distributed user-moderated content.
Not sure what dang et all are handling here but it seems higher than that nowdays (but not more than 10x higher with all the newer technology they've developed?)
And it fell to Eternal September.
The really only way to moderate a group is keep the group small.
In the last year I've only had to kill two posts, both from the same troll.
On the other hand, I use my personal killfile quite liberally.
Before MS there was no bad stuff on the commodore 64. It just didn't exist. Loading things from tapes, disks or the internet doesn't matter. You switch it off and on again then load the next thing. I see no reason why this cant scale. You would have problems if you allow unchecked io and remote code execution and you would have to deal with that but even then a simple reset would clean it up. There is no need to give strangers the keys to your home and offer them a place to hide where you cant find them.
> Virus scanners and content moderation are essentially the same thing - they look for people abusing the system
The problem is that it is not your page. This forces you to live up to someone else's standards (if not forein laws) It is like the PC architecture where the computer is not yours.
Facebook is really what web standards should have offered. I would probably have been against it myself but in hindsight it is what people really wanted.
>...content moderation is a Sisyphean task, where we can certainly reduce the problem, but almost by definition cannot solve it.
I don't know, perhaps we can. Should we want to?
> I wonder how differently newsfeeds and sharing will work in 5 years
Ill still be using RSS and Atom.
as a nitpick, C64 wasn't really "before MS" - it booted to a BASIC which was derived from Microsoft BASIC!
Say you physically contain each app by giving it its own processor and its own memory/storage. If it wants access to anything else it needs specific permission managed by something baked onto a chip. A complex detailed permission system. You don't just get to read keyboard input but if you do you don't just get to read it when the application is not focused. If an application is registered to work with a file type it can have access to those but only in defined locations and only that file type, reading and writing separately. If it wants access to anything outside its scope of permission the user not the developer can pick what permission is to be granted and for how long.
Add 2 mechanical switches that make it physically impossible to write to and/or read a drive.
It seems annoying and bureaucratic but it would allow us to download and run any code from the internet which would be truly exciting.
Right now I cant even plug in an usb device and when I do there is no way of knowing what happens to it. It seems absurd.
[0] - https://www.researchgate.net/profile/Stephen-Cobb-4/publicat...
> an app can’t run in the background, watch what you do, and steal your bank details, because the move to a sandboxed model means applications can’t run in the background and watch what you do.
It can't because it can't. Got it.
To talk about the topic in the title though, content moderation is a dead end, not because of the need to do it, but because people will not tolerate it. Nobody has a monopoly on information sharing on the internet, no amount of justification will get people to sit and allow it. All moderation is good for is keeping your community on focus, trying to prevent any ideas, even bad ideas, from bring shared is a bad idea in itself. It is a dead end not because it is an unsolveable problem, there's a deeper root here. It is a dead end because the people you're trying to moderate don't see a problem that needs a solution in the first place, and they're going to avoid your attempt at controlling their voices and the flow of information. And they should.
One possible approach is something like Stack Exchange does: new users acquire their rights gradually. New accounts can only do little damage (post an answer that appears at the bottom of the list, and is made even less visible when someone downvotes it), and if they produce bad content, they will never acquire more rights.
Another possible approach would be some vouching system: moderator invites their friends, the friends can invite their friends, everyone needs to have a sponsor. (You can retract your invitation of someone, and unless someone else becomes their sponsor, they lose access. You can proactively become a co-sponsor of existing users. Users inactive for one year automatically retract all their invitations.) When a user is banned, their sponsor also suffers some penalty, such as losing the right to invite people for one year.
There are probably other solutions. The idea is that accounts that were easy to create should be even easier to remove.
https://en.wikipedia.org/wiki/Web_of_trust
I've not examined it closely, but Web of Trust follows that train of thought at least to an extent.
But to me, a more closed system is less valuable. Certainly it lacks the network effects that seem to be needed these days to make it financially.
For moderated deliberation to achieve consensus in decision making, here's a write-up for a system that combines ideas from StackOverflow, Reddit, and Wikipedia:
https://bitbucket.org/djarvis/world-politics/raw/master/docs...
Why not just delete [most] offending comments, immediately, no questions asked (and ban repeat offenders)? For maybe 95% (as a wild guess), there's no question - it's clear that the comment is inflammatory or disinformation or whatever. It surprises me that I see so many of them permitted in so many forums, even here on HN. Why tolerate them? One click and move on.
Tell people about the policy, of course, and if the comment is partly offending and partly constructive, delete it. They can re-post the constructive part. It's not hard to behave - we all do it in social situations. If you want your comment to be retained, don't do stupid stuff.
------------
Also, it's telling IMHO that in this conversation among people relatively sophisticated in this issue, organized disinformation is barely discussed. It's well-known, well-documented, and common, yet we seem to close our eyes to it. It's a different kind of moderation challenge.
How do you define offending? Some people are offended by a great many things. China is offended if you point out they have human rights abuses. The US is offended if you point out their interference in other country's affairs. Thailand literally makes it illegal (jail time) to offend some people that in the US is not only legal, but encouraged by our culture!
It's not so simple or easy.
But you can always moderate the moderators.
In reality, the human mind doesn't need and very rarely uses the extreme of hard and fast algorithms; we are not computers. I can judge good from bad, constructive from destructive, etc. and it is easy to identify most of the problematic comments. When it's a forum with rules, it's easy to identify (again, a wild guess) 95% of them.
But we are just speculating; can someone with actual experience and expertise say how it would work?
At that point it becomes about attack surface areas and patterns.
We're not talking about prison and the law of the land; we're making decisions about the disposition of some comments on an Internet forum. Far more consequential decisions are made without any due process - for example, managers decide on whether people will keep their jobs; they are 'tyrants'.
You are outnumbered. Spammers or bad actors run 24/7 whereas you need to sleep or holiday. You are criticised for moderating (“censorship!”) or not moderating, if you are ever inconsistent you’re “playing favourites”. Persistent bad actors skirt the boundaries and dedicate themselves wholly to getting past you. Trolls work in the grey areas. You and your parents get doxxed for handing out a minor imposition.
You are up against spam bots working constantly to bypass you. Up against bots skirting what contravenes rules, or spammers asking an “innocent question” to set up another account to provide a spam answer.
Any question remotely involving politics/gender/race devolves immediately into a shitshow (this is a newish phenomenon).
That’s essentially editorial work, and the “good” part is subjective.
Mods also spend 90% of their time dealing with bad content, which means they end up be far more sophisticated at smelling false arguments or traps that would snare normal users.
Eventually, attackers start targeting the mod team, and the rule set - since the experience/content delta between mods and users keeps growing, attackers are able to exploit it for PR.
Therefore, having a relatively miniscule number of people be the judge and expect them to not abuse power, or thinking some clever algorithm won't be exploited, is short sighted and maybe technically naïve. I don't know what the solution is, but it might be having the Internet become independent from all nations, and have it be its own nation with laws, etc... not sure, but it does seem like an analogous problem to physical humans living together in a civilization, it's just still in the making, it seems.
“Who gets to decide what’s true?” is the wrong question. We should be asking “how do we determine what’s true?” https://blog.nillium.com/fighting-misinformation-online/
Maybe it’s okay to be waist deep into QAnon schizophrenia so long as the rest of your life is also okay, and vice versa. Though those imaginations aren’t the kind of powdered grape juice to me.
In my view the only proper way to handle content moderation is that every user of these “free“ social media platforms over the age of 18 should should be required to moderate some proportion every month to understand what’s actually going on.
Does that mean no vetting at all of the moderators? Anybody can become a moderator? But then you have QAnon in large numbers moderating content on like the CNN Facebook page or something? I really, really, really don't think that is a "proper" or even tenable moderation solution.
There are too many people who would abuse the moderation power. Moderation should at least be a paid position, paid well in fact, and vetted before allowed to moderate. Otherwise it will be worse than before.
I believe very much that bad material no matter how disgusting is best handled through public exposure, not censorship.
No interrogation of why it spread was investigated. It immediately made me wonder, maybe people just thought "false" information was a whole lot funnier, and hence sharable.
The problem with misinformation is it vaguely doesn't exist. There's parody, non-orthodoxy, true things people disagree with for political reason, unproven things, urban myths, rumors, etc etc. These are all classes of information people blame all the ills of society on.
You don't think that people lie on the internet?
The transmission cascades and cascade sizes for false info are studied. Misinformation for example is repeated, like a nuclear feedback loop, or more like gossip being repeated between people till it becomes a “fact”.
Science related info propagates through the network only a few times, as if people who read it understand it and move on.
Do you think that a site should be forced to host other people's vile content?
I'm to be responsible for what someone else's views are?
Nonsense. This is the cry of the censorship apologist. This moderation draft you speak of... what guidelines will the draftees follow? Will they moderate out of the goodness of their hearts? Or will they follow some standard? (I'm sure many people would be extremely eager to author! You mean I get to decide what is permissible to discuss online? What a wonderful avenue to advance my political causes by force!)
The author is right. The solution is not to double down on moderation.
An alternative is to generate an in-forum currency that can be spent on comments, either on a per-post or per-word basis. This currency could be earned based on reputation--but as we see here on HN, and many other places, upvotes does not always go to the most thoughtful and engaging comments--or some other metric (statistical distinguish-ability of posts from the corpus of posts? not sure).
Per-account limits don't apply to those with unlimited accounts.
Some form of payment (or hash cash) is an option, but hugely favours those with the ability of those who can pay (or hash). This correlates strongly with current sources of disinformation.
off topic but, yikes, what a bad analogy.
Considering the fact that motor vehicle crashes are the leading cause of death for children, young people, and unnatural death in general I dunno if one should compare anything positively to our society's decision to embrace the car.
Accepting the insane deaths, dangers and terrible second order effects deriving from car oriented infrastructure and then just shrugging because of convenience has been one of the most misguided policy mistakes of the 20th century.
It's been an absolute disaster for the environment, and safety is not great, as you say. But I don't think it's as cut and dry as you make it.
Obviously we've made some mistakes and possibly have gone too far in one direction on that axis, but particularly with regard to personal safety, people need to be willing to draw a line as to what is "good enough". Cars at this point are extremely safe by any reasonable metric, even if the absolute totals feel large.
Cars are not safe. They're the number one killer.
In contrast, how many people are killed by the automobile alternatives of cycling, public transit? Seems that those are safe to me.
"safety not great" is an understatement
Singapore has similar GDP per capita to the US but about a fifth of the number of cars. Most of Europe is at half the number in the US.
https://www.bicycling.com/news/a30433288/oslo-vision-zero-go...
This is effectively already what many newspapers do in disabling commentary on certain articles.
I often notice this on the CBC that whenever there's any article about indigenous people posted, comments will be disabled to prevent incredible amounts of racism.
That isn't how cars generally work. Without advanced electronic controls, it's not possible to build a car that can't exceed the speed limit. You can try to limit the acceleration but not the speed. It's basic physics. It's worrying that the author used it as an analogy (metaphor?) but doesn't seem to wonder why cars really don't have speed limits.
Everything comes down to political will. The German manufacturers have an informal agreement to limit all their cars to 155mph as to avoid the government adding further limits to the autobahn.
https://www.remarkbox.com/remarkbox-is-now-pay-what-you-can....
I’ll argue that the main product of reddit is not the community, its the content moderation. Maybe eventually the content moderation toolkit.
At this point reddit mods and users must have collectively built the largest collection of regexes to identify hate/harmful speech For a huge number of sub cultures.
And good submitters != good reviewers.
That said, a variant in which less vetted content has to be reviewed by more vetted reviewers, before gaining wider disclosure, seems to me to have merits.
Maybe that applies to iPhones but Android certainly allows it.
Why would you stop reading after a true statement about iPhones...?
I'm delighted that the net is a way for nazis and other jihadis to find each other. It's a global honeypot. Driving them underground doesn't make them go away, just harder to find. We saw this with the January 6th crowd.
We also saw this with the publicity-seeking attorneys general who got rid of craigslist hookups and back page: sex trafficking continues but is harder to find and prosecute.
Spam, google bombing, and related activities. These are noise, generally.
Misinformation is slippery. Often this gets conflated with differences of opinion. That is happening a lot right now as moderation is politicized and weaponized. More than we think is debatable and should be debated rather than legislated or canonized into an orthodoxy, flirting with facism.
Clearly criminal speech, kiddie pr0n, inciting violence, etc. These are not noise and can be linked to either real harm as a matter of the production of the speech (kiddie pr0n), or can be linked to the very likely prospect of harm. Material harm, is an important distinction segway to:
Offensive material.
Being offended is as harmful as we all think it is. Here me out, please:
To a person of deep religious conviction, some speech can offend them just as deeply. They may struggle to differentiate it from criminal speech, and in some parts of the world this is resolved by making the speech criminal anyway. Blasphemy.
That same speech might be laughable to some who are not religious, or who perhaps hold faith of a different order, sect.
Notably, we have yet to get around to the intent of the speaker.
Say the intent was nefarious! That intent would hit the mark sometimes, and other times it would not.
Say the intent was benign. Same outcome!
With me so far?
Before I continue, perhaps it makes sense to match tools up with speech.
For the noise, rule based, AI type systems can help. People can appeal, and the burden here is modest. Could be well distributed with reasonable outcomes more than not. Potentially a lot more.
Misinformation is a very hard problem, and one we need to work more on. People are required. AI, rule based schemes are blunt instruments at best. Total mess right now.
For the criminal speech, people are needed, and the law is invoked, or should be. The burden here is high, and may not be so well distributed, despite the cost paid by those people involved.
Offensive material overlaps with misinformation, in that rule based, and AI systems are only marginally effective, and people are required.
Now, back to why I wrote this:
Barring criminal speech, how the recipient responds is just as important as the moderation system is!
I said we are as offended as we think we are above, and here is what I mean by that:
Say a clown calls you an ass, or says your god is a false god, or the like. Could be pretty offensive stuff, right?
But when we assign a weighting of the words, just how much weight do the words of a clown carry? Not much!
And people have options. One response to the above may be to laugh as what is arguably laughable.
Another may be to ask questions to clarify intent.
Yet another option is to express righteous indignation.
Trolling, along with misinformation share something in common, and that is they tend to work best when many people respond with either righteous indignation (trolling), or passionate affirmation and or concern. (Misinformation)
Notably, how people respond has a major league impact on both the potency and effectiveness of the speech. How we respond also has a similar impact on how much of a problem the speech can be too.
There are feedback loops here that can amplify speech better left with out resonance.
A quick look at trolling can yield insight too:
The cost of trolling is low and the rewards can be super high! A good troll can cast an entire community into grave angst and do so for almost nothing, for example.
However, that same troll may come to regret they ever even thought of trying it in a different community, say one where most of its members are inoculated against trolling. How?
They understand their options. Righteous indignation is the least desirable response because it is easily amplified and is a very high reward for the troll.
Laughing them off the stage can work well.
But there is more!
I did this with a community as it was very effective:
Assign a cost to speakers who cost more than their contributions deliver value! Also, do not silence them. Daylight on the whole process can be enlightening for all involved as well as open the door for all possible options to happen.
People showed up to troll, stayed for the high value conversation and friends they ended up with.
Others left and were reluctant to try again.
The basic mechanism was to require posts conform to one or more rules to be visible. That's it.
Example costs:
No 4 letter words allowed.
Contribution must contain, "I like [something harmless]"
Contribution may not contain the letter "e".
And they have to get it right first time, and edits are evaluated each edit. Any failure renders the contribution hidden.
Both of these did not limit expression. They did impose a cost, sometimes high (no letter "e"), sometimes subtle (no four letter words)...
But what they did do was start a conversation about cost, intent, and
Sorry, it is incomplete.
- (effectively all) humans are first of all impulsive, driven by emotion and tribal identity, and most of all, subject to cognitive errors and for that reason and others, predictably effective manipulation
- the tools/practices for such manipulation, which many on HN are actively advancing because yaaay $$$, are now so powerful as to in aggregate overwhelm all current defenses (including good intentions, individual discernment, and various technical solutions ITT)
- the financial/power incentives to abuse such tools and short circuit, bypass, block, subvert, etc., attempts to defend against these tools are overwhelming
Look at the news cycle, for the 99th time the intersection of Facebook's utter sociopathology and the utter toothlessness of its sham oversight board and nominal internal protocols and standards is headline news.
I got 99 problems, and stochastic mind control is one of em
The only thing remotely effective is total deplatforming, and yeah, that's not the sharpest or most popular tool.
Left hook climate change Right hook surveillance capitalism
DOWN WE ALLLLL GOOOOOOOO
[1] https://genius.com/George-orwell-nineteen-eighty-four-append...
Your comment on the other hand seems to react (negatively, which is fine) to the central analogy, but its knee-jerk hostile phrasing is unlikely to be appreciated (or even understood) by anyone who doesn't already know what you're talking about and share your sentiments. So, if you were to phrase your sentiments in a more friendly (to the public, not necessarily to the writer of this essay) manner it could make this comments section more interesting and potentially fruitful.
It was always a terrible point, but it's especially ridiculous to see techno-utopians turn techno-fatalists in an eyeblink. The same people will go right from "innovation will save the world" to "I guess progress has now stopped utterly". And what they never grapple with is who is bearing the brunt of them giving up. I promise you it's not venture capitalist and rich guy Ben Evans who will be experiencing the bulk of bad behavior. It's easy enough for him to sacrifice the safety of others, I suppose, but to me it seems sad and hollow.
As it happens, I would suggest that the idea that somehow we CAN just stop all of bad human behaviour online would be the most extreme techno-utopianism possible.
Huh. Not totally sure you understand what "ad hominem" means.
But moving on from that, I'm not objecting to the notion that we might need different models. The way we do anything today is unlikely to be the best way for all of time. Given that I've spent years trying to improve things, perhaps you can take it as read that I think we can improve things.
What I'm objecting to is your fatalism that bad shit is probably going to happen to somebody (a note you include in your closing paragraph) combined with your failure to examine exactly who's going to bear the brunt of it. Something you conspicuously didn't do in your reply here, instead suggesting it's some sort of shocking rudeness to point out that as a rich person, it's unlikely to be you.
All utopias and utopic dreams rely too much on human nature being entirely good.
The comunist utopia relied on the goodness of the people in government. The capitalist utopia relied on the goodness of the entrepreneurs. Online communities and communities in general rely on the goodness of the members.
The reality is human nature contains both good and bad. And as a utopist, being faced with pure destructiveness like you are in content moderation is demoralizing.