The Impossible Job: Inside Facebook’s Struggle to Moderate Two Billion People | Hacker News Reader

The Impossible Job: Inside Facebook’s Struggle to Moderate Two Billion People

motherboard.vice.com

67 pointsdsr1252 comments

Facebook didn't need much moderation until it started allowing strangers to enrage each other through public comments.

My experience on FB is that 100% of the value of the platform comes from parties I choose to engage with (friends, local businesses, groups), and almost all of that value is wiped out by having to see comments from trolls.

Remember when "friends of friends" used to be a (readily accessible) setting?

Now, it's "friends" or "public".

Reflect on that, for a minute.

"Friends of friends" isn't/wasn't perfect, but it had a moderating effect.

A lot of this, is Facebook's own fault. Because greed.

I'm guessing it was a lot harder for stuff to go viral when it was only "friends of friends" vs "public"...

but i remember reading somewhere they throttle delivery of messages to friends until they can gauge feeedback.. not sure if that was confirmed or a rumor.

Didn’t privacy researchers show that “friends of friends” was misleading and effectively public because of how easy it is to become a friend of a friend ?

I can imagine this, especially with friends who "friend all over the place". My own network of friends, and most of their friends, have remained much more contained. People who want to be in touch with real friends, not "products" or random encounters or "networking".

However, did Facebook use this as an argument for their change? I've not seen them doing so.

And I'm reminded of the whole "profile pictures are now forcibly changed to be public" policy, earlier. It may help people who don't know each other much and have no other avenue of communication, to "friend" each other. (Is that the right John/Jane Smith?) But, again in my anecdotal case, real friends can communicate out-of-band which is the right profile. They're already in touch; Facebook, just allows for asynchronous group sharing and discussion (or did, once upon a "simpler" time).

The response to the "profile pictures must be public"? Those with privacy concerns no longer have a profile picture that shows them, if they ever did.

Real names, too. That one helps to align collected data with data sales and advertising.

And, for connecting less strong social connections.

I understand the desire to "connect the world", especially in its more altruistic aspect.

But, once again, Facebook quasi forced it. Whereas, social connections and sharing tend to work better when they are "opt in". Not opt out, or even no ability to opt out.

It's the passive/active question again. It might be trivial to actively become a fof but unlike public, it's not available unless you do take action. And every action is one extra filter.

What do you mean by "trolls"? Someone who disagrees with you? Like half the country is "trolls"?

And what stops you from simply unfriending these trolls, if they are so toxic?

I'm not buying that narrative at all. What I think actually happened is that groups of people always had irreconcilable differences, but these groups were mostly physically separated, because people tend to form bubbles of like-minded people. But then FB came, and everyone's opinions became visible. Half of your coworkers that you went out drinking with turned out to be Trump supporters or antivaxxers, and that just makes you and them angry.

The solution is to stop demonizing the other side, and start having rational conversations. But good luck with that.

P.s. Calling them "trolls" will only drive them further away from your politics, you're helping Trump to get reelected.

Anything short of enthusiastically agreeing with Trump someone will accuse you of being the reason Trump was elected. In one ear and out the other.

That's false. I disagree with Trump all the time, nobody ever told me I was the reason he got reelected.

I'm not going to disagree with you, because what you say is spot on.

What I am going to do is point out that hey, maybe there are people out there that troll just to troll or troll for a particular cause, and there's a lot of them. When you have a few different sides that are generally isolated from each other and already struggling to have a conversation and throw trolls and paid trolls (astroturfing) onto the same platform, you don't have a platform that is conducive to these various sides having any kind of conversation because there are groups of people that are actively working to prevent this. It doesn't even take a very strong push to keep them apart either, they were already inclined to isolate themselves in meatspace, why would cyberspace be any different?

Facebook has more than a quarter of the Earth's population on their platform. Undoubtedly this is a hard problem to solve. An authoritarian approach is probably what they are going to end up having to go with because they're not a country or a religion, they're a bloody online advertising company.

> they were already inclined to isolate themselves in meatspace, why would cyberspace be any different?

Because people generally "friend" everyone in sight. I got probably 50+ FB friends from work, the people I wouldn't call friends IRL. There's still some unhealthy obsession with the friend counts.

When I used FB, I did the bi-annual purge, unfriended everyone I hadn't talked to in months. Got some weird reactions too!

I agree with you, FB will likely implement/force the echo chambers in the end, the current open social network isn't working, and banning one side will lead to a massive loss of ad revenue.

To add to your point, nobody actually trolls in person or in real life because 1) the body language gives them away immediately and 2) they're risking getting punched in the face for their effort.

The problem with online debate is that it's too easy for people to purposefully poison a discussion that is already on shaky ground by acting in bad faith.

Usually when people on opposite sides of a discussion engage in good faith, they might not agree on everything afterwards, but it won't give them a terrible view of the opposing viewpoint and humanity in general.

Plenty of people troll in real life: streakers, hecklers, cat-callers, bullies, performance artists, pranksters, protesters, radio personalities.

The consequences for pissing people off are somewhat higher in real life, which means people tend to either troll only for causes really important to them (protesters, performance artists), or in ways that are innocuous enough they're not likely to get punched (streakers, pranksters). But if trolling is defined as "intentionally trying to get a strong emotional reaction out of people", that happens all the time. When BLM blocks an interstate, that's trolling (for a good cause IMHO, but it doesn't make drivers any less pissed). When Opie & Anthony started "Whip 'em out Wednesdays" or announced that Mayor Menino was dead, that was trolling. When people prank-call the neighbors or call in bomb threats to public places, that's trolling.

>To add to your point, nobody actually trolls in person or in real life because 1) the body language gives them away immediately and 2) they're risking getting punched in the face for their effort.

What do you count as trolling? Purposeful inflamatory remarks? Arguments that aren't honestly believed but are politely used to test the extent of some line of reasoning or the limits of a position?

For example, one common line of reasoning used with drugs and abortion is that it will happen anyways so outlawing it only makes it more dangerous. Now imagine someone taking this same argument and applying it to something like FGM or some other extreme position (though an extreme position that some on the planet do hold, just not likely the one presenting the argument). Would this be trolling, and would it depend upon presentation, in which case would someone with poor presentation presenting an argument they believe in not also be trolling?

> To add to your point, nobody actually trolls in person or in real life because 1) the body language gives them away immediately and 2) they're risking getting punched in the face for their effort.

A couple of people already got to it before me, but I wanted to add on: plenty of people do troll in meatspace.

I don't think it is quite as effective as online trolling, and certainly doesn't have the same impact to effort ratio, but it happens.

When some kid calls in a bomb threat to get out of school (happened to my Middle School post-9/11), it has an effect largely contained to that school and the immediate vicinity around it. When somebody trolls online, it has a much more subtle effect, but with enough contributors over enough time, a potentially more poisonous effect on society.

In the first instance cops can come in and clear the building discovering there is in fact, zero bombs (or alternatively, there is, but I've never personally seen that happen). In the second instance, you have a faceless group of people that are shaping and poisoning debates or preventing them from even occurring in the first place, helped by the very platforms they're trolling on. Fear, uncertainty and doubt has always been easy to spread around like so much manure, not so much with the antidotes to FUD because it is basic human nature to be cautious and skeptical of that which is different from what they know, or beyond their understanding.

In meatspace you also have more dimensions through which to interpret someone's intended message: body language, tone, choice of words, facial language, gestures, whether they can stifle their laughter and so on, and even all of that might not be enough (trolling people is easy even in meatspace, even if you're half-laughing when you do it and is a common enough flirting gesture). Cyberspace, you often only have what they wrote, and how they wrote it. It really isn't the same.

I completely disagree. I grew up as the outsider in the south - liberal ideology, atheist, etc. Through the Clinton, Bush, and Obama eras it was No Big Deal. I'd have spats with friends and neighbors, and they'd end friendly. In South Carolina I went from being called the "n-lover whiteboy" to convincing my white friends that my black friends are just as cool, and have good BBQs. We'd all disagree with eachother, but it'd be respectful. Online and off. This is when Facebook/Myspace is just coming out, and "normies" are learning about interacting online, and about how easy it is to forget about the face on the other side of the screen. Even there it wouldn't flare out too horribly.

Something's different, these last few years. Sometime around the tail end of the Obama era is when I began noticing it, that it wasn't a "political disagreement," it was a fundamental inability to reconcile values. It was "freedom of speech vs political correctness," it was "nazis vs antifa," it was "black lives matter vs all lives matter." People getting mass-defriended on facebook, me being mocked by my southern friends for my liberal values (that are the same as they've always been). Things have gotten far more divisive, and if I had to make a guess, I blame the political organizations paying trolls to disseminate divisive fake information across the internet, and prominent politicians engaging in disgusting divisive language (Duterte, Le Pen, Trump, Netanyahu, Erdoğan, ISIS high-production-videos).

The Democrats were still on "don't ask don't tell" in the Clinton era and that policy continued to 2011 so it just seems to me that the rise of this divisiveness came as a natural reaction to Progressives views on race and gender gaining mainstream traction. If you look at how overall society has changed on those issues in the last 20 years the change seems pretty big.

Maybe you're right. I wonder how much of it is non-stop political articles in our faces if we consume social media combined with the fact that we rely so little on our neighbors and friends because we don't have to so understanding and sympathy skills have plummeted.

On the latter point I wonder if lower human social contact and ease of escape online means lower tolerance towards differing opinions, or even things we deem wrong. It feels more and more like typed opinions are people shouting into the sky, and when disagreed with they get offended and lash out or they say "you're not who I was referring to," almost like that racist grandma stereotype.

I'm not sure. I remember having the realization in ~2015 that you aren't allowed to be a moderate, in ANY discussion, on reddit (back when that was the only forum I used regularly). Be the discussion about racism, police brutality, or even whether or not sourdough should be steamed when baked. You are RED or you are BLUE. And if you're BLUE, you're FULL BLUE or you're a nazi.

So like I'd come in talking about being a 3rd party voter and get shit on. Or about how there may be middle ground, on, say, gun control, and get downvoted to obvlivion.

I don't know, polarization is powerful, I guess. I mean, it seems absurd to me that the 300 million people in this country are literally split black and white down the middle on all issues. Surely not. It defies logic that 150 million all perfectly support total access to guns while also opposing all forms of abortion while also hating all forms of welfare (this definitely can't be possible because shitloads of republican voters are on welfare). So, something's up. My guess is that it behooves those in power to keep the population divided? No idea.

The risk with this is that eventually it will erode Facebook's legal argument that it isn't responsible for the content that its users post to the network or share through Facebook Groups.

As it is there are tens of thousands or even more Facebook groups dedicated to some kind of either illegal, rights-infringing, or grey market activity in the US alone. The more that Facebook holds itself responsible for moderating and controlling that content, the less weight that Facebook's arguments that it is just a dumb pipe will hold. This has been going under-remarked relating to this cluster of issues.

Strongly doubt that FB or any similar company will successfully navigate this issue in the long run.

No company should be liable for the content users post, regardless of moderation policies.

They definitely screwed up how they handled the crises. Zuck & Sandberg's apology tour has made them responsible for any future liability or slip ups which are inevitable given the size of operation.

There were 4 separate issues

1. People posting fake news

2. Recommandation serving fake news due to similarity

3. Apps stealing data

4. Advertisers able to target vulnerable audience

They should have kept these issue absolutely separate and taking responsibility for all but #1. They should have stayed very firm on not accepting blame for what people decide to share on their platform. Instead they ended up accepting blame for everything and now they are screwed.

Right. This is a giant multi-billion dollar target that is increasingly portrayed as a machine for the mass production of liability. People seem to be under the impression that internet companies putting themselves under the cloak of free speech came purely from 'idealism.'

The real reason is to avoid being Napster'd. How can FB turn around to a copyright holder and say "Ha ha, this content is totally not something we take responsibility for -- you should take it up with the user" if it is actively policing content on the platform with an army of staffers and actively shaping what people can and cannot write on the platform? That looks a lot more like editorial behavior than 'moderation'.

It's different when it's just automated systems that are dinging people for potentially posting copyrighted material or a snuff film. The system doesn't exercise human judgment on an active, day-to-day basis. To the extent that services like FB took a hands-off approach had much more to do with avoiding liability than any iron moral commitment to allow people to share flat earth and lizardman related content.

Why do social networks have to be so big and centralized?

Because we don’t have good open source software like Wordpress for blogs that people can install and manage their own networks on their own servers. And why? Because a social network is far more complex than a blog, including contacts integration, permissions, realtime notifications and so on.

https://m.youtube.com/watch?v=pZ1O_gmPneI

Notabug.io is doing good work in this area. It's a reddit-alike with a distributed architecture. It has federated nodes hosting the website but also the in-browser client distributes posts p2p between nodes in the federation. Additionally, voting is proof of work based.

It's the early days now and the lens/filter system for individual users hasn't rolled out yet but the idea and implementation are far ahead of anything else I know about.

Because of the 'network effect', networks are more powerful when they are larger.

> The Impossible Job

Leads me to two possible conclusions

1) Facebook gets a free pass to ignore the law because 2bn members requires x million moderaters and that doesn't make economic sense, is hard etc.

2) Facebook has to abide by the law, and since it doesn't make economic sense...Facebook closes.

The question is whether Facebook is a communications company or a media company. Are they like GMail, where they aren't responsible for everything that goes through their pipes, or are they like Vice, where they clearly are?

Facebook obviously wants the publicity of a media company and the legal protections of a communications company. Which is obviously stupid, but that's the weird situation we're in.

Facebook also obviously thinks of themselves as a tech company first and foremost, so the question of moderating and scaling moderating is considered almost solely from the standpoint of it being a technical problem.

The problem is a social and human one. Newspapers have traditionally handled it with editors and people and human policies. Facebook isn't inclined to explore editors and people and human policies, not because they don't scale or we don't know how to scale them (Wikipedia has scaled relatively okay, for what that is worth as an example; Newspapers have used editors and researchers for a long time), but that it's not an interesting technical problem to scale people.

Facebook even had people hired in some of these moderation roles for a while, but it was sexier for them to use essentially blackbox algorithms that turned out were easy to game by malevolent entities. Now they only seem to question "fixing" the blackboxes rather than finding solutions that use smart or trained people.

Which law are you referring to?

dont shoot the messenger

>For example, its revenge porn policy and recently created software tool—which asks people to preemptively upload nude photos of themselves so that they can be used to block anyone from uploading that photo in the future—was widely mocked.

This is hilarious. I understand the logic, but to believe that users will willingly submit nudes as a preemptive measure seems to widely underestimate the ramification of the user's decision-making process, and the reliability of Facebook when it comes to user privacy protection. Hey, let me upload my nudes to a Facebook software, they've been handling privacy really well so far, said no one ever.

Part of the problem is Facebook doesn’t want anyone to have a negative experience because those people disconnect, at least temporarily.

I can go on Facebook, praise Hitler, and get likes. I think that skews peoples’ perception of what’s acceptable and what’s not. Maybe Facebook needs a way to give negative feedback.

It’s such a hard problem though. The quantity over quality friend groups everyone has now makes it difficult to trust others. When I was growing up, if one of my good friends disagreed with me it was a good indication I needed to reevaluate my opinion. I think that’s been lost.

I’m not sure that is really facebooks problem. If I saw you praising Hitler, I’d unfriend you, so it’s not like there isn’t social consequences on Facebook.

I think the real issue is the users. I mean, Facebook makes it easier to find people who are like you and believe the things you do, but Facebook didn’t make you believe the earth was flat.

The fact that people believe the earth is flat shouldn’t be blamed on social media, because it’s a much bigger problem than that. Those people have managed to grow up wrong, with terrible education and horrible life choices, and maybe they find each other in groups on Facebook but they sure weren’t made this way by the internet alone.

I think Facebook has some responsibility because they're such a large platform. The problem is constant positive reinforcement of everything, even things that are unacceptable. Part of the reason those people grow up "wrong" is because they're growing up on social media where the feedback they get is unnaturally skewed to agreed with them.

I don't think it's intentional, but I think it's going to be impossible for Facebook to solve with watered down feedback like the "empathy" button. They need a way for people to express disagreement, but it's going to be difficult to do that without reducing engagement, so it probably won't happen.

Nobody believed all this nonsense before Facebook/Youtube, dumb people had the good sense to defer to others. This is absolutely new and caused by these tech conglomerates, it is not inherent in people.

We optimized the hell out of user engagement, and created these closed loops where you just swim in nonsense for years. The marketing teams loves it, shareholders love it. Nothing foreign or unpleasant penetrates the engagement loop. Keep growing those numbers. What's the content? Hmm? who cares?

This issue doesn’t depend on conglomerates (unless you’re implicating all companies developing the Internet); it’s going to happen when these people get online whether it’s on Facebook or incel forums. It’s a very unfortunate side effect of global peer-to-peer communication as enabled by the Internet.

If Facebook's revenue was not tied to keeping people accessing the service enough to see ads to prop up the stock price, then they would have more options. Maybe they should consider charging instead of serving ads. But it would likely reduce their overall revenue.

Think about the scale of 2.2B users. Imagine how difficult it would be to moderate just America. Now imagine handling 6.8 x America. Or 1.6 x China or India. That is a massive undertaking for any centralized entity. Probably bigger than many countries national security efforts to feasibly implement. Is it really worth it to Facebook when their primary concern as a corporation is their bottom line?

In which case the argument becomes

"If a platform/service provided by a corporation becomes large enough to credibly threaten the fabric of society and/or government, and said corporation won't tamp down on said problems, should they be allowed to exist?"

The impossibility is not just a problem of scale.

Even if Facebook does everything right, and moderates every single thing they aim to moderate, they will still be blamed by conservatives and/or the alt-right if they censor too much, and by liberals if they censor too little. There is no type of moderation that will make all sides happy.

Wait, so the conservatives want less censorship and the liberals want more censorship?

Is this one of those mirror universe things?

To be fair it has flipped considerably, if you consider the extremists in each group.

In the 2000s the extremist conservatives wanted to censor everything (mostly due to religion) and liberals wanted freedom of expression.

Now it seems that extreme liberals want to censor "hate speech" which is absolutely valid, however the lines are moving or getting blurry to the point where they want to censor conservative ideas altogether, and conservatives want "freedom of expression".

I've seen this shift and I'm sure many others have as well.

Impossible only because they have underpriced the service. The notion that only good things will happen when you create a context and let humans interact is at best naive. In this case, I think it's willfully blind. I was talking recently with people who set up the first web-based chat; abuse was almost instant and they what we'd now call a doxxing within the first year. And they were hardly the first people to see abuse happen online.

In the real world, if you want to have something as anodyne as a parade, you have to provide for sufficient security to keep things safe. There's no reason that companies pocketing billions in profits should be exempt from this principle.

The thing that makes it hard is differing cultural standards between parts of the world. You either acknowledge them and push moderation decisions down as far as possible to local authorities, users, and group moderators or (what they seem to be doing) build an authoritarian, culturally-imperialistic global AI censorship algorithm that makes people mad and (hopefully) causes them to leave.

The smartest engineers in the world have been figuring out how to make people click on ads. If they have to figure out how to quell genocidal movements I think that's an improvement.

Not impossible, impossible for hoards of humans.

This is going to be a major use-case for AI, which can scale to billions of people.

The real solution is probably hybrid (i.e. AI-assisted human moderation).

I've assisted with moderation for a bunch of communities over the years, and IMO the problem isn't that there's not enough humans — for every bad actor in a community there are many good ones — but that Facebook doesn't provide good actors and moderators with anything beyond the most basic tools imaginable. Had Facebook invested even 1% of their resources into this, I don't believe that Facebook would be the raging culture infection that it currently is.

It would be awesome if AI could handle the worst cases. From what i've read moderating the worst content taxes humans emotionally pretty heavily. Many have ended up in therapy because of it.

So I've never managed a subreddit or a Facebook page/group. I once managed a forum where my options were to Delete and Edit comments, but I only used Delete and would message the user why, it was a small forum though so if it were say, 3x or 5x larger I'm not entirely sure I could have moderated the entire thing by myself.

I'm explaining my ignorance on this subject matter because I want to ask you, and you seem like someone who might know, how do Reddit's and Facebook's tools compare? From my point of view, Reddit is pretty well moderated, each sub having its own rules that at least in the subs I frequent, are actually fairly well moderated. They have tools like automoderator and I've heard of subs I don't frequent preemptively banning users for having too much karma in subs they dislike. Probably overkill, but given the types of communities they are it seems to work for their purposes regardless of whether I agree with them or not.

If I look at a comment thread on Facebook, there doesn't really seem to be any kind of conversation happening, just "conversation". There's some basic threading now but even with that a Facebook "conversation" seems to just be one long string of human consciousness.

Now there is a difference in scale, Facebook has over a quarter of the Earth's population on its platform. That said, there is also a difference in scale between Facebook's resources and Reddit's, so is there some reason they choose not to develop new and better mod tools? Or are their mod tools cutting edge already and the percentage of humanity on their platform is just so large that even with cutting edge mod tools it just isn't enough and throwing more money at it doesn't make it any less hard?

Reddit gives the masses downvotes, but those don't get used for moderation, they get used to make the echo chamber more impenetrable.

Pretty sure it's impossible.

You're assuming there is a global maxima in line with the local maxima. I personally don't believe its the case. Moderation practices across one group will not make sense to another, but when you try to moderate each group individually, said groups will point out inconsistencies between moderation in each group.

For example a user with 2 accounts

Group A is moderated for doing x

Group B is not moderated for doing x

Therefor the user believes the system is unfair, and will point that out to others spreading the idea of unfairness in the moderation system.

Reply on news.ycombinator.com