The trauma of Facebook’s content moderators
restofworld.org
restofworld.org
This quote from the article
“It gets to a point where you can eat your lunch while watching a video of someone dying. … But at the end of the day, you still have to be human.”
reminded me of a similar article published by WIRED six years ago [1].
Eight years after the fact, Jake Swearingen can still recall the video that made him quit. He was 24 years old and between jobs in the Bay Area when he got a gig as a moderator for a then-new startup called VideoEgg. Three days in, a video of an apparent beheading came across his queue.
“Oh fuck! I've got a beheading!” he blurted out. A slightly older colleague in a black hoodie casually turned around in his chair. “Oh,” he said, “which one?” At that moment Swearingen decided he did not want to become a connoisseur of beheading videos. “I didn't want to look back and say I became so blasé to watching people have these really horrible things happen to them that I'm ironic or jokey about it.”
Most of their workload is bullshit. People with colds and sore throats, depressing people using ER as primary care, assholes using 911 to score Medicaid cab vouchers.
And at any time, any number of people can show up with any kind of personal tragedies from stokes to various traumas. My sister quit after a 13 year old bled out from a gsw, and she walked out of the room and got kicked in the head by an prisoner who had been stabbed after he bit the ears off of three other prisoners, and broke the arm of a guard.
Bite a 2nd inmate's ear...shame on him.
Bite a 3rd inmate's ear...that's fricking badass.
That said, I would also wager that the curve can be skewed towards one side or the other depending on the location, timing of your shift, or any other factors.
Let’s say a content moderator got paid $200k per year, that doesn’t change the images they see. This isn’t about money as much as it’s about the requirements and preparation required for the job.
The root of the problem, which is rarely mentioned in these discussions, is that many people enjoying comfortable living conditions believe themselves to be “pure-blooded” and unable to deal with “unclean” things because of the possible “traumas”. (What is an ideal person free of “traumas”, anyway? A vegetable stoned on drugs?) You can easily see here the resemblance of a old time countess complaining about her “nerves” while her serfs do all the dirty work. Is there a natural law that says a poor grunt is bound to clean the junk from what you see on the website? Do it yourself.
The same happens with medical, law enforcement, and other professions. They are considered “special” jobs for “special” people (which is not true, the one who defines it like that is “special”). Then, any worries and doubts are neutralized by convenient “but it's their profession, they chose it, they are paid for it, etc.” This creates a stupid rift between “us” and “them”, and affects how both sides reason about themselves.
The resulting imbalance is routinely abused. Social networks are competing in presenting a rose-colored walled fantasy to their human cattle to increase yield — everyone can see they use “The Matrix” as a business plan. This creates a lot of shitty jobs where people waste their emotional capacity to make others pleased, not unlike in prostitution. On the other hand, media is hooking the softened consumer with the sensationalist outrage bait about maniacs, wars, corruption, presumed orgies, etc. To makes things worse, the limits to all of that are set not by some moral standards or concern about someone's well-being, as it is often implied, but by advertisement contracts. The bureaucratic machine decides what is “suitable” for the visitors. For example, this makes most of the talk around “Free The Nipple” moot: there is no undercover agents of patriarchy to battle, it's advertisers who bring money and judge what is unwanted “sexual content” (some might say it's a perfect example of capitalist oppression — cue “L'Internationale”).
Even this discussion is full of overreacting people. No, the video on your screen is not as “horrible, horrible” as experiencing something firsthand, nor the pain of the tortured can be compared to the pain of the random witnesses on the internet (if we don't count their own wild fantasies). If it happens, then it's no good to pretend it's not. Who knows, it may even make you think. All in all, humans don't show a lot of originality in what they produce, so the number of things you can expect to see is quite limited. I can only conclude by reiterating that knowledge is power, ignorance is bliss, and there's a lot of people who'd like to benefit from others' ignorance in many ways.
(Note how much the "thin blue line" imagery is used in conjunction with this - that's how you know who's the target audience.)
It all ultimately comes from Dave Grossman, who's one of the most worshipped figures in modern American policing: https://www.policeone.com/police-products/training-products/...
This guy now runs the "Grossman Academy", which offers police training. It's popular enough that they have train-your-entire-PD discounts.
Oh I'm sure FB could think of a few ways.
A decent compensation structure might be a good place to start.
I still maintain expecting companies to do this on their own is a category error.
Companies think moderation is a cost center. Even if the moderators do the best possible job, they aren't adding value, only removing liabilities. The company is the wrong entity we should be asking for solutions to this problem. They are going to minimize costs and shift liabilities to 3rd parties (which will in turn hire independent contractors and play shell games in case they are sued).
We have to demand adversarial outside actors (government, labor unions, lawyers, non-profits, mental health professionals, etc) to start shaping the direction of the solution.
Modest proposal: Find people who enjoy horrific content, and have them rate it. Then black-hole whatever they rate as most extreme.
I was is a dark place after 9/11, both from the trauma of that, and some personal issues. So I spent considerable time on Rotten, Something Awful, and worse. Based on that experience, many would likely do content moderation for free.
Thanks for sharing that part about being in that kind of place, I have a feeling many could relate.
These media stories always focus on those who picked the wrong job for their caracter, and then make a big drama of it.
I mean, there are people cutting open other humans daily. We call them surgeons. I would puke right there and could never do that. Different people have different abilities. Simple.
Yup - these are exactly the kind of people you want to rub shoulders, and stock your company's ranks with.
I also get the downside of employing them. But people need to look at that stuff, even if software does most of the work. So it might as well be people who can tolerate it, and perhaps even enjoy it.
It's obviously horrible doing this as a job and it's obviously a bit more common on Facebook than say, a large old school internet forum, but... at least in this case people are paid to do this. Reddit mods and old school forum mods have to deal with this stuff for free.
I can go door to door asking people to change something, and dozens of my volunteers will feel useful but overwhelmed by how much work there is to do in a city of a million people. Or I can go to city council meetings, and get more leverage there. Or if I can convince one high profile citizen to participate, others will follow. Or I can get into contact with the mayor, state congressional members, the governor, federal congressional members.
It's possible that if I convince the correct five people that they will take half of this burden off of my hands. You'd be crazy not to reach for a lever like that. They are a juicy target, and not just from a punitive standpoint. Of course when you are rebuffed repeatedly by a potential ally, then it's a very human (if questionable) response to seek out some sort of punishment.
edit: What I mean to say is, people have been organizing against FB, AMZN, and tech in general for the past few years now. The last is reactionary but opposition to FB and AMZN has meaningful academia, ngo/gov, and general support.
But... that group wouldn't be able to do nearly what it has if it weren't for funding from foundations and the department of neighborhoods, and resources on permanent loan from another public sector organization. You might argue it couldn't exist at all.
I think I would say that both are necessary but insufficient.
edit: https://qz.com/1704143/the-antitrust-case-against-facebook/
I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health).
That's an interesting point, and has me wondering something.
If a site like Facebook opened up a volunteer moderator scheme where people could sign up to filter out stuff that broke their rules, would it make things better or worse on a moral level?
What about a US/Europe based agency that did this thing onshore as a contract deal?
https://www.theverge.com/2019/2/25/18229714/cognizant-facebo...
As for the volunteer side, it's a really interesting debate. Others in this thread have pointed out Reddit as an example of this in practise. I think the sheer amount of money that Facebook has means that relying on volunteers to (potentially) inflict emotional damage upon themselves wouldn't be a great look.
I think they tend to take an "if in doubt, remove it" approach which basically allows anyone who doesn't like a niche page for whatever reason to have it taken down through sheer attrition, if you report it enough it'll probably go down eventually even for stupid reasons. I actually ended up writing an app to cut Facebook out of our system altogether (it was kind of similar to Yik Yak with a much better moderation system and better anonymity protection - I really can't be arsed to deal with a GDPR lawsuit...) but we couldn't launch it in part because it took me ages to develop as I've since graduated and work as a full-time developer and because coronavirus meant our launch event got cancelled. I might take it live when students go back this year, but I'm concerned that with nothing much going on it'll fizzle and die because there's nothing interesting to post about. It'll all depend on how the Welsh government lift the coronavirus restrictions.
Disclosure: I work for but don't represent Google. This is my own opinion.
I've done commercial moderation before and I would sooner go jobless than go back to exposing myself to the kinds of things people have to when doing this professionally. It's the kind of things you wish there was a brain cleaning solution for, but there isn't so it's stuck there basically forever (though I suspect the alcoholism rate is higher among people who have to handle this kind of thing than in the general population).
The commercial moderation team I was on actually ended up separating out a task force for handling particularly gnarly content so that people had a way of opting out of being exposed to it (in our case we were responding to user reports rather than randomly stumbling on things, so manually filtering reports was feasible), which I think is the responsible approach since there are plenty of other moderation tasks to handle and some people are more or less sensitive to this content.
'Drug-induced amnesia' to reduce long term memory generation is a thing and is sometimes used before traumatic surgical procedures. It needs to be taken in advance and is probably not very healthy. Plus I would have serious ethical concerns with jobs that would encourage that.
Just out of college in the mid-90s, I worked for a few months at a small local software company in the last throws of death. As a last grasp at survival, all the programmers switched to doing cold call sales. With phones.
I would stare at mutilated children all day long rather than go back to cold calling. I was miserable and I hated every moment of every day until the company (happily, for me) closed up. Yet I know people who somehow think it's perfectly normal to dial random numbers and try to pitch them something. They're made of different stuff.
I've talked to people who make the same claim about computer programming. They'd rather starve.
Facebook has 10,000 moderators. I'm pretty sure there are at least 10k people in the world who are psychologically fit for content policing. It's doesn't have to be for everyone.
There exist jobs, like slaughterhouse knockers, which are linked to causing PTSD and increased criminal behavior. It's not far fetched to believe these moderator jobs do too.
It wasn't that long ago HN was abuzz about large numbers of suicides at Foxconn... until someone realized that the suicide rate was actually slightly below the community average. I simply don't trust arguments without statistics anymore.
Big companies like Facebook (although they are not alone) don't do that. That costs money. They use disposable workers, like on-shore or off-shore contractors, and do industrial-style quotas/piecework methods to maximize productivity.
When you hire like that, you'll have a normal distribution of people. On the left there will be pre-broken people who get pleasure from watching decapitation videos and nasty shit, on the right you'll have people who are traumatized and broken by the work, and in the middle you have people who will cope for some period of time, and burn out.
I don't think many people are immune to PTSD, but I suppose sociopaths need jobs too. But I'm not sure you want to hire a department of sociopaths.
Why not, if they do the work well?
And anyway, they'd be working from home, these days.
My point is, some people simply aren’t traumatized by photos/videos, either via talent or training, and because of this they can process it enough to categorize it and keep their emotional/social/intellectual distance. That doesn’t make them sociopaths and it certainly doesn’t make them bad people.
A bleeding anus is just a thing that happens to people. It's a bit gross but I don't know why anyone should be traumatized by it.
The kind of content that these moderators have to deal with goes well beyond the mere abject.
Fixing a sewer is gross. Extermination is gross, not traumatic, unless you have extreme empathy for small rodents and insects. Doctors deal with very few remotely bleeding anuses on the order of one a week/month maybe versus one a minute. None of these should be called real sickos, but none would be expected to develop PTSD either.
A significantly bigger portion of the population would be traumatized and develop PTSD by the level of exposure of content moderation. While not nearly all are sociopaths, they 100% will be represented higher even without factoring in selection bias. Hiring a department of them would not be inconceivable, putting the validity of that aside.
You also make the jump from sociopath to bad person, which many I'm sure would have some qualms with as a categorical.
PTSD is real but lots of people still go to war and not everyone comes back a basketcase. Most don't, in fact.
No, I've never had a job watching pictures of mutilated children all day. But like a lot of naturally curious people, I've searched out and watched bloody videos of beheadings and car accidents and whatnot. It wasn't pleasant, but it wasn't really traumatic either. If I didn't already have a much more lucrative career, I don't think I'd suffer too much as a Facebook censor.
Does that make me a sociopath? My wife and son don't think so, nor do my close friends. I'm not sure if I should feel a little offended here.
On the other hand, I may still have PTSD from those days of cold calling as a junior software engineer 25 years ago. I hate phones. Maybe people are just wired differently.
It sounds too simple to work, but worth a shot.
I don't think at the same level. Social media usage went from 10% of the population to 79% from 2005 to 2015 and was even lower a few years before this. So current social media companies operate at unprecedented scales. Moreover, the smaller percentage of people usage social media in earlier times tended to be "sophisticated users" who didn't necessarily expect their feed to be systematically filtered.
https://www.pewresearch.org/internet/2015/10/08/social-netwo...
There are dedicated communities to all these things.
I do not mean this as a hit against Reddit users. It is just that there are small pockets of people who are psychologically capable of tolerating this or even enjoy it. It seems more ethical to hire people from /r/watchpeopledie.
Edit: and saying things like "well clearly these people are already traumatized if they're seeking this out, it's not good for them, we shouldn't give these jobs to them" seems laughably paternalistic when the alternative is to traumatize more people by making them look at it instead.
That they seek it out doesn't mean they're not messed up.
Maybe it helps dealing with that trauma. Maybe it makes the trauma worse. Maybe there was no trauma to begin with, and the person was seeking out that content simply due to morbid curiosity, but then it evolved into something else. Maybe there was no trauma to begin with and no trauma appeared after the exposure. Who knows.
But just because there are communities on reddit with people seeking that kind of material and not going on crazy violent sprees doesn't mean that it doesn't negatively affect them. It could be the case, but we should probably look into that first before jumping to "obvious solution" claims.
tl;dr: we gotta do more research and weigh pros and cons before simply declaring "we have an obvious solution, let's just hire all those gore content enthusiasts to moderate violent content, because they already do it on their own and don't seem to be going crazy in obvious ways due to it".
People come to their choices with all manner of existing problems and conflicts, and they don't always do things in their own best interest.
Human are so diverse.
Is is really hard to believe that there are people that are not negatively affected by this ?
That would do wonders for the Facebook brand.
Look at how much trouble they got in when it was revealed they were using Onavo (even before acquisition).
Obviously not everyone, but they are there.
My basis for thinking this is I've seen videos like that before and clicked away quickly. It's disturbing, but I think I'm okay.
It's a strange conundrum because ideally you want good people who are comfortable watching things good people rarely want to watch.
Don't have much experience with this demographic -- maybe there are more sane, emotionally healthy people watching snuff films for fun than I instinctively expect.
It'd be like having a jury pool made up of serial killers.
These kinds of bottom of the barrel places of employment tend to have their policies and procedures tuned to make sure employees do things the way the company wants them done. That's just the nature of doing an unskilled, non-laborious job for a BigCo. There will be automatic spot checks, audits, reviews, etc. It is no different than the cube farm on the next floor where they review images of documents that AI systems have flagged as possibly fraudulent. Such systems are likely in place already for their current moderators.
Yes, but their hiring pool tends to be "everyone".
Those procedures might not work as well if your hiring pool is explicitly confined to "the people who watch ISIS torture videos for fun". I suspect standard policies and procedures include "fire the people who get off on this stuff".
They have little power to abuse and obvious accountability. If they block kitten and baby pics and allow people being tortured to death they'll get fired quickly after the inevitable shitstorm.
Insane and unhealthy or creepy doesn't neccessarily mean dangerous. Even if they got twisted pleasure from it they would be gross but not inflicting actual harm. It isn't like people are being brutally murdered to give them kicks - they aren't a source of demand.
I found most of the videos less offensive than extremely graphic horror movies, which ironically I have less stomach for. If anything that subreddit made me generally more aware of my surroundings and safer in my daily life.
Probably not, since there are no documented instances of any snuff films ever found.
Regardless my point was just that in my experience many would consider them 'snuff' even if it doesn't meet the exact technical definition.
And those people would simply be wrong.
I mean, let’s back up a bit. Why is it important whether or not “snuff” films exist? Because many people want to believe the common urban legends about it – i.e. that there exists an (be it ever so small) underground production and market for it, even though, by all accounts, this does not exist, and has never existed. Believing such urban legends helps people feel justifiably horrified and helps validate their cynicism; basically the same reason why many people believe many other urban legends. However, no movies fitting that particular model have ever been found to exist.
If anyone wants to to be picky about the definition and claim that some movie or other fits the definition of “snuff”, or is “considered” to fit even though it doesn’t(!), this is only because they want to be able to say “see, snuff films exist!”, and thereby validate their preexisting belief in the urban legend. But this is a motte-and-bailey fallacy; it tries to defend the urban legend (which is hard) by trying to defend a technical definition of a word to be slightly wider than it really is (which is comparitively easier). Luckily, even though it is easier to defend, this latter position is still demonstrably wrong.
Thanks to Wikileaks publishing the Dutroux Dossier in 2008, we know hundreds of snuff films were recovered from the notorious pedo-rapist Marc Dutroux.
Just because the police keep those films as sealed evidence that are never published does NOT mean snuff films are an Urban Legend. I can see why the police always keep snuff films secret. Imagine what would happen if snuff films were uploaded all over the Internet? There would be mobs of angry villagers armed with torches and pitchforks descending on jails and prisons to lynch pedos. It would make harassment from QANONs look like school yard bullying.
(The films were, according to your reference, made only for blackmail purposes, and therefore certainly not made for enjoyment of any viewer, nor were they ever distributed to other people for their enjoyment. The reference does claim that people were killed in a specific place which was also filmed, but does not, what I can see, explicitly state that any murders were actually filmed, other than incorrectly calling all the films “snuff” films.)
For example, instead of showing one person torturing another person, a DL filter could turn the scene into two teddy bears torturing eachother.
I wouldn't assume "watches this stuff" and "is psychologically capable of tolerating this stuff" are one and the same. Those people might be actively harming themselves by watching the content. That's one thing when they're volunteering to do so (though that's an interesting conversation in and of itself) but it's quite another when Facebook is paying them to do it.
One day I had an anxiety attack because I had a dream that I had a heart attack and suddenly I had a deep fear of randomly dying.
All the random deaths on that sub started to really get to me, I even started to get depressed. It became harder and harder to watch content on it. Eventually I unsubbed from it and gradually bounced back to normal.
I don't really regret it because it gave me a new perspective on life and how valuable it is. But I would be lying if I said it didn't fuck me up in a way.
take ER doctors. they daily witness human trauma far worse than viewing horror show videos and images. do you ever hear about ER docs who become depressed or mentally ill from being exposed to so much horrific human suffering? Doctors have a personal sense of doing good in the world through healing. They witness so much trauma that they become so desensitized that non doctors often criticize them for seeming so detached and for treating human bodies like mechanistic processes. But doctors have to be somewhat detached in order to do their jobs well.
It's not specifically about ER doctors. Anecdotally, I'm a lawyer, and they warned us in law school that lawyers have abnormally high rates of substance abuse and depression, and constant jokes to the contrary not withstanding, most lawyers believe they do good. Emotionally hard jobs are hard, even if they are indisputably good.
Yes you do! But I would imagine it is at least somewhat offset by the fact that they are doing a lot of good: they are saving people from death while also witnessing death. I think a content moderator would struggle to feel such a strong sense of purpose.
Serious question, I don't mean this facetiously. I think is an extremely complex question and I wonder if introducing someone who is openly comfortable with gore is dangerous to the psychological health of the company.
For example, at a previous company a co-worker publicly shared, without a trigger warning and in great detail, a very gorey thing he enjoyed watching to relax. Lots of people were extremely disturbed by this -- not disturbed by him, per se, but disturbed that he shared this without any kind of warning. I don't even resent him despite myself being pretty disturbed because what one considers normal is subjective, and if you regularly relax to this content you probably don't realise that this might not be anyone else's cup of tea. But it really makes me wonder -- what if the guy who sits next to me starts telling me that ISIS beheadings are relaxing to him?
It seems like if you acknowledge that: a) Terrible job exists b) Terrible job is necessary
You should also allow for a person to do terrible job without thinking that they must be a terrible person by extension. Otherwise you continue the well-trodden history of certain professions like hide-makers, executioners, or coroners being ostracized from society for doing the job we told them to do.
I agree that I wouldn't personally spend time with said person, but my question was less about personal social responsibilities outside of work. My example I pointed out happened at work, during normal work hours, in a social space where people have inherently less control over who they surround themselves with. This is the part of it all that is fuzzy to me, and I don't have a clear answer for.
And while a lot of content moderators do focus purely on clicking thru things, a lot of other ones still regularly interface with the company. I used to work at Discord, where content moderation of this level (and traumatic nature) was a constant concern. As engineers you may be staffed to work on a tool to help with moderation. Or you may be working on some fancy AI to help sort and tag unsafe content. And this is only from the product engineering perspective; content moderators will have to work with customer success to create content policy, or enforce bans for unsavory behavior. So I don't think they really are siloable away from the rest of the company, and I'm not even certain that that is a humane way to treat them.
The entire point of this is to find those who deviate enough from normal to not be as impacted.
Desensitization is certainly a thing. The first few times seeing those things were shocking and disturbing. Then it got exciting. Then it got boring. I don't actively look for that content, but never having left 4chan, I still sometimes stumble upon it and it causes no big reaction. I think I could totally do that content moderation job, I have to admit some morbid curiosity about what I would stumble upon exists in me. The only content that never stopped to disturb me was violence against animals.
Talking with my "normie" friends about this I realized I am the odd one out. I wonder just how much consuming that content in my youth changed me.
Same for Twitter, and for Reddit, and Instagram... and probably TikTok.
I don’t believe the use of these platforms can be considered ethical.
What specifically is the reason you consider those particular platforms to be unethical, and what is the solution.
It's not enough to say "social media bad". You can say that about anything including the internet as a whole. We need the reasons it's bad and solutions.
These platforms are so large that they require hundreds of thousands of moderators in order to remove vile, illegal content, causing trauma for a vast underclass of foreign workers.
Here’s a few solutions:
Limit the number of people that a person can be friends with to 150, the Dunbar number.
Restrict the posting of photos to those which have been identified as having you or your friends in the picture, or no humans at all. Require permission from all human participants to post.
Charge a fee for access.
Is it just that you're worried about the outsourcing of content moderation? Because other than than the problems are the same. Someone will always need to moderate the content.
My concern is not primarily with the outsourcing of moderation, but with the type of moderation that is required. There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both. Of course, these are two suggestions which are antithetical to the ad-riddle, growth-driven social network model, so there is no way they would ever be implemented.
I should add another solution to my wish-list: remove ads entirely.
But the problem doesn't go away when you shrink the size of the social media site. The same people will still try to post disturbing content. In fact, you're giving them more places to post that content, so potentially more people will have to moderate it.
> There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both.
Your suggestions can be implemented on smaller websites as well as larger ones.
It seems like your issue is with the ease of access to social media. Thus your solution is to limit access. However, I don't think there's a reasonable implementation of your idea that could work. Nor do I think any kind of social media is going to want to go that route. It doesn't matter if the site is funded by ads or is trying to grow. The point of these sites is to share content. People don't want to limit that.
Also, breaking large social media sites into smaller ones makes it much more difficult to deal with troublesome individuals. Right now, it only takes a few bans before you're blocked from most mainstream sites. If we had, say, 10× the number of sites, that's 10× the number of bans required to get these individuals out of the system and 10× the number of moderators who had to look at their content before banning them.
I'm not saying there's no solution here, but I think people are misidentifying the problem. We all want simple answers and simple solutions even when there aren't any.
I’m not sure I follow. A site like Facebook already has mechanisms for detecting multiple accounts, and in any case, multiple accounts don’t mean that there are more “places” to post. It’s still Facebook.
I don’t propose breaking Facebook into smaller websites. I propose limiting the reach of a single individual on those sites. Basically, normalizing the localness of social media to be more meaningful.
As well, I think Facebook and other platforms need to reckon with the consent when it comes to posting images of people, in general. That’s why I suggested limiting the posts to only those of friends who consent.
I agree that no one wants this.
And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
Cancer patients don't want chemo, but it's better than dying, some might say.
>Other sites like Hacker News, Reddit, youtube, and various other forums aren't even designed with the concept of friends that are the sole consumers of your content.
Right, which is why those sites (besides HackerNews) would require slightly different solutions.
Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes. Verify accounts. Limit posting.
YouTube: Not sure. This one is video-forward, probably the most difficult problem in terms of bad content. Definitely remove the algorithm for targeting people based on interest, though.
>That's not a social media thing. That's just an internet thing where most things are visible to everyone.
This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information. Pre-social media, to get your idea out you had to build a website. There was friction. The earl y web had little moderation because you really had to go searching for bad stuff.
>And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
Dividing the problem up is a strategy that I propose lessens the impact to both the users (because content can't spread as fast) and the moderators (because there will be less content to moderate).
Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
What good is chemo if your patients refuse it?
> Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes.
This doesn't stop people from posting disturbing content, and it destroys reddit's entire model. This solution doesn't work.
> Verify accounts.
In what way? Do you want to abolish anonymity? Because I can tell you right now that most of us aren't interested in such a solution.
> Limit posting.
In what way? Throttling posting speed? Sure if you just lessen the amount of content overall there will also be less disturbing content... but there's still disturbing content that needs to be moderated.
> This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information.
The default state of the internet is publicly viewable information. Thus it's an inherent feature that you can access almost anything out there.
> Pre-social media
As far as I'm aware, there was no such time. There have almost always been bulletin boards, forums, chat rooms, image/file sharing sites, and other forms of social media.
> there will be less content to moderate
Dividing up the content doesn't reduce the amount of it that needs to be moderated any more than cutting a cake reduces the amount of cake.
> Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
This just shuts people out according to income bracket. It reduces the total population, but I doubt it helps reduce the fraction of that population that are interested in posting and sharing disturbing content. It's just throwing the baby out with the bath water.
I don't envy the work that Dan and Scott have to do in the slightest, but I don't think they'll end up with PTSD from it. At least, that's what I gathered when I read "The Lonely Work of Moderating Hacker News"[0], especially this description of it: "Pressed to describe Hacker News, they do so by means of extravagant, sometimes tender metaphors: the site is a “social ecosystem,” a “hall of mirrors,” a “public park or garden,” a “fractal tree."
have you ever seen foxes laughing? they sound like babies. without social media, i never would have known that, nor experienced one minute of happiness while watching it.
I agree with that. But I also assert that people have an ethical obligation to opt-out of systems that cause harm when / if they are able, or at least advocate to change them.
I don't have the ability to change Facebook. But I have the ability to opt out of it, though my actions on the Internet, and the ability to advocate for others to do the same. So, here I am.
I did not like the movie much to be honest.
Not dumb, but I do think it's not thought-through. You're proposing a simple solution to what is a hugely complex problem, and throwing ML at it just isn't going to work. To get "plenty of training data," a human still has to classify all of that, leading to the problem of viewing that much unpalatable content by a human. You also have to train your network, which requires humans to verify the accuracy of training, hence viewing the content again.
If it were as easy as text-based spam filtering, this wouldn't even be a discussion.
How does it differentiate between normal ranting and hate speech? How is hate speech classified in the US vs Saudi Arabia? How does it tell the difference between someone asking a child innocent questions and asking them sexually-related questions on camera? Does the algorithm get trained to flag videos about depression that might lead to suicide, or does it say they're supporting getting help FOR depression and leave the video?
What about subjects that aren't already in the corpus of "flag these naughty things"? You still have to get a human to look at those; most likely a data scientist who knows what the algorithm is doing and what needs to be done to correct the training. Machine learning as-is will not get us there, so in the meantime the only option is moderation by people described in the article. It can be outsourced elsewhere, but it's just shifting the responsibility to a different subset of people.
That doesn't matter from a staff point of view though. You have a queue to work through. You'll be putting in an 8 hour day dealing with the stuff the system doesn't catch. The automation just means they don't need as much staff.
You're still going to get false positives and false negatives that need human review, and at a scale of Facebook, that's a lot of humans.
https://news.ycombinator.com/item?id=22487403
https://news.ycombinator.com/item?id=5068626
Is deeply offensive content generated by a handful of users frequently, or many users less often? What's the volume look like?
Anyone know? Is it like 10%? 80%?
There must be, by now, a whole lot of data on whats getting flagged region/language wise.
Maybe we need a Cloudflare for Content.
Most websites it seems when you upload an image it's immediately available, I always wonder if it goes through some basic moderation system or just waits to get reported.
I've done tagging jobs on MTurk before it's weird... seeing random people's images.
I think vetting users rather than content is probably the most efficient method of community curation though. If you have an approval process or other ways to validate that users aren't nefarious then you would reduce your workload by a bunch. Most bad images seem to come from burner accounts.
Unfortunately the only thing that comes to mind is a social credit as a service style system and maybe that's not where we want to steer this ship...
I can get from the user side about the burner accounts. I feel a physical response when I get denied to post a first post say in a sub and I'm like "Excuse me..." But yeah that's definitely discouraging/limiting to stick around before contributing.
The local governments won't acknowledge this as problem because of the money flowing in and a lot of people's livelihood is dependent on this. But we should call it what it is, exploitation.
> According to Dr. K. Jyothirmayi, a Hyderabad-based psychiatrist, stigmas, such as the perceived impact of a mental health diagnosis on one’s marital prospects, often prevent young Indians from seeking treatment.
From growing up in India, my view is that meditation and the general state of the mind has been taken seriously for, well, centuries. But move the focus towards anything with 'psych' or 'mental' in the title and people in India will shy away from it even if it is affordable or free. "Seeing a mental health counselor" is a recipe for considerable social stigma in India [1], not an empowering practice like it's perceived as in the United States.
The solution to this is hard, and in my opinion not something that Facebook or their Indian subcontractors can accomplish.
-----------
[1] I'm talking "All your relatives will refer to you as the crazy person" or "Your girlfriend's parents will call off the engagement" levels of stigma, not just "I don't feel comfortable talking about it in a bar" levels. Like a lot of other things, families with higher levels of income and education are sometimes an exception.
Real life has always been filled with messed up situations where the punchline is "... and the best/worst part is it qualified as an improvement!".
[1] https://www.fbi.gov/history/famous-cases/operation-innocent-...
It's not the flat-earther and somesuch conspirary communities that worry me. It's the people who share and get enjoyment from this kind of content, and the networks which enable it.
I do think we need to think about the side effects of laws where some US citizen demands that Facebook must employ content moderations else be liable for hosting it, it doesnt result in Facebook executives, engineers et al. bearing the burden. It results in some poor person in a 3rd world country bearing the burden.
Kind of like the environmental aspect of things I would think the best investment is in total prevention of the heinous acts in the first place. ie, it's much cheaper to prevent CO2 than it is to clean it up after the fact.
- Facebook: it's violating rules? Delete, next.
- Forensic IT on a multi TB disk full with child porn: document every photo, what it shows, extract identifiable faces to cross reference with other content (to check for recurring places and victims), and the process is even more gory for video content. You have to watch every second or the defense can attempt "you didn't watch the video in full where the perp gives the victim an ice cream at the end" or whatever else. The amount of time you spend with documenting a single photo or video is many orders of magnitude worse than FB content mods.
Can you crowdsource the moderation task to double check what the ai is flagging?
I highly doubt that. These reported posts might contain private data or highly shocking content; putting that on a public platform, especially associated with the name of Facebook, is very unlikely to happen.
Is that not what the person who originally posted the content being moderated is doing?
The person that posted the content does not expect a random person from the internet to see a video of his children because it was accidentally flagged by an AI as child porn. :)
"Facebook sent me child porn" is probably not a headline Facebook wants out there.
This would help with false positives, which is a noble goal. But it would not help with false negatives, which is imo a very disastrous consequence in this specific scenario.
Something that isn't CP got marked as CP? Sure, a moderator can manually doublecheck that flag and override if needed. Something that is CP didn't get marked as CP? You got a problem on your hands, because there is no way to manually doublecheck that, since that would mean manually checking every single post marked as "no issues here", which gets us back to square one of manual moderation of all posts.
Not even mentioning the exposure of FB users to CP that it would bring, which would mentally ruin quite a lot of people and bring very negative press to FB on the level of severity that we haven't seen before.
Or to restate, sites that rely purely on manual moderation of content that breaks rules still has content that breaks rules leak through all the time.
> Can we build better AI
> Can you crowdsource the moderation task
Users don't want the back-stop to be "a critical mass of community members object;" for content like this, they want to see zero of it, ever, and will choose to use another site that satisfies that hard constraint if FB cannot.
> they want to see zero of it, ever
> When the goal is to prevent users from
> seeing this content, crowd-sourcing moderation
> defeats the purpose.
No system can ever stop 100% of users from seeing 100% of the material they find objectionable. Every user has their own idea of what material is beyond the pale, and even if we invent AI that is competent at blocking content, and capable of tailoring its filters per-user, it will still fail some of the time, because users are people, and people's tastes change over time.
Aside from that, the way Youtube, Reddit, Twitter et al, currently operate, a user who reports objectionable content typically waits days, weeks, or years (or forever) for the company to take action. If you give a user a little real agency, it goes a long way to mitigate their displeasure over occasional exposure to unwanted content.
Meanwhile, if your system requires users to view beheading videos regularly, people will just migrate to a site that doesn't require that.
> What real agency does the system you propose offer?
> if your system requires users to view beheading videos regularly
> people will just migrate to a site that doesn't require that.
Precisely. How often? Close to never.
https://www.bangkokpost.com/thailand/general/1853804/mass-sh...
I could swear there was also a Christchurch copy-cat attack in Europe, but I can't remember sufficient details about it to find an article. Perhaps that one wasn't streamed on FB.
I don't have encyclopedic knowledge of Facebook atrocities (I closed my account nearly a decade ago), but without vetting content before it goes live, I don't see how they will entirely prevent these videos from reaching some users.
(That's before we factor in unintended consequences such as the risk that a critical mass of moderators decide the garbage is signal and start passing it. It's more risk than FB wants to take on for a problem they already solve via paid employees).
I walked out of there shellshocked.
It would be worse to not invest in more automation, even malpractice.
I can't believe there's so much original content for this. Maybe they could share a db of hashes of known bad videos across sites and government agencies.
Clearly, there's near-universal agreement that some moderation is fine; only a handful of (downvoted) comments on this thread saying "The easiest way to fix this problem would be for Facebook to stop censoring posts and let users block what they don't want to see." But we clearly don't think that's a good solution in this case.
The question of whether QAnon should also be blocked is one of degree, not quality.
https://www.politicalite.com/latest/facebook-employee-if-som...
In this video it is shown that regarding beheadings, Facebook made it a point to send their moderation contractors a memo that the image that was frequently reported of President Trump's face with a knife at the neck where he is being beheaded was an exception to the content policy on beheadings/violence/incitement because it was considered to be "art" by some museum somewhere in Portland - which from what I can see on TV - appears to be an extremely volatile and violent place in some parts.
There is a reference to a cartoon post of Elmer Fudd shooting a cartoon gun at Beto O'Rourke with the text "I'm here for your guns" at the top - which was not given any similar exception.
I don't think Facebook or their partners are serious about this kind of thing at all since they seem ok with advocating violence and gore when it resonates with some personal opinions.
Also noteworthy is that the name of this whistleblower garners fewer search results on Google, than the name of another whistleblower Eric Ciaramella who, if you uttered a-la Voldemort, would get you banned instantly.
Broadly, content policy at these websites seems to be a lord of the flies, unprincipled hackjob of whatever the political machinery at the organization deems worthy of steering society in the direction of their whims. Given the influence and power of Facebook and other tech platforms, this should worry people. I'm less inclined to be sympathetic to these folks and the difficulty of their jobs, given the apparent willingness to become social constructionists with the power that they wield so irresponsibly.
I understand there are some jobs in the world that need to deal with dark stuff (like law enforcement), but social networks just aren’t worth the human cost.
What you’re proposing is not just shutting down social networks, it’s shutting down any website that involves user content, anything that allows photo/video upload, comments, or any kind of user interaction. That’s impossible.
You point out that public safety jobs are view more “worth it,” and certainly they are, but that logic brings up the question of who judges what job is worth undergoing trauma.
In other words, is a subway or freight train driver’s job “worth it,” if they have to see someone commit suicide on the tracks? What about crime scene cleanup companies? Funeral services? Bus drivers? Truck drivers? Nobody’s going to agree on where to draw the line in the sand.
A more realistic solution might be to make comprehensive support systems, mental health resources, and treatment a legally mandated, completely free service provided to any employee that works in these kinds of fields.
Finally, I think there are most certainly people out there who are not as sensitive and affected by this content who would be candidates for these kinds of roles. Perhaps there’s a way to test for that sensitivity before the real job starts.
As far as “worth it” goes, some people have to be exposed to it so long as we have law enforcement (but I’m certainly open to alternatives here). I’m not sure the train operator is a fair comparison, because seeing a suicide is an exceptional circumstance in their job, it’s not the norm. The content moderators, however, are sadly expected to be exposed to traumatizing content as part of their job description — it’s essentially the point of their job.
There are plenty of kinds of work we deem as hazardous to people’s health, and thus are either banned or regulated. I’m not sure if there’s a healthy way to expose people in these moderator jobs to the traumatizing content they face. It just doesn’t seem worth the tradeoff to endanger them like this.
> [shut down] just large / public sites that require this sort of moderation
Let’s say I start a restaurant review website that allows comments and photos to be uploaded. It does modest business for a while, I now have 50 employees. I’m following the law because my site isn’t big enough to violate this “no user content for big prominent websites” law.
Soon, it becomes big, like a major competitor to Yelp, and I’ve got 1,000 employees. But suddenly, this new law kicks in that says that I have to stop accepting uploads because my site is too high profile. Now, I lay everyone off and go out of business.
This just isn’t a workable solution, at least not in the particular way you’re proposing it be constructed.
And really, you’re asking the second largest advertiser on the web (Facebook), a Fortune 50 company, to just pack up its bags and shut down.
It’s not like I love Facebook or anything, but I’m sure their 45,000 employees wouldn’t be happy about that.
People would be less inclined to post this type of content to Facebook if it their account was very connected to their real identity, if they were 18+, and if there was meaningful punishment for making fake accounts or accounts with stolen identities.
I wonder if prosecuting people who post horrible stuff would have a deterrence effect, though.
Facebook doesn’t have to let people post material to their site with very low barriers to entry. Granted they have to if they want to continue to be what they are and to make tons of money... but maybe they shouldn’t continue to be what they are and make tons of money if this kind of abuse is necessarily coupled to those outcomes.
“Well this sucks but we can’t keep operating if we don’t do it”. Well... sure but the solution is right there in that statement. Don’t keep operating.
[edit] thought experiment: how many of the people making tons of money off Facebook—c-suite, major shareholders—would find some other way to make money if continuing to make Facebook mega bucks meant they had to do this 5 days a week? What would we think of any of them who chose “bring on the trauma, I want those sweet greenbacks” and kept it up for years?
I think the problems that Facebook, Youtube, etc are seeing stem from the fact that the content is much more public, thus in much more need of moderation.