A Facebook content moderator’s resignation note
twitter.com
twitter.com
That's a massive issue as lawmakers are pushing for more moderation. It's my belief that Facebook, and others, have sold our politicians on the idea that moderation can be done, at scale, with AI. Those of us who followed along knows that the human moderators have complained that at least Facebooks A.I. simply isn't up to the task.
I don't think Facebook and similar sites can exist, not at that scale, without massive human cost.
AI won't be able to do the job until we manage to digitize brains. Stuff like sarcasm, irony, fair use or "dog whistling" (e.g. using "globalists" as a replacement for "jews") is something current AI's can't grasp at all, and even a digitized brain will struggle with these concepts - not to mention the ethical concerns that would come with digitizing a brain. Is it morally acceptable to torture a virtual brain that, in contrast to a human, can't even leave? To reset it to default state when it's too damaged?
As a concrete example: "Congressman Falls for The Onion's Planned Parenthood 'Abortionplex' Story" https://www.theatlantic.com/national/archive/2012/02/congres...
There is not free lunch.
Side note = firms increasingly see their policy divisions as responding to government and political pressure, not content moderation.
The next couple of years in the policy space are going to such for employees, as govts demand more power and oversight from SM firms.
It absolutely can - all you have to do is to discourage posting bad content in the first place by attaching actual consequences to breaking the rules.
Things like restrictions on new accounts such as limits on posting links, media, number of posts, etc until the account slowly levels up its trust level. This would protect both against spam from new accounts (as they aren't able to do any significant damage) as well as incentivizes people to not break the rules as to not lose their "leveled up" account. This worked very well for a decade on various forums often run by volunteers with near-zero budget.
The real problem is that unlike the forums of the good old days, Facebook's business model is not to provide a safe environment for quality discourse - instead it's to keep as many people "engaged" as possible - in this case banning rule-breakers is actually counter-productive, and is only done as a last-resort when their misdeeds are serious enough to become a PR risk for Facebook.
Their complaints about moderation being impossible is a sleazy attempt to make people accept this and for bad content to eventually stop being a PR risk, thus theoretically allowing them to completely do away with moderation and still earn ad revenue from bad/criminal content.
P.S. Former Facebook employee
Endless images of young children. Being sexually assaulted, penetrated, made to do degrading things.
Think about how that statement makes you feel, and then imagine you’ve been tricked into looking at this stuff for money as a “content moderator.” It’s a form of cruel economic punishment.
Facebook wouldn’t have these problems if they weren’t obsessed with acquiring as many users as possible. They are the ones who should be forced to pay for the long term mental health of these workers.
California ballot initiative, anyone?
Here’s a starting point, from one very repressed culture where the act of going to the police puts great shame on your family and turns you into an outcast:
> There seems to be no sign of decline in the number of sexual abuse cases against children. According to the Ministry of Health, Labor and Welfare, child consultation centers nationwide responded to 2,077 cases of child sexual abuse in fiscal 2019, up 2.75 times from fiscal 2000, when the Act on the Prevention, etc. of Child Abuse went into force. In 1,056 of these cases, the perpetrators were biological fathers.
https://mainichi.jp/english/articles/20210322/p2a/00m/0na/01...
Please, listen to children and believe their stories.
I don’t think there’s a magical line where it suddenly has an impact on mental health, and the details will vary from person to person, but there is most certainly a point where you go from functioning adult to justifying a formal diagnosis of PTSD or something similar.
When I was at FB ~2012+ -- I commuted in with one of the content mods on the CP/abuse -- and shit I heard from him was HORRIFIC. and there is TONS of it.
And with all the pedo-network shit coming to light these days.. child abuse is WAY WAY WAY worse than anyone can imagine on this planet.
Ill take your downvotes without comment as a tacit acquiescence to child abuse.
You might be getting downvoted by your former colleagues. Would you be surprised?
They should be transparent about it, IMO - as we really need a growing outcry on child safety rights.
We constantly talk about the rights and protections for all these various groups, yet, in really really dire situations like this, sex trafficking etc, by comparison is way more important than kvetching over identity pronouns and such.
We need a global anti-pedo-abuse force.
> I'm shocked by the implication that there's so much of it - that there's enough to damage somebody's mental health
And I reply that just a few images had an outsized impact on me. And down voted. Normally, anecdotal evidence is replied to with the general response that "the plural of anecdote is not data."
Maybe the down note is that I'm not referencing mental health? Am I supposed to connect all the dots in my reply and go from "if a few images can have a visceral impact on me, that it is within reason that much more exposure especially over time could lead to mental health issues"?
The stats there say they've dealt with 1.2m reports over 23 years, with 560,000 takedowns. That's certainly more than a couple a month.
It just doesn't make sense to me. Also, why is it that facebooks content moderators have to look at everything while Google just removed content and says nothing more about it? Does Google have more confidence in their AI while Facebook doesn't and therefore need people to manually check it? Or is it that there might be more content on Facebook uploaded that an AI would misinterpret as child abuse (baby photos and the likes that people seem so desperate to share on Facebook these days).
Very eye-opening stuff: https://youtu.be/qv_hokG2oSo
I don't disagree... However, this doesn't do anything to address the issue that someone still has to sift through this disturbing content, and society as a whole probably still pays the price.
I don't think any sort of compensation or mental help would prevent the kinds of side effects this sort of role is exposed to.
Most photos and videos are crappy. They could throttle it to 10 per person per day and increase quality. I’m sure people would complain and FB would think they lose out to a competitor with unlimited uploads. However, a reduced load means easier job to address the bad stuff.
It’s probably unworkable, but making people more discerning rather than less isn’t a bad thing.
Do we just sit back and say "let's stop subjecting those poor officers from having to see evil things"? Of course not. We hire people with strong stomachs and a strong sense of duty. Not everyone who sees this stuff is scarred for life.
See also: Soldiers. Firefighters. Emergency medical workers.
Personally, I'd rather moderate gross things than, say, work in a coal mine. But that's me. I don't appreciate the insinuation that this makes me some sort of monster.
It's difficult to have a strong sense of duty for the megacorp that doesn't even bother to employ you directly, but pays another megacorp which then pays you peanuts. When you don't even get to have the context of what you're doing, don't get to follow up on whether what you did was meaningful, whether the perp was caught and tried, whether the child was saved.
It's difficult to have a strong stomach if you don't see this as an occiasional element of your generally tame and meaningful work, but see hours upon hours, days upon days of some of the worst shit that the human race is capable of.
I know this isn't what you're doing, but this sentiment does come awful close to, "they know what they signed up for" logic. Most Marines know before they sign up that they are signing up for a high risk job, mentally and physically. In bootcamp you go through fairly extreme (to an every day person) conditioning and you are regularly conditioned to stress throughout your tenure. It takes an insane amount of work to make a person disregard concern for their own safety and run towards the sound of gun fire. Even then, this conditioning is problematic when people get out, due to numerous reasons, which is why (imo) you see so much disarray in veteran communities. Conditioning people to be to handle extraordinary circumstances is necessary, imo, but you have to find a way to decondition them.
I happen to know cops. The suicide and dropout rate for those in IT evidence and sexual crimes departments is enormous.
https://www.theatlantic.com/national/archive/2014/09/police-...
> Two studies have found that at least 40 percent of police officer families experience domestic violence, in contrast to 10 percent of families in the general population. A third study of older and more experienced officers found a rate of 24 percent, indicating that domestic violence is two to four times more common among police families than American families in general.
The 40% study itself even says that the victims reported a 10% rate of physical domestic violence from their partner, but specifically does not indicate who the aggressor is - the officer or their spouse. The study is also 30+ years old at this point. [0]
More recent research from 2009, which imo has a far superior methodology and a larger sample size, says that "over 87 percent of officers reported never having engaged in physical domestic violence in their lifetime" [1]
Other studies [2] [3] report rates between about 2% and 11% for mixtures of one time and recurring domestic violence, which are in line with population averages.
[0] https://babel.hathitrust.org/cgi/pt?id=umn.31951003089863c
[1] http://scholarcommons.usf.edu/etd/1862
[2] http://webapp1.dlib.indiana.edu/virtual_disk_library/index.c...
[3] https://digitalcommons.law.umaryland.edu/cgi/viewcontent.cgi...
I'm sorry, are we reading the same thing?
"Using a survey created from existing scales, 250 officers were contacted within several departments in Central Florida, of these, 90 officers responded."
Asking cops "do you do crimes" is an interesting methodology, for sure.
How do you think the population wide numbers are gathered?
The core point is that the 40% number is exaggerated BS, if for no other reason than it literally counts "shouting" and "loss of temper". There's no reliable evidence indicating the number for police officers is any different than the general population.
Not merely by asking the perpetrators.
13% of active cops admitting in a survey to domestic violence certainly means the actual number is higher than that.
Which means 12-13% _admitted_ committing physical domestic violence, what has to be a significant undercount.
Even police forces have developed unions and employee care and training practices to keep them sane and healthy.
In contrast, for CM you have the magic of the market outsourcing the work to replaceable humans.
To be honest this assertion of “strong stomachs” is a unsubstantiated and unhealthy fantasy of how people in stressful jobs handle that stress.
I have numerous friends who are police.
They had three months training, then were sent out to see the worst humanity has to offer.
One of my police friends has a coffee table book of interesting dead bodies from crime scenes.
I'm not sure what 500 years of best practice you're referring.
Compare to things like Spanish brotherhoods or the military handling policing in Rome.
As a firefighter and paramedic, I wouldn't even begin to compare my job, even dealing with the occasional horrible things, in the real world, with a job that consisted of hours on end of viewing things that would have been deleted from rotten.com for being too problematic, and the effects that that would have on my mental health.
Some time back, the child of some friends died, rather badly. This was a child I knew. They asked me to go through the police report because they could not bear to do so, just to see if anything was missed. In the interests of shielding them, I did. I had to detach in a peculiar way, and that may be while I have been able to attend autopsies and such.
Could I do it forty hours a week? I don't know, I won't claim to it, but I think looking for some kind of quality like that would be where to go until AI takes this over -- and frankly I think that will be a long time coming.
Most of these people grew up in cities, I grew up in the country side cutting chicken or pigs for food; this makes you less sensitive to some extent.
My friends that are now MDs told me that year was the worst experience of their lives. Being a Facebook moderator can be worse than that, so it can be really bad for some of the people doing this job.
This is just incredibly telling how shielded these people are.
Psychological trauama and PTSD appear fairly prevalent for first-responders (worldwide)[1].
And first-responders tend to receive a lot of community support, respect and hazard pay in return for what they do.
[1] - https://en.wikipedia.org/wiki/Trauma_and_first_responders
Is there any venture capitalist that would generally believe "there is value in not trying to connect everyone on Earth together?"
It feels like when you market your hot startup idea, connecting the world or enacting change is merely a cool new thing that carries no long-term downside. Find a market, connect with a target audience, grow. Maybe that's how Facebook started (or not).
I feel that, because of human nature, if it weren't Facebook then some other company would have inevitably taken its place. Being able to connect to someone faster and easier than ever sounds like a tangible gain for average people. The idea, with no strings attached, was too tempting to pass up for someone that truly wants to change the world and be a hero of some kind. It worked for a single university, after all. It's the sense of starting a technology company when the future is completely unknown and the potential for change is exciting. I'm not sure there was anyone that would have understood the consequences of taking such an idea to its logical conclusion, and now Facebook has become irreversibly baked into human culture.
The more people that want to connect, the more moderation staff is needed to match. Unfortunately, the desire to connect with others is wired into pretty much everyone on Earth at the deepest level (with exceptions, of course), and so your potential market is... pretty much everyone on Earth.
Even small sites have people upload child porn and other stuff into them, and some IT guy has to go handle it.
The only thing that can really reduce the scale of this stuff is for the people doing it to suffer a consequence like jail time. That's clearly not happening in most cases, despite Facebook and others sharing information with the authorities.
The same is true for other issues like people being hounded by death/rape threats and the like online. The police mostly ignore it as an issue, so moderators have to read a bunch of it, and the people doing it just get a suspension or ban from the one platform.
It's part of what makes Facebook enjoyable for end users, and that they've managed to do: keep the platform clean of most awful things, but it comes at a heavy cost. People don't notice this cost, and that's by design.
Or use ML to auto blur specific parts that are likely to be gore etc, and allow moderator discretion to remove the occlusion.
I bet some pretty basic tricks like this would help a lot. A high resolution image of all the gore (or whatever) inside it is probably not needed to come to a decision in most cases.
Its quite common for companies to store all of their data in excel files on a share drive with no backups. Why would companies do that when there are so many better ideas?
The letter makes it clear that these contractors have basically no interaction with FB proper, and are treated as disposable. Is that the sort of environment that leads to tools improved in creative ways just to make it less painful to work there?
It's similar to Amazon's warehouse pickers, callcenter agents and farm laborers. The less skills your job requires the more abuse employers can get away with.
Facebook has none of these restrictions, and these employees don't even work for Amazon, and they are behaving the way you would expect a company to behave with these incentives.
Isn't it more likely that they did, but some highly paid employee or consultant said "you know what, just let them wear diapers or pee in bottles" and Bezos applauded such a brave and innovative approach?
Actively push to get on the roadmap and implemented, when the moderators are in another state employed via a third party contractor with no direct contact to said managers/engineers? Less likely.
- fake reviews which make the ratings useless
- scam apps that riddle the various app stores
- abusive / illegal content on platforms that can't afford better moderation
- moderators who struggle with mental health issues, as per this tweet
- footballers receiving racist comments on their socials
Is there space for a startup to solve this problem with AI? And should it be a B2B or B2C scenario, as in working with platforms to moderate in the backend or being a user app / extension that does it in the frontend?
For example, Fakespot works great with Amazon but it's an extension you have to install, so it's not available on Safari / on Mobile. So it's still quite unknown and I'm not sure how much it can grow
Doing a really good job with moderation will materially and negatively impact KPIs in the short term.
We barely understand how biased it is, and even if/why it's necessarily biased, not by design, but by nature. As in, biased, because the reference you use is subjective to the time and society and culture it is built in. Which is different from the whole context 30 years ago, and will be different in 30 years.
Why, also, would tech solve a problem tech did not create, but amplified?
The reason it's a cost centre is because it's trivial to stand up a new platform with a higher bar to entry that precludes almost all evil content, with the limiting factor that it won't have exposure to the total size of the legacy platforms that have no bar to entry. So for FB, better moderation is not a differentiator with an ROI, it's a risk management cost. This puts a ceiling on what they'll pay for it.
The ideal solution for an FB-like platform that is already committed to open doors user policy means that the cost model/value prop is that the solution must be able to show it's doing something above all.
If you sold them an AI solution that "just worked" for 98% of material and had a tiny relative cost, FB's real risk from the 2% that still got through doesn't change, because when FB gets asked by regulators and others what they are doing about it, your incredibly cost-effective AI solution just shows FB isn't committing resources to the problem.
When FB can show rooms full of workers at screens on 24/7 shifts responding to the problem (at some tiny foreign hourly wage that their employer with the moderation contract is probably making a 300%+ margin on) that is the solution. These peoples' job is to essentially be psychological human sacrifices to regulators. This also describes most of the economics of the managed security services SOC business model.
If you want to get into horror, find out if there is a correlation between the production of evil content and the regional presence of a moderation subcontractor. It's like anti-virus and security companies releasing viruses and exploits, but next level worse. If there is $100m/year market for moderation services, that literally creates the perverse incentive to ensure a flow of evil content.
Could there be a CrowdStrike of content moderation? Maybe. Especially in markets like China, Singapore, and increasingly Britain, where policing content could be creating the same regulatory environment that hackers created for security.
Darkly true. I think this shows that the incentives are not aligned.
Outsourcing makes this a “solved” problem. Content moderation which ends up reducing time on site but saving people will not see the light of day. So the job is to flag and send content down the moderation pipeline.
For now the frontier is dealing with the politics, not improving the lot of content moderators.
Policy people will get hired because of how well they know/navigate the politics of their region, not because of their ability to design good moderation policy.
Facebook could out, shame, and be a significant aid to prosecution of the individuals who post illegal and obscene content.
But there are no regulations compelling them to do so. There's no business incentive to do it themselves.
And maybe most important, it's very clear that Facebook has no ethics in their own internal culture to take a stand on principle.
None of us are perfect.
If the Facebook execs had to do a day of content moderation every quarter, I imagine that meaningful change would happen a lot sooner.
In this analogy, are moderators of a given platform equivalent to mall cops/neighborhood watch?
But hey if you work for BuzzFeed...