More than 140 Kenya Facebook moderators sue after diagnoses of PTSD
theguardian.com
theguardian.com
I had an Uber from the campus one day, and my driver, a twenty-something girl, was asking how to become a moderator. I told her, "no amount of money would be enough for me to do that job. Don't do it."
I don't know if she eventually got the job, but I hope she didn't.
It's just kind of ridiculous how people think war is like Call of Duty. One minute you're sitting in a trench, the next you're a pile of undifferentiated blood and guts. Same goes for car accidents and stuff. People really underestimate how fragile we are as human beings. Becoming aware of this is super damaging to our concept of normal life.
It's that we have technologically engineered things that are destructive enough to get even past that threshold. Modern warfare in particular is insanely energetic in the most literal, physical way - when you measure the energy output of weapons in joules. Partly because we're just that good at making things explode, and partly because improvements in metallurgy and electronics made it possible over time to locate targets with extreme precision in real time and then concentrate a lot of firepower directly on them. This, in particular, is why the most intense battlefields in Ukraine often look worse than WW1 and WW2 battles of similar intensity (e.g. Mariupol had more buildings destroyed than Stalingrad).
But even our small arms deliver much more energy to the target than their historical equivalents. Bows and arrows pack ~150 J at close range, rapidly diminishing with distance. Crossbows can increase this to ~400 J. For comparison, an AK-47 firing standard issue military ammo is ~2000 J.
It's almost military precision.
On behalf of dead WWI soldiers I find this offensive.
Brutal murder is low tech.
It’s like the original massive scale organization.
(Thankfully in the US we worship the 2A and its most twisted interpretation. So our toddlers do shooter drills. /s)
Or, gather a group of people, tell them to find a rock, and go bash the other sides head.
The ak-47 famously only needs the first item in that list.
That being the key to its popularity.
https://www.youtube.com/watch?v=DX73uXs3xGU
https://www.youtube.com/watch?v=YAneTFiz5WU
As far as cleaning, AK, like many guns of that era, carries its own cleaning & maintenance toolkit inside the gun. Although it is a bit unusual in that regard in that this kit is, in fact, sufficient to remove any part of the gun that is not permanently attached. Which is to say, AK can be serviced in the field, without an armory, to a greater extent than most other options.
But the main reason why it's so popular isn't so much because of any of that, but rather because it's very cheap to produce at scale, and China especially has been producing millions of AKs specifically to dump them in Africa, Middle East etc. But where large quantities of other firearms are available for whatever reason, you see them used just as much - e.g. Taliban has been rocking a lot of M4 and M16 since US left a lot of stocks behind.
And Western supplies are mostly 5.56 for obvious reasons, although there are some exceptions - mostly countries that have switched fairly late and still have substantial stocks of 5.45, such as Bulgaria. But those are also limited in quantity.
So in practice it's not quite so simple, and Ukraine seems to be aiming for 5.56 as their primary cartridge long-term, specifically so that it's easier for Western countries to supply them with guns and ammo.
Soldiering isn’t sexy, it’s digging trenches, cleaning kit, and eating concussive blasts waiting to fight or die.
You don’t sit in a bunker all day waiting to defend a trench and not clean your gun.
They are much less likely to.
We have instinctive repulsion to violence, especially extending it (e.g. if the rock does not kill at the first blow).
It is much easier to kill with a gun (and even then people need training to be willing to do it), and easier still to fire a missile at people you cannot even see.
If you caused enough damage such that someone could not later be recognized, I wonder why you are not in prison.
Funny you mention crossbows; the Church at one point in time tried to ban them because they democratized violence to a truly trivial degree. They were the nuclear bombs and assault rifles of medieval times.
Also, I will take this moment to also mention that the "problem" with weapons always seem to be how quickly they can kill rather than the killing itself. Kind of takes away from the discussion once that is realized.
On the other hand, never seeing or reckoning with or preparing for how brutal reality actually is can lead to a pretty bad shock once something bad happens around you. And maybe worse, can lead you to under-appreciate how fantastic and beautiful the quotidian moments of your normal life actually are. I think it's important to develop a concept of normal life that doesn't completely ignore that really bad things happen all around us, all the time.
there’s a difference between a one or two or even ten off exposure to the brutality of life, where various people in your life will support you and help you acclimate to it
Versus straight up mainlining it for 8 hours a day
I get your idea but in the context of this topic I think you're overreaching
Emergency personnel might need to braze themselves for car accidents every day. That Kenyans need to be traumatized by Internet Content in order to make a living is just silly and unnecessary.
Even the wording is wrong - those aren’t accidents, it is something we accept as byproduct of a car-centric culture.
People feel it is acceptable that thousands of people die on the road so we can go places faster. Similarly they feel it’s acceptable to traumatise some foreigners to keep social media running.
You're literally not allowed to acknowledge that this stuff is bad and adopt one of the religions that see this and try to remove suffering - i.e. Jainism, because at least historically doing so meant you couldn't use violence in any circumstances, which also meant that your neighbor would murder you. There's a reason that Jain's population are in the low millions
Reality is actually bad, and it should be far more intuitive to folks. The fact that positive experience is felt "quickly" and negative experience is felt "slowly" was all the evidence I needed that I wouldn't just press the "instantly and painlessly and without warning destroy reality" (benevolent world-exploder) button, I'd smash it!
> There's a reason that Jain's population are in the low millions
The two largest Vedic religions both have hundreds of millions of followers. Is Jainism that different from them in this regard? I know Jainism is very pacifist but on the question of suffering.
EDIT: If you’re interested what actually happened is that I was missing the prerequisite early childhood experience that enables one to feel secure in reality. If you check, all the people who have this feeling of philosophical/ontological pessimism have a missing or damaged relationship with the mother in the first year or so. For them, not even Buddhism can help, since even the abstract idea of anything good, even if it requires transcendence, is a joke
OP got spooked to stop suffering and love life instead? Is that your cautionary tale?
My point really is that you can feel one way for your entire life and then suddenly feel a different way. I'm not suggesting psychoanalysis specifically. Perhaps for others, CBT or religion or just a change in life circumstances will be enough.
The fact that these philosophies are dependent on the life situation to me is a reason to be a little sceptical of their universality. In my personal experience, in those 30 years of my life, I thought everyone thought the way I did, that reality was painful and a chore and dark and dim. Psychoanalysis helped me realise that other people actually were happy to be alive, and understand why I have not been my entire life.
YMMV = not everyone hates life
I’m not sure why people act coy when a straightforward mirroring of their own comment is presented. “What could this mean?” Maybe the hope is that the other person will bore the audience by explaining the joke?
> I don't look up to Freud and psychoanalysis doesn't work for everyone! I don't even necessarily recommend it.
Talking about your infant parental relationship as the be-all-end-all looks indistinguishable from that.
> > If you check, all the people who have this feeling of philosophical/ontological pessimism have a missing or damaged relationship with the mother in the first year or so.
.
> I'm not suggesting psychoanalysis specifically. Perhaps for others, CBT or religion or just a change in life circumstances will be enough.
Except for people who have “this feeling of philosophical/ontological pessimism”.
> > For them, not even Buddhism can help, since even the abstract idea of anything good, even if it requires transcendence, is a joke
Which must paint everyone who defends “suffering” in the Vedic sense. Since that was what you were replying to. (Saying that reality is suffering on-the-whole is not the same as “I’m depressed [, and please give me anecdotes about how you overcame it]”.)
> > The fact that these philosophies are dependent on the life situation to me is a reason to be a little sceptical of their universality. In my personal experience, in those 30 years of my life, I thought everyone thought the way I did, that reality was painful and a chore and dark and dim. Psychoanalysis helped me realise that other people actually were happy to be alive, and understand why I have not been my entire life.
I don’t know how broad your brush is. But believing in the originally Vedic (Schopenhauer was inspired by Eastern religions, maybe Buddhism in particular) concept of “suffering” is not such a fragile intellectual framework that it collapses once you heal from the trauma when your mother scolded you while potty training at a crucial point in your Anal Stage of development.
> YMMV = not everyone hates life
Besides any point whatever.
And the Vedic version of suffering is all full of love for reality, not wanting to delete it by smashing a button
You trained personally for a few years and yet you make such sweeping statements/strokes that a neophyte is prompted to point out basic facts about this practice (apparently an adequate retelling since you don’t bother to correct me)? You might think this bolsters something (?) but I think the case is the opposite.
It helps to point out exactly what part that you are talking about (apparently not the Vedic gang). In fact this initial reply (just the above paragraph before the edit) seemed so out of place. Okay, so what are they talking about?
> And the Vedic version of suffering is all full of love for reality, not wanting to delete it by smashing a button
Oh, so it’s about the small wish to commit biocide.
It’s a clear category error to talk about love/want/hate when it comes to that statement. Because that’s beside the point. The point is clearly the wrongheaded, materialistic assumption that suffering will end if all life would end by the press of a button. And if you think that life on the whole is suffering? Then pressing the button is morally permissible.
It’s got nothing to do with hate.
It seemed interesting to me that someone would have such a “Schopenhauer” (not that I have read him) view of existence. You don’t see that every day.
> > Reality is actually bad, and it should be far more intuitive to folks. The fact that positive experience is felt "quickly" and negative experience is felt "slowly" was all the evidence I needed that I wouldn't just press the "instantly and painlessly and without warning destroy reality" (benevolent world-exploder) button, I'd smash it!
> This is coming off as incoherent rambling to me
You do like to gesture vaguely and tell me that "I don’t know what this is". Meanwhile I have pointed out at least one instance where you flat out just contradicted yourself on psychoanalysis. Or "incoherent rambling" (on psychoanalysis) if you will
Join the club apparently. How nice.
> Are you saying that if all life is suffering then ending all life is morally permissible?
Yes. To demonstrate that the original, maligned comment wasn’t about ending all life because they hate life and existence. But in order to minimize suffering.
The premise is wrong from a Vedic perspective because it denies Moksha. But if you believe that Moksha doesn’t exist? Then it’s rational.
And the push-button argument will depend on what your stance on utilitarianism is. Specifically how you can make the choice for every life in existence (instead of them making it). Which addresses your question.
I’m not gonna answer that because that goes beyond the point I was making.
> In that case, what do you do with the life of someone who says they're not suffering?
I used to live in the Disney headspace until my dog had to be put down. Now with my parents being in their seventies, and me in my thirties I fear losing them the most as the feeling of losing my dog was hard enough.
I say animal to explicitly include humans. Finding my hamster dead in fifth grade did change me. But watching my mother slowly die a horrible, haunting death didn't make me a better person. I'm just saying that there's a spectrum that goes something like: easy to forget about, I'm able to not worry, sometimes i think about it when i dont want, often i think about it, often it bothers me, and do on. You can probably imagine the cycle of obsession and stress.
This really goes for all traumatic experiences. There's a point where they can make you a better person, but there's a cliff after which you have no guarantees that it won't just start obliterating you and your life. It's still a kind of perspective. But can you have too much perspective? Lots of times i feel like i do
1) I don't have squeamishness about trauma. In the end, we are all blood and tissue. The calls that get to me are the emotionally traumatic, the child abuse, domestic violence, elder abuse (which of course often have a physical component too, but it's the emotional for me), the tragic, often preventable accidents.
2) There are many people, and I get the curiosity, that will ask "what's the worst call you've been on?" - one, you don't really want to hear, and two, "Hey, person I may barely know, do you think you can revisit something traumatic for my benefit/curiosity?"
I won’t watch most movies or TV because they are just some sort of tragedy porn.
100% agree. Most TV series nowadays are basically violence porn, now that real porn is not allowed for all kinds of reasons.
What's interesting now is how many patients will say "You're not going to give me fentanyl are you? That's really dangerous stuff", etc.
Their perfect right, of course, but is sad that that's the public perception - it's extremely effective, and quite safe, used properly (for one, we're obviously only giving it from pharma sources, with actually properly dosed solutions for IV).
The chefs butcher and serve the fish right in front of you, and because it was alive merely seconds ago the meat will still be twitching when you get it. If they also serve the rest of the fish as decoration, the fish might still be gasping for oxygen.
Japanese don't really think much of it, they're used to it and acknowledge the fleeting nature of life and that eating something means you are taking another life to sustain your own.
The same environment will likely leave most westerners squeamish or perhaps even gag simply because the west goes out of its way to hide where food comes from, even though that simply is the reality we all live in.
Personally, I enjoy meats respecting and appreciating the fact that the steak or sashimi or whatever in front of me was a live animal at one point just like me. Salads too, those vegetables were (are?) just as alive as I am.
Isn't this similar to why people unfamiliar with that style of seafood would feel sick -- cultural views on what is and is not normal food -- and not because of their view of mortality?
Imagine that to cook that pork chop, the chef starts by butchering a live pig. Also imagine that he does that in view of everyone in the restaurant rather than in the "backyard" kitchen let alone a separate butchering facility hundreds of miles away.
That's the sushi chef butchering and serving a live fish he grabbed from the tank behind him.
When you can actually see where your food is coming from and what "food" truly even is, that gives you a better grasp on reality and life.
It's also the true meaning behind the often used joke that goes: "You don't want to see how sausages are made."
The issue most people would have with seeing the sausage being made isn't necessarily watching the slaughtering process but with seeing pieces of the animal used for food that they would not want to eat.
The analogy here would be watching a live cow get slaughtered and then butchered from scratch in front of you, which I think most Western audiences (more than a few) might not like.
I think you are doing the same exact thing with the above statement as well.
That you insist on contriving one zany situation after another instead of just admitting that people today are detached from reality illustrates my point rather ironically.
Whether it's butchering animals or mining rare earths or whatever else, there's a lot of disturbing facets to reality that most people are blissfully unaware of. Ignorance is bliss.
Recall that this whole thread started with your proposition that seeing live fish prepared in front of someone "will likely leave most westerners squeamish or perhaps even gag simply because the west goes out of its way to hide where food comes from, even though that simply is the reality we all live in." You had no basis for this as far as I can tell, it's just a random musing by you. A number of folks responded disagreeing with you, but you dismissed their anecdotal comments as being wrong because it doesn't comport with your view of the unwashed masses who are, obviously, feeble minded sheep who couldn't possibly cope with the realities of modern food production in an enlightened way like you have whereby you "enjoy meats respecting and appreciating the fact that the steak or sashimi or whatever in front of me was a live animal at one point just like me." How noble of you. Nobody (and I mean this in the figurative sense not the literal sense) is confused that the slab of meat in front of them was at one point alive.
Then you have the audacity to accuse someone of coming up with "zany" situations? You're the one that started the whole zany discussion in the first place with your own zany musings about how "western" "commons" think!
I am asserting that you could do all of that in front of a random assortment of modern Americans, and then cut steaks off of it and grill them and serve them to half of the crowd, and most of those people would not have an problem eating those steaks.
Then if you were to scoop up all the leftover, non-steak bits from the ground with shovels, throw it all into a giant meat grinder and then take the intestines from a pig, remove the feces from them and fill them with the output of the grinder, cook that and serve it to the other half of the crowd, then a statistically larger proportion of that crowd would not want to eat that compared to the ones who ate the steak.
I am asserting that the majority of western audiences, including Americans, would dislike being present for the slaughtering and butchering portion of the experience you describe.
I probably also wouldn't want to eat that, but more because that steak will taste bad without being aged properly.
Some things just tickle the brain in a bad way. I've killed plenty of fish myself, but I still wouldn't want to eat one that's still moving in my mouth, not because of ickiness or whatever, but just because the concept is unappealing. I don't think this is anywhere near as binary as you make it seem, really.
Not to mention the millions of Americans working in the livestock and agriculture business who see up close every day how food comes to be.
A significant portion of the American population engages directly with their food and the death process. Citing one gimmicky example of Asian culture where squirmy seafood is part of the show doesn't say anything about the culture of entire nations. That is not how the majority of Japanese consume seafood. It's just as anomalous there. You only know about it because it's unusual enough to get reported.
You can pick your lobster out of the tank and eat it at American restaurants too. Oysters and clams on the half-shell are still alive when we eat them.
In case you missed it, you're talking to a Japanese.
Some restaurants go a step further by letting the customers literally fish for their dinner out of a pool. Granted those restaurants are a niche, that's their whole selling point to customers looking for something different.
Most sushi bars have a tank holding live fish and other seafood of the day, though. It's a pretty mundane thing.
Now, ask a laptop worker to butcher an animal whom used to have a name and to literally turn its meat into sausages and see what said worker’s reaction would be.
There is a lot of skill going in to it, so I couldn't do it myself. You need guidance of someone who is knowledgeable and has the proper tools and facilities for the job.
It is also ridiculous how people think every soldier's experience is like Band of Brothers or Full Metal Jacket. I remember an interview with a WWII vet who had been on omaha beach: "I don't remember anything happening in slow motion ... I do remember eating a lot of sand." The reality of war is often just not visually interesting enough to put on the screen.
What can I say? People like to live dangerously.
Not being aware of this is also a cause of traffic accidents. People should be more careful driving.
I've seen the crumpled metal of a car, I don't need to see the people inside to know it is not good.
You don't have to excessively see hours upon hours of such footage.
But a single harsh and gruesome traffic death footage sure as hell made me more careful in traffic. No amount of crumpled car metal was going to properly internalize in me how sudden and unexpected death can be and how fragile life really is.
Later in the year I moved to Florida, just in time for Helene and Milton. I didn’t spend much time thinking about either of them (aside from during prep and cleanup and volunteering a few weeks after). But I had frequent dreams of catastrophic storms and floods.
Different stressors affect people (even myself) differently. Thankfully I’ve never had a major/long-term problem, but I know my reactions to major life stressors never seemed to have any rhyme or reason.
I can imagine many people might’ve been through a few things that made them confident they’d be alright with the job, only to find out dealing with that stuff 8 hours a day, 40 hours a week is a whole different ball game.
It's also part of almost every American cop and military show and movie. Of course it's not real but it looks the same.
I beg to differ. TV shows and movies are silly. Action movies are just tough-guy dancing.
The organizer is even called a "fight choreographer".
And I wasn't talking about action hero movies.
Society would be a very different place if everyone had to do customer service or janitorial work one weekend a month.
Living in other Asian nations where there are often defacto invisible caste systems can be nauseating at times - you have parents that won't allow their children to participate in clean up efforts because their child is 'above handling trash.' That's gonna be one well adjusted adult...
Is that increase worth traumatizing moderators? I have no idea. But I frequently see this sentiment on HN about the old internet being better, framed as criticism of big internet companies, when it really seems to be at least in part criticism of how the median internet user has changed -- and the solution, coincidentally, would at least partially reverse that change.
Content discovery may even be able to remain centralized.
No idea if there's a way for it to work out economically without ads, but ads are also unhealthy so maybe that's ok.
What makes moderation different? and how should it be handled so that it reduces harm and risks? surely banning social media or not moderating content aren't options. AI helps to some extent but doesn't solve the issue entirely.
What seems novel about moderation is the frequency that you confront disturbing things. I imagine companies like Meta have such good automated moderation that what remains to be viewed by a human is practically a firehose of almost certainly disturbing shit. And as soon as you’re done with one post, the next is right there. I doubt moderators spend more than 30 seconds on the average image, which is an awful lot of stuff to see in one day.
A doctor just isn’t exposed to that sort of imagery at the same rate.
This doesn’t make sense to me. Their automated content moderation is so good that it’s unable to detect “almost certainly disturbing shit”? What kind of amazing automation only works with subtleties but not certainties?
So the things that are flagged for human review would be on the boundary, but trend more towards disturbing than legitimate, on the grounds that the human in the loop is there to try and publish as many posts as possible, which means sifting through a lot of disturbing stuff that the AI is not sure about.
There’s also the question of training the models - the classifiers may need labelled disturbing data. But possibly not these days.
However, yes, I expect the absolute most disturbing shit to never be seen by a human.
—
Again, literally no experience, just a guy on the internet pressing buttons on a keyboard.
They'd prefer that as many posts are published, but they probably also don't mind some posts being removed if it meant saving a buck. When canada and australia implemented a "link tax", they were happy to ban all news content to avoid paying it.
This probably means fewer human reviewers reviewing a firehose, not the same number of human reviewers reviewing content at a slower rate.
I never seen blood or gore in my life and find seeing it shocking.
But I’d imagine gore is a weekly situation for surgeons.
I also feel like there’s something intangible regarding intent that makes moderation different from being a doctor. It’s hard for me to put into words, but doctors see gore because they can hopefully do something to help the individual involved. Moderators see gore but are powerless to help the individual, they can only prevent others from seeing the gore.
Another fascet to this is the moderators willfully agreed to review this content and had full autonomy to leave the job at any point.
My guess is that's why it's after bulldozing hundreds of Palestinians, instead of 1 or 10s of them, that Israeli soldiers report PTSD.
If you haven't watched enough videos of the ongoing genocides in the world to realize this, it'll be a challenge to have a realistic take on this article.
I exposed myself to actual gore vids in the aughts and teens... That stuff still sticks with me in a bad way.
Context matters a lot.
My understanding is that during surgery, your body is most definitely in pain. Your body still reacts as it would to any damage, but anesthetics block the pain signals from reaching the brain.
Content moderator is a min wage job with bad working hours, no psychological support, and you spend your day looking at rape, child porn, torture and executions.
>Content moderator is a min wage job
So it's purely a monetary dispute?
>bad working hours, no psychological support, and you spend your day looking at rape, child porn, torture and executions.
Many other jobs have the same issues, though admittedly with less frequency, but where do you draw the line?
How about grouping the jobs into two categories: A) Causes PTSD and B) Doesn't cause PTSD
If a job as a constantly high percentage of people ending up with PTSD, then they aren't equipped well enough to handle it, by the company who employs them.
I fail to see how this addresses my previous questions of "it's purely a monetary dispute?" and "where do you draw the line?". If a job "Causes PTSD" (whatever that means), then what? Are you entitled to hazard pay? Does this work out in the end to a higher minimum wage for certain jobs? Moreover, we don't have similar classifications for other hazards, some of which are arguably worse. For instance, dying is probably worse than getting PTSD, but the most dangerous jobs have pay that's well below the national median wage[1][2]. Should workers in those jobs be able to sue for redress as well?
[1] https://www.ishn.com/articles/112748-top-25-most-dangerous-j...
[1] https://www.policepac.org/uploads/1/2/3/0/123060500/the_effe...
I wouldn't say purely, but substantially yes. PTSD has costs. The article says some out; therapy, medication, mental, physical, and social health issues. Some of these money can directly cover, whereas others can only be kinda sorta justified with high enough pay.
I think a sustainable moderation industry would try hard to attract the kinds of people who are able to perform this job without too much negative impacts, and quickly relieve those who try but are not well suited, and pay for some therapy.
On the contrary I would expect that it would be the edge cases that they were shown - why loop in a content moderator if you an be sure that it is prohibited ont he platform without exposing a content moderator?
In this light, it might make sense why they sue: They are there more as a political org so that facebook can say: "We employ 140 moderators in Kenya alone!" while they do indifferent work that facebook already can filter out.
US workman’s comp covers PTSD acquired on the job, and these kinds of jobs are rife with it.
Even with images, you can have different policies in different places or the cultural context can be relevant somehow (eg. some country makes you ban blasphemy).
Also, I have heard of outsourcing to Kenya just to save cost. Living is cheaper there so you can hire a desk worker for less. Don’t know where the insistence you’d only hire Kenyans for political reasons comes from.
Why assume they're just token diversity hires who don't do useful work..?
Have you ever built an automated content moderation system before? Let me tell you something about them if not: no matter how good your automated moderation tool, it is pretty much always trivial for someone with familiarity with its inputs and outputs to come up with an input it mis-predicts embarrassingly badly. And you know what makes the biggest difference.. is humans specifying the labels.
I am also not saying that content moderation should catch everything.
What I am saying is that the content moderation teams should ideally decide on the edge cases as they are hard for automated system.
In turn that also means that these people ought not to be exposed to too hardcore material - as that is easier to classify.
Lastly I say that if that is not the case - then they are probably not there to carry out a function but to fill a political role.
I imagine that they have a system that is somewhere between shitty and none functional. This is the company that will more often than not flag marketplace posts as "Selling animal", either completely at random or because the pretty obvious "from an animal free home" phrase is used.
If they can't get this basic text parsing correct, how can you expect them to correctly flag images with any real sense of accuracy?
My guess is even standing at the ambulance drive in of a big hospital, you'll not see as much horrors in a day as these people see in 30 minutes.
They are obviously not identical scenarios. They have similarities and they also have differences.
Not in the sense that matters here: the caller doesn't pay (unless the call is frivolous), leading to more calls that are preemptive, overly cautious or for non-live-threatening cases. That behind the scenes people and equipment are paid for and a whole structure to do that exists isn't really relevant here
> Do we think paramedics in Germany are more susceptible to PTSD
No, we think that there are far more paramedics than ICU or ER staff, and helping them in small ways is pretty easy. For ICU and ER staff you would obviously need other measures, like staffing those places with people less likely to get PTSD or giving them regular counseling by a staff therapist (I don't know how this is actually handled, just that the problem is very different than the issue of paramedics)
Content moderators see all of that.
And it used to be priests who had to deal with all the nasty confessions.
Why not? What good has social media done that can't be accomplished in some other way, when weighed against the clear downsides?
That's an honest question, I'm probably missing lots.
I think not. The gatekeepers of the old media pretty much died for a reason, that they seriously sucked at their job. Open social media and everyone having a camera in their pocket is what allowed us to basically disprove UFO sightings and prove routine police abuse of power.
You probably DO NOT want to read it.
There's a very good reason moderators are employed in far-away countries, where people are unlikely to have the resources to gain redress for the problems they have to deal with as a result.
Doctors and lawyers can’t afford that sort of option, but they tend to embrace alcoholism at higher rates and collect ex-wives.
Moderation may be worse in some ways. All day, every day, you see depravity at scale. You see things that shouldn’t be seen. Some of it you can stop, some you cannot due to the nature of the rules.
I think banning social media isn’t an answer, but demanding change to the algorithms to reduce the engagement to high risk content is key.
That's bullshit. Ever talked to a paramedic or firefighter?
The difference? The reputable hotlines care a lot more about their employees' mental health, with mandatory breaks, free counseling, full healthcare benefits (including provisions for preventative mental health care like talk therapy).
Another important difference is that suicide hotlines are decoupled from the profit motive. As more and more users sign up to use a social network, it gets more profitable and more and more load needs to be borne by the human moderation team. But suicide and mental health risk is (roughly) constant (or slowly increasing with societal trends, not product trends).
There's also less of an incentive to minimize human moderation cost. In large companies, some directors view mod teams as a cost center that takes away from other ventures. In an organization dedicated only to suicide hotline response, a large share of the income (typically fundraising or donations) goes directly into the service itself.
A content moderator for Facebook will invariably see more depravity and more frequently than a doctor or police officer. And likely see far less support provided by their employers to emotionally deal with it too.
This results in a circumstance where employees don’t have the time nor the tools to process.
Being a doctor and helping people through horrific things is unlike helplessly watching them happen.
IIRC, PTSD is far more common among people with sleep disorders, and it’s believed that the lack of good sleep is preventing upsetting memories from being processed.
Casual Facebook viewers may stumble across something disturbing on it, but they certainly don't get PTSD at the rate of the poor moderators. Likewise the professionals usually have their own professional exposure levels to messed up stuff. Meanwhile child pornography investigation departments who have to catalogue the evidence are notorious for suffering poor mental health even with heavy measures taken.
There is already the 'blacklist hash' approach to known bad images which can help reduce exposure. So they don't all need to be exposed to say the same cartel brutal execution video, the bot takes care of it. I don't know anything about Facebook's internal practices but I would presume they already are doing this or similar with their tech hiring base.
Dilution is likely the answer for how to make it more palitible and less traumatizing. Keep the really bad stuff exposure at similar proportions to what other careers experience. Not having 'report button' and 'checking popular content' as separate tasks and teams would probably help a little bit. I suspect the moderators wouldn't be as traumatized if they just had to click through trending posts all day. A dillution approach would still have to deal with the logistical trade-offs for what could be viable. Increasing the moderation payroll a hundred-fold and making them work at effectively 1% efficiency would make for better moderator experiences, but Facebook would be understandably reluctant to go from 5% revenue content moderation budget to 50% revenue content moderation.
It should cost some amount of money to post anything online on any social media platform: pay to post a tweet, article, image, comment, message, reply.
(Incidentally, crypto social networks have this by default simply due to constraints in how blockchains work.)
> The images and videos including necrophilia, bestiality and self-harm caused some moderators to faint, vomit, scream and run away from their desks, the filings allege.
You might have heard the saying, common in policy and mechanism design: "Show me the incentives, and I'll show you the outcome."
If you want to reduce spam, you increase the marginal cost of posting spam until it stops. In general if you introduce a small cost to any activity or service, the mere existence of the cost is often a sufficient disincentive to misuse.
But, you can think through implications for yourself, no? You don't need me to explain how to think about cause and effect? You can, say, think about examples in real life, or in your own town or building, where a service is free to use compared to one that has a small fee attached, and look at who uses what and how.
"Change" in itself is not a virtue. What I think you want is good or beneficial change? That said, what evidence do you have that poor people specifically are catalysing positive change online?
> This is a great idea to prevent bots, but that’s not who posts the bad stuff this thread is talking about.
There is no difference between a bot and a human as far as a network is concerned. After all, bots are run by humans.
The article specifically says that: "The images and videos including necrophilia, bestiality and self-harm caused some moderators to faint, vomit, scream and run away from their desks, the filings allege."
> Is that your goal?
Simply making it cost something to post online will mean that people who want to post spam can directly pay for the mental healthcare of moderators who remove their content.
If it turns out that you can find a group of people so poor that they simultaneously have valuable things to say online yet can't afford to post, then you can start a non-profit or foundation to subsidize "poor people online". (Hilariously, the crypto-bros do this when they're trying to incentivize use of their products: they set aside funds to "sponsor" thousands of users to the tune of tens of millions of dollars a year in gas refunds, airdrops, rebates and so forth.)
We would probably disagree on what change we think is beneficial, but in terms of catalyzing the changes I find appealing, I see plenty of it myself. I'm not sure how I could dig up a study on something like this, but I'm operating on the assumption that more poor people would advocate the changes I'm interested in than rich, because the changes I want would largely be intended to benefit the former, potentially at the expense of the latter. I see this assumption largely confirmed in the world. That's why I find the prospect of making posting expensive threatening to society's capacity for beneficial change. The effect depends on what model you use to price social media use, how high you set the prices, how you regulate the revenue, etc, but I think the effect needs to be mitigated. In essence, my primary concern with this idea is that it may come from an antidemocratic impulse, not a will to protect moderators. If you don't possess that impulse, then I'm sorry to be accusing you of motives you don't possess, and I'll largely focus on the implementation details that would best protect the moderators while mitigating the suppression of discourse.
>you can start a non-profit or foundation to subsidize "poor people online".
Where are all the foundations helping provide moderator mental health treatment? This is a pretty widely reported issue; I'd expect to see wealthy benefactors trying to solve it, yet the problem remains unsolved. The issue, I think, is that there isn't enough money or awareness to go around to solve all niche financially-addressable problems. Issues have to have certain human-interest characteristics, then be carefully and effectively framed, to attract contributions from regular people. As such, I wouldn't want to artificially create a new problem, where poverty takes away basically the only meaningful voice a regular person has in the modern age, then expect somebody to come along and solve it with a charitable foundation. Again, if charity is this effective, then let's just start a foundation to provide pay and care to moderators. Would it attract contributions?
>the crypto-bros do this when they're trying to incentivize use of their products
The crypto-bros trying to incentivize use of their products have a financial incentive to do so. They're not motivated by the kindness of their own hearts. Where's the financial incentive to pay for poor people to post online?
>There is no difference between a bot and a human as far as a network is concerned. After all, bots are run by humans.
Most implementations of this policy would largely impact bot farms. If posts cost money, then there's a very big difference in the cost of a botnet and a normal account. Costs would be massively higher for a bot farm runner, and relatively insubstantial for a normal user. Such a policy would then most effectively suppress bots, and maybe the most extreme of spammers.
What I don't understand, then, is the association between bots/spammers and the shock garbage harming moderators. From what I know, bots aren't typically trying to post abuse, but to scam or propagandize, since they're run by actors either looking for a financial return or to push an agenda. If the issue is spammers, then I'd question whether that's the cause of moderator harm; I'd figure as soon as a moderator sees a single gore post, the account would get nuked. We should expect then that the harm is proportionate to the number of accounts, not posts.
If the issue is harmful accounts in large quantity, and easy account creation, then to be effective at reducing moderator harm, wouldn't you want to charge a large, one-time fee at account creation? If it costs ten dollars to make an account, bad actors would (theoretically) be very hesitant to get banned (even though in practice this seems inadequate to, e.g., suppress cheating in online games). I'd also be relatively fine with such a policy; nearly anyone could afford a single 5-10 usd fee for indefinite use, but repeat account creators would be suppressed.
>Simply making it cost something to post online will mean that people who want to post spam can directly pay for the mental healthcare of moderators who remove their content.
I don't think that adding a cost to the posts will end up paying for mental healthcare without careful regulation. The current poor treatment of moderators is a supply-demand issue, it's a relatively low-skill job and people are hungry, so you can treat them pretty bad and still have a sufficient workforce. They are also, if I'm correct, largely outsourced from places with worse labor protections. This gives the social media companies very little incentive to pay them more or treat them better.
An approach that might help is something like this: Require companies to charge a very small set amount to make each individual post, such that a normal user may pay in the realm of 5 usd in a month of use, but a spammer or bot farm would have to spend vastly more. Furthermore, but very important, require that this additional revenue be spent directly on the pay or healthcare of the moderation team.
In reality, though, I'd be very worried that this secondary regulation wouldn't enter or make it through a legislature. I'm also concerned that the social media companies would be the ones setting the prices. If such a cost became the norm, I expect that these companies would implement the cost-to-post as a subscription to the platform rather than a per-post price. They would immediately begin to inflate their prices as every subscription-based company currently does to demonstrate growth to shareholders. Finally, they'd pocket the gains rather than paying more to the moderators, since they have absolutely zero incentive to do anything else. I think this would cause the antidemocratic outcomes I'm concerned with.
My question for you, then, is whether you'd be interested in government regulation that implements a flat per-post or per-account-creation fee, not much more than 5usd monthly or 10usd on creation, not adjustable by the companies, and with the requirement that its revenue be spent on healthcare and pay for the moderation team?
To clarify:
> That's why I find the prospect of making posting expensive threatening to society's capacity for beneficial change.
I suggested making it cost something. "Expensive" is a relative term and for some reason you unjustifiably assumed that I'm proposing "expensive", however defined. Incentive design is about the marginal cost of using a resource, as you later observed when you suggested $5.
We often observe in real life (swimming pools, clubs, public toilets, hiking trails, camping grounds) that introducing a trivial marginal cost often deters bad actors and free-loaders[^0]. It's what's referred to in ideas such as "the tragedy of the commons".
> An approach that might help is something like this: Require companies to charge a very small set amount to make each individual post
Yes that's a marginal cost, which is what I suggested. So basically, we agree. The rest is implementation details that will depend on jurisdiction, companies, platforms and so forth.
> I don't think that adding a cost to the posts will end up paying for mental healthcare without careful regulation.
Without data or case studies to reference, I can't speculate about that and other things that are your opinions but thank you for thinking about the proposal and responding.
> Where are all the foundations helping provide moderator mental health treatment? This is a pretty widely reported issue; I'd expect to see wealthy benefactors trying to solve it, yet the problem remains unsolved.
I don't mean to sound rude but have you tried to solve the problem and start a foundation? Why is it some mysterious wealthy benefactor or other people who should solve it rather than you who cares about the problem? Why do you expect to see others and not yourself, solving it?
Raising funds from wealthy people for causes is much easier than people imagine.
---
You're right that fundamentally there's an imbalance between the absolute mass of people producing the garbage, and the few moderators dealing with it. But we also don't have an option to just cut everyone's internet.
Designing platforms and business models that inherently produce less of the nasty stuff could help a lot. But even if/when we get there, we'll need automated mechanisms to ask people if they really want to be jerks, absolutely need to send their dick picks, or let people deal with sheer crime pics without having to look at them more than two seconds.
That is why this type of work will not go away.
And AI is just not good enough to do this, I fully agree.
I will go ahead and assume that on the wild/carefree time of PornHub, when anyone could be able to upload anything and everything, from what that lady said, the numbers of pedophilia videos, bestiality, etc. was rampant.
Laila Mickelwait is a director at Exodus Cry, formerly known as Morality in Media (yes, that's their original name). Exodus Cry/Morality in Media is an explicitly Christian organization that openly seeks to outlaw all forms of pornography, in addition to outlawing abortion and many gay rights including marriage. Their funding comes largely from right-wing Christian fundamentalist and fundamentalist-aligned groups.
Aside from the fact that she has an axe to grind, both she (as an individual) and the organization she represents have a long history of misrepresentating facts or outright lying in order to support their agenda. They also intentionally and openly refer to all forms of sex work (from consensual pornography to stripping to sexual intercourse) as "trafficking", against the wishes of survivors of actual sex trafficking, who have extensively document why Exodus Cry actually perpetuates harm against sex trafficking victims.
> everything, from what that lady said, the numbers of pedophilia videos, bestiality, etc. was rampant.
This was disproven long ago. Pornhub was actually quite good about proactively flagging and blocking CSAM and other objectionable content. Ironically (although not surprisingly, if you're familiar with the industry), Facebook was two to three orders of magnitude worse than Pornhub.
But of course, Facebook is not targeted by Exodus Cry because their mission - as you can tell by their original name of "Morality in Media" - is to ban pornography on the Internet, and going after Facebook doesn't fit into that mission, even though Facebook is actually way worse for victims of CSAM and trafficking.
I have a throwaway Facebook account. In the absence of any other information as to my interests, Facebook thinks I want to see flat earth conspiracy theories and CSAM.
When I report the CSAM, I usually get a response that says "we've taken a look and found that this content doesn't go against our Community Standards."
Why would pay in different countries be equivalent? Pretty sure FB doesn’t even pay the same to their engineers depending on where in the US they are, let alone which country. Cost of living dramatically differs.
Because it's exploitative otherwise. It's just exploiting the fact that they're imprisoned within borders.
An adjacent question is whether there are too many people on the planet for that to be an option anymore even if it were legal.
Do you mean for everyone to be hunter-gatherers? Yes, that would be impossible. If you mean for a smaller number then it depends on the number.
How would land allocation work without "enclosure of the commons"? Does it just become a free-for-all? What happens if you want to use the land for grazing but someone else wants it for growing crops? "enclosure of the commons" conveniently solves all these issues by giving exclusive control to one person.
“There is no ethical X under capitalism” is not license to stick our heads in the sand and continue to consume without a second thought for those who are being exploited. It’s a reminder that things need to change, not only in all the little tiny drop-in-a-bucket ways that individuals can afford to contribute.
That has nothing to do with the ethics of capitalism, though. The poor becoming poorer and the rich becoming richer is not a foregone conclusion of a capitalist society, nor is it guaranteed not to happen in a non-capitalist society.
>they're imprisoned within borders
What's the implication of this then? That we remove all migration controls?
But the thing is she's not gonna say "no", she would never say "no" because of the implication.
Why 8 times less?
Note that I'm equating all jobs here. No amount of compensation makes it worth seeing horrible things. They are separate variables.
I would hope not.
My point is, job shittiness can be priced in.
> just do it for one day and then return to your old job. --> Cool mill in the bank and dreadful images in your head. Perhaps Apitman feels he has enough cash and wont be happier with a million (more?).
Also your point is true but lacks of Facebook interest to elevate that number. I guess it was more a theorical reflexion than an argument for concrete economie.
Median US salary is about $59k. Gross national income (not an identical measure but close) in Kenya about $2.1k.
1/8th is disproportionately in favor of the contractors, relative to market.
If the moderators in Kenya are providing the same quality labor as those from the US, why the difference in price of their labor?
I have a friend who worked for FAANG and had to temporarily move from US to Canada due to visa issues, while continuing to work for the same team. They were paid less in Canada. There is no justification for this except that the company has price setting power and uses it to exploit the sellers of labor.
Btw, it’s probably a different team handling copyright claims, but my run-in with Meta’s moderation gives me the impression that they’re probably horrifically understaffed. I was helping a Chinese content creator friend taking down Instagram, YouTube and TikTok accounts re-uploading her content and/or impersonating her (she doesn’t have any presence on these platforms and doesn’t intend to). Reported to TikTok twice, got it done once within a few hours (I was impressed) and once within three days. Reported to YouTube once and it was handled five or six days later. No further action was needed from me after submitting the initial form in either case. Instagram was something else entirely; they used Facebook’s reporting system, the reporting form was the worst, it asked for very little information upfront but kept sending me emails afterwards asking for more information, then eventually radio silence. I sent follow-ups asking about progress, again, radio silence. Impersonation account with outright stolen content is still up till this day.
If given control, I think many/most people would opt into a significant amount of censorship.
And it was only for iCloud synced photos.
This was also during a time where Apple hadn’t pushed out e2ee for iCloud, so it didn’t even make sense.
If you are storing your data in a large commercial vendor, assume a state actor is scanning it.
"$service_provider sent a tip to NCMEC" or "uploaded a known-to-NCMEC hash", ranging from GMail, Google Drive, iCloud, and a few others.
https://www.missingkids.org/cybertiplinedata
"In 2023, ESPs submitted 54.8 million images to the CyberTipline of which 22.4 million (41%) were unique. Of the 49.5 million videos reported by ESPs, 11.2 million (23%) were unique."
This is analogous to the police's use of genealogy and DNA data to narrow searches for murderers, who they then collected evidence on by other means. There's is risk there, but (at least in the US) you aren't going to find a lot of supporters of the anonymity of serial killers and child abusers.
There are counter-arguments to be made. Germany is skittish about mass data collection and analysis because of their perception that it enabled the Nazi war machine to micro-target their victims. The US has no such cultural narrative.
I wouldn't be so sure.
When Apple was going to introduce on-device scanning they actually proposed to do it in two places.
• When you uploaded images to your iCloud account they proposed scanning them on your device first. This is the one that got by far the most attention.
• The second was to scan incoming messages on phones that had parental controls set up. The way that would have worked is:
1. if it detects sexual images it would block the message, alert the child that the message contains material that the parents think might be harmful, and ask the child if they still want to see it. If the child says no that is the end of the matter.
2. if the child say they do want to see it and the child is at least 13 years old, the message is unblocked and that is the end of the matter.
3. if the child says they do want to see it and the child is under 13 they are again reminded that their parents are concerned about the message, again asked if they want to view it, and told that if they view it their parents will be told. If the child says no that is the end of the matter.
4. If the child says yes the message is unblocked and the parents are notified.
This second one didn't get a lot of attention, probably because there isn't really much to object to. But I did see one objection from a fairly well known internet rights group. They objected to #4 on the grounds that the person sending the sex pictures to your under-13 year old child sent the message to the child, so it violates the sender's privacy for the parents to be notified.
The cultural narrative is actually extremely popular in a 10% subset of the population that is essentially fundamentalist christian who are terrified of the government branding them with "the mark of the beast".
The problem is that their existence actually poisons the discussion because these people are absurd loons who also blame the gays for hurricanes and think the democrats eat babies.
Curious, I did not realize Apple sent themselves a copy of all my data, even if I have no cloud account and don't share or upload anything. Is that true?
Other service providers can scan for CSAM from the cloud, but Apple cannot. So Apple might be one of the largest CSAM hosts in the world, due to this 'feature'.
I thought the topic was on-device scanning? The great-grandparent claim seemed to be that Apple had to choose between automatically uploading photos encrypted and not scanning them, vs. automatically uploading photos unencrypted and scanning them. The option for "just don't upload stuff at all, and don't scan it either" was conspicuously absent from the list of choices.
Why, do other phone manufacturers do this auto-upload-and-scan without asking?
Apple itself has other options, of course. It could offer encrypted or unencrypted storage without any kind of scanning, but has made the choice that it wants to actively check for CSAM in media stored on its servers.
With the (optional) iCloud, you can (optionally) activate iCloud Photos to have a photo library backed up in the cloud and shared among all your devices (and, in particular, with only thumbnails and metadata stored locally and the full resolution pictures only downloaded on demand).
These are always encrypted, with either the keys being with Apple ("Standard Data Protection") so that they're recoverable when the user loses phone or password, or E2E ("Advanced Data Protection") if the user so choses, thus irrecoverable.
It seems to me that in the latter case images are not scanned at all (neither on device nor in the cloud), and it's unclear to me whether they're scanned in the former case.
Yet another reason why my next phone will be an android.
That’s a very very different issue.
I support big tech using AI models running on their own servers to detect CSAM on their own servers.
I do not support big tech searching devices they do not own in violation of the wishes of the owners of those devices, simply because the police would prefer it that way.
It is especially telling that iCloud Photos is not end to end encrypted (and uploads plaintext file content hashes even when optional e2ee is enabled) so Apple can and does scan 99.99%+ of the photos on everyone’s iPhones serverside already.
They do this today. https://www.apple.com/child-safety/pdf/Expanded_Protections_...
Every photo provider is required to report CSAM violations.
https://forums.appleinsider.com/discussion/238553/apple-sued...
It hasn’t been redefined. The legal definition of it in the UK, Canada, Australia, New Zealand has included computer generated imagery since at least the 1990s. The US Congress did the same thing in 1996, but the US Supreme Court ruled in the 2002 case of Ashcroft v Free Speech Coalition that it violated the First Amendment. [0] This predates GenAI because even in the 1990s people saw where CGI was going and could foresee this kind of thing would one day be possible.
Added to that: a lot of people misunderstand what that 2002 case held. SCOTUS case law establishes two distinct exceptions to the First Amendment – child pornography and obscenity. The first is easier to prosecute and more commonly prosecuted; the 2002 case held that "virtual child pornography" (made without the use of any actual children) does not fall into the scope of the child pornography exception – but it still falls into the scope of the obscenity exception. There is in fact a distinct federal crime for obscenity involving children as opposed to adults, 18 USC 1466A ("Obscene visual representations of the sexual abuse of children") [1] enacted in 2003 in response to this decision. Child obscenity is less commonly prosecuted, but in 2021 a Texas man was sentenced to 40 years in prison over it [2] – that wasn't for GenAI, that was for drawings and text, but if drawings fall into the legal category, obviously GenAI images will too. So actually it turns out that even in the US, GenAI materials can legally count as CSAM, if we define CSAM to include both child pornography and child obscenity – and this has been true since at least 2003, long before the GenAI era.
[0] https://en.wikipedia.org/wiki/Ashcroft_v._Free_Speech_Coalit...
[1] https://www.law.cornell.edu/uscode/text/18/1466A
[2] https://www.justice.gov/opa/pr/texas-man-sentenced-40-years-...
If you look at the question from an originalist viewpoint: did the legislators who drafted the First Amendment, and voted to propose and ratify it, understand it as an exceptionless absolute or as subject to reasonable exceptions? I think if you look at the writings of those legislators, the debates and speeches made in the process of its proposal and ratification, etc, it is clear that they saw it as subject to reasonable exceptions – and I think it is also clear that they saw obscenity as one of those reasonable exceptions, even though they no doubt would have disagreed about its precise scope. So, from an originalist viewpoint, having some kind of obscenity exception seems very constitutionally justifiable, although we can still debate how to draw it.
In fact, I think from an originalist viewpoint the obscenity exception is on firmer ground than the child pornography exception, since the former is arguably as old as the First Amendment itself is, the latter only goes back to the 1982 case of New York v. Ferber. In fact, the child pornography exception, as a distinct exception, only exists because SCOTUS jurisprudence had narrowed the obscenity exception to the point that it was getting in the way of prosecuting child pornography as obscene – and rather than taking that as evidence that maybe they'd narrowed it a bit too far, SCOTUS decided to erect a separate exception instead. But, conceivably, SCOTUS in 1982 could have decided to draw the obscenity exception a bit more broadly, and a distinct child pornography exception would never have existed.
If one prefers living constitutionalism, the question is – has American society "evolved" to the point that the First Amendment's historical obscenity exception ought to jettisoned entirely, as opposed to merely be read narrowly? Does the contemporary United States have a moral consensus that individuals should have the constitutional right to produce graphic depictions of child sexual abuse, for no purpose other than their own sexual arousal, provided that no identifiable children are harmed in its production? I take it that is your personal moral view, but I doubt the majority of American citizens presently agree – which suggests that completely removing the obscenity exception, even in the case of virtual CSAM material, cannot currently be justified on living constitutionalist grounds either.
If Apple isn’t free to publish e2ee software for mass privacy without the government demanding they backdoor it for cops on threat of retaliation, then we don’t have first amendment rights in the USA.
Apple does indeed in theory have a right to release whatever iOS features they like. In practice, they do not.
Everyone kind of tacitly acknowledged this, when it was generally agreed upon that Apple was doing the on-device scanning thing "so they can deploy e2ee". The quiet part is that if they didn't do the on-device scanning and released e2ee software without this backdoor (which would then thwart wiretaps), the FBI et al would make problems for them.
User privacy. Almost nobody trades in CSAM, but everyone deserves privacy.
Honestly, this isn’t about CSAM at all. It’s about government surveillance. If strong crypto e2ee is the hundreds-of-millions-of-citizens device default, and there is no backdoor, the feds will be upset with Apple.
This is why iCloud Backup (which is a backdoor in iMessage e2ee) is not e2ee by default and why Apple (and the state by extension) can read all of the iMessages.
> Honestly, this isn't about CSAM at all.
It literally is the only thing the technology is any good for.
People designed images that were FPs of real images. So apps like WhatsApp that auto-save images to photo albums could cause people a big headache if a contact shared a legal FP image.
If the automated process flags something as a potential hit, then they, the humans, would then review those images to verify. Every image/video that is discovered to be a hit is also inserted into a larger dataset as well. I can't remember if the Feds have their own DB (why wouldn't they?), but the National Center for Missing and Exploited Children run a database that I believe government agencies use too. Not to mention, companies like Dropbox, Google, etc.. all has against the database(s) as well.
If it’s already bad blurred or as a thumbnail don’t click on the real thing.
`select * from posts where author_id in @follow_ids order by date desc`
At least 90% of the ills of social media are caused by using algorithms to prioritize content and determine what you're shown. Before these were introduced, you just wouldn't see these types of things unless you chose to follow someone who chose to post it, and you didn't have people deliberately creating so much garbage trying to game "engagement".
> select * from posts where author_id in @follow_ids order by date desc
SELECT post FROM posts JOIN follows ON posts.author_id = follows.author_id WHERE follows.user_id = $session.user_id;
Why would that be?
Currently content is flagged and moderators decide whether to take it down. Using AI, it's easy conceive a process where some uploaded content is preflagged requiring an appeal (otherwise it's the same as before, a pair of human eyes automatically looking at uploaded material).
Uploaders trying to publish rule-breaking content would not bother with an appeal that would reject them anyway.
I already addressed this and you're talking over it. Why are you making the assumption that AI == no appeal and zero staff? That makes zero sense, one has nothing to do with the other. The human element comes in for appeal process.
You didn't address it, you handwaved it.
> Why are you making the assumption that AI == no appeal and zero staff?
I explicitly stated the reason -- it is cheaper and it will work for the majority of instances while the edge cases won't result in losing a large enough user base that it would matter to them.
I am not making assumptions. Google notoriously operates in this fashion -- for instance unless you are a very popular creator, youtube functions like that.
> That makes zero sense, one has nothing to do with the other.
Cheaper and mostly works and losses from people leaving are not more than the money saved by removing support staff makes perfect sense and the two things are related to each other like identical twins are related to each other.
> The human element comes in for appeal process.
What does a company have to gain by supplying the staff needed to listen to the appeals when the AI does a decent enough job 98% of the time? Corporations don't exist to do the right thing or to make people happy, they are extracting value and giving it to their shareholders. The shareholders don't care about anything else, and the way I described returns more money to them than yours.
Their copyright takedown system has been around for many years and wasn't contingent on AI. It's a "take-down now, ask questions later" policy to please the RIAA and other lobby groups. Illegal/abuse material doesn't profit big business, their interest is in not having it around.
You deliberately conflated moderation & appeal process from the outset. You can have 100% AI handling of suspect uploads (for which the volume is much larger) with a smaller staff handling appeals (for which the volume is smaller), mixed with AI.
Frankly if your hypothetical upload is still rejected after that, it 99% likely violates their terms of use, in which case there's nothing to say.
> it is cheaper
A lot of things are "cheaper" in one dimension irrespective of AI, doesn't mean they'll be employed if customers dislike it.
> the money saved by removing support staff makes perfect sense and the two things are related to each other like identical twins are related to each other.
It does not make sense to have zero staff in as part of managing an appeal process (precisely to deal with edge cases and fallibility of AI), and it does not make sense to have no appeal process.
You're jumping to conclusions. That is the entire point of my response.
> What does a company have to gain by supplying the staff needed to listen to the appeals when the AI does a decent enough job 98% of the time?
AI isn't there yet, notwithstanding, if they did a good job 98% of the time then who cares? No one.
So what? It could rely on tea leaves and leprechauns, it illustrates that whatever automation works will be relied on at the expense of any human staff or process
> it 99% likely violates their terms of use, in which case there's nothing to say.
Isn't that 1% the edge cases I am specifically mentioning are important and won't get addressed?
> doesn't mean they'll be employed if customers dislike it.
The customers on ad supported internet platforms are the advertisers and they are fine with it.
> You're jumping to conclusions. That is the entire point of my response.
Conclusions based on solid reason and evidenced by past events.
> AI isn't there yet, notwithstanding, if they did a good job 98% of the time then who cares? No one.
Until you realize that 2% of 2.89billion monthly users is 57,800,000.
Each government and in some cases provinces and municipalities should have teams to regulate content from their region?
Freedom of speech is between you and the government and not you and a private company.
As the saying goes, if don't like your speach I can tell you to leave my home, that's not censorship, that's how freedom works.
If I don't like your speach, I can tell yo to leave my property. Physical or virtual.
The first amendment is multiple freedoms. Your freedom of speech is that the government shouldn't stop you from using your own property to do something. You are free to print out leaflets and distribute them from your porch. If nobody wants to read your pamphlets that's too damn bad, welcome to the free market of ideas buddy.
The first amendment also protects Meta's right of free association. Forcing private companies to platform any content submitted to them would outright trample their right. Meta has a right to not publish your work so that they can say "we do not agree with this work and will not use our resources to expand it's reach".
We have, in certain cases, developed system that treats certain infrastructure as a regulated pipe that is compelled to carry everything, like with classic telephone infrastructure. The reason for that, is it doesn't make much sense to require every company to put up their own physical wires, it's dumb and wasteful. Social networks have zero natural monopoly and should not be treated as common carriers.
Furthermore, I’ll rather have the picture of my aunt’s vacation taken down by ai mistake rather than hundreds of people getting PSTD because they have to manually review if some decapitation was real or illustrated on an hourly basis.
Worth reading Zitron's essay if you haven't already. It sounds obvious, but the simple cataloging of all the indignities we take for granted builds up to a bigger condemnation than just Big Tech. https://www.wheresyoured.at/never-forgive-them/
I'll take a look at the essay, thanks.
https://apnews.com/article/kenya-facebook-content-moderation... - Facebook content moderators in Kenya call the work 'torture.' Their lawsuit may ripple worldwide
I found this one while looking for salary information on these Kenyan moderators. This article mentioned that they are being paid $429 per month.
> post-traumatic stress disorder caused by exposure to graphic social media content including murders, suicides, child sexual abuse and terrorism.
If you want a taste of the legal portion of theses just got to 4chan.org/gif/catalog and look for a "rekt", "war", "gore", or "women hate" thread. Watch every video there for 8-10 hours a day.
Now remember this is the legal portion of the content moderated as 4chan does a good job these days of removing illegal content mentioned in that list above. So all these examples will be a milder sample of what moderators deal with.
And do remember to browse for 8-10 hours a day.
edit: it should go without saying that the content there is deep in the NSFW territory, and if you haven't already stumbled upon that content, I do not recommend browsing "out of curiosity".
What drew you to 4chan?
I have a kid now and my plan is to not give her a smartphone/social media till she’s 16 and heavily monitor internet access until she’s atleast 12. Obviously I can’t control what she will see with friends but she goes to a rigorous school and I’m hoping that will keep her busy. Other than that I’m hoping the government comes down hard on social media access to kids/teenagers and all the restrictions are legally codified by the time she’s old enough.
Could it be:
- the fact that moderators are hired and paid
- that kids are young and a lot more tolerant
- that moderators aren't intended audiences
- backgrounds, sensitivity in media at all
- the amount, of disturbing images
- the amount, in total, not just bad ones
- anything else?
Personally, I'm suspecting that difference in exposure to _any kind of media_ might be a factor; I've come across stories online that imply visiting and staying at places like Tokyo can almost drive people crazy, from the amount of stimuli alone.Doesn't it sound a bit too shallow and biased to determine it was specifically CSAM or whatever specific type of data that did it?
I also have a relative who is stuck with their profile being unable to change any contact details, neither email nor password because FB account center doesn't open for them. Again, there is no human support.
BigTech companies must be mandated by law to have the number of live support people working and reachable that is a fixed fraction of their user number. Then, they would have no incentive to inflate their user numbers artificially. As for the moderators, there should also be a strict upper limit on the number of content (content tokens, if you will) they should view during their work day. Then the companies would also be more willing to limit the amount of content on their systems.
Yeah, it's bad business for them but it's a win for the people.
The problem is, someone has to do it. These platforms are mandated by law to moderate it or else they're responsible for the content the users post. And the companies can not shield their employees from it because the work simply needs doing. I don't think we can really blame the platforms (though I think the remuneration could be higher for this tough work).
The work tends to suit some people better than others. The same way some people will not be able to be a forensic doctor doing autopsies. Some have better detachment skills.
All the people I know that do this work have 24/7 psychologists on site (most of them can't work remotely due to the private content they work with). I do notice though that most of them do have an "Achilles heel". They tend to shrug most things off without a second thought but there's always one or two specific things or topics that haunt them.
Hopefully eventually AI will be good enough to deal with this shit. It sucks for their jobs or course but it's not the kind of job anyone really does with pleasure.
Also, it'll make social media even more fake than it already is. Everyone trying to be as fake as possible. Just like LinkedIn is now. It's sickening, all these people toting the company line. Even though they do nothing but complain when you speak to them in person.
And I don't think it'll actually solve the problem. People find ways to get through the validation with fake IDs.
Sure some companies would outsource also to africa but it doesn't mean this work is only available to third-world countries. And there's not that many jobs in it. It's more than possible to be able to find enough people that can stomach it.
There was another article a few years back about the poor state of mental health of Facebook moderators in Berlin. This is not exclusively a poor people problem. More of a wrong people for the job problem.
And of course we should look more at why this is the only form of social mobility for them if it's really the case.
The sociopaths of the world will forever be coming up with new and god-awful types of content to post online, which current AI moderators haven't encountered before and which therefore won't know how to classify. It will therefore be up to humans to label that content in order to train the models to handle that new content, meaning humans will have to view it (and suffer the consequences, such as PTSD). The alternative, where AI labels these new images and then uses those AI-generated labels to update the model, famously leads to "model collapse" [1].
Short of banning social media at a societal level, or abstaining from it at an individual level, I don't know that there's any good solution to this problem. These poor souls are taking a bullet for the rest of us. God help them.
The nature of the job really sucks. This is not unusual; there are lots of sucky jobs. So my concern is really whether the employees were informed what they would be exposed to.
Also I’m wondering why they didn’t just quit. Of course the answer is money, but if they knew what they were getting into (or what they were already into), and chose to continue, why should they be awarded more money?
Finally, if they can’t count on employees in poor countries to self-select out when the job became life-impacting, maybe they should make it a temporary gig, eg only allow people to do it for short periods of time.
My out-of-the-box idea is: maybe companies that need this function could interview with an eye towards selecting psychopaths. This is not a joke; why not select people who are less likely to be emotionally affected? I’m not sure anyone has ever done this before and I also don’t know if such people would be likely to be inspired by the images, which would make this idea a terrible one. My point is find ways to limit the harm that the job causes to people, perhaps by changing how people interact with the job since the nature of the job doesn’t seem likely to change.
(I'm not saying that I'd assemble a high-speed yacht full of commandos, who travel around the world, righting wrongs when no one else can. Though that would be more compelling content than most streaming video episodes right now. So you could offset the operational costs a bit.)
Switch to a federated social systems like Mastodon, with only a few thousand or ten thousand users per instance, and perpetrators will never be able to grow too large. Easy for the moderators to shut stuff down very quickly.
From a normal person perspective on not seeing bad stuff, the design of a federated network is inherently better than a global network.
On Mastodon, admins will just have to do the additional work to make sure new accounts are not posting weird stuff.
The #2 and #3 most popular Mastodon instances allow CSAM.
Not everyone can be a forensic investigator or coroner, too.
I know lots of people who can and do look at horrible pictures on the internet and have been doing so for 20+ years with no ill effects.
I’m not talking about obvious “scream and run away” reaction here. One may think that it doesn’t affect them or people on the internet, but then it suddenly does after they binge it all day for a year.
The fact that not less than 100% got PTSD should be telling something here.
Some people come out with no problems, while their trenchmate facing almost identical situations suffers for the rest of their lives.
In this case, the claim is that "it traumatised 100% of hundreds of former moderators tested for PTSD … In any other industry, if we discovered 100% of safety workers were being diagnosed with an illness caused by their work, the people responsible would be forced to resign and face the legal consequences for mass violations of people’s rights."
Do those people you know look at horrible pictures on the internet for 8-10 hours each day?