XCheck at Meta: Why it exists and how it works
blog.nindalf.com
blog.nindalf.com
The fact that moderation is a hard problem is orthogonal. The fact remains that there are different rules for different people, both for flagging and for content. The fact that there is more nuance than "two sets of rules" doesn't really change the fundamental problem that private companies now get to arbitrarily police publication by normal people.
I have the distinct impression that in 100 years the default mode is going to be "if it's legal you can post it, if it's not you can't, and if you have a problem with the boundary, take it up with the lawmaking authority and not us". We will look back at this time at the emergence of any-to-any publishing like the dawn of radio or TV or newspaper where a few rich guys got to decide what could or could not be published.
Do you really want pictures like this everywhere on Instagram? So now we have a no sex acts rule. But then an art museum is hosting an exhibition of art, and that art is depicting sex. Maybe even pedophilic sex, like a vase in the Met that has an old man stroking a young teenagers hairless genitals on it. That's very likely child pornography, except that the law can't ban images of artistic value, and the image of the vase has a lot of value, and I doubt anyone would find it erotic. Should the Met not be allowed to use a picture of a bunch of vases to promote its exhibitions?
There's lots of reasons why reasonable seeming results are not possible from simple rules, and why it's legal, let's have it is not a great rule. Having a moderation system that addresses e.g. art studios and museums differently from other people is quite sensible.
Also in practice I think this is covered for %99 of cases people via an NSFW & NSFL tag system. That would be an NSFW art and thats that.
The children issue will boil down to parents are the real root user of their children's accounts and parents will have control over the tagging set for moderation until they are of age.
IMO I think most people are fine with mandatory labeling of goods, and actually welcome it because it gives more control and informed consent. It's the mandatory censorship that become the issue.
Abusive false labelling might become the next hot topic, but that is far smaller of a surface area.
So in your world, say the same person posts biased national political opinions and tags it with #city. Unless someone is moderating that tag, the tag becomes absolutely useless. It seems like how things work today, just with extra steps?
Also a big part of this is matter of scale and choice. I don't think people care that much about a small subreddit and their bullshit most of the time, or petty facebook group politics. It's when the platform itself does the censoring, and when the platform is where %95 of content exists due to network effects and has user counts that are in the billions that it starts having a duty of neutrality. It's much like city roads or housing or similar, the people you filter out from housing and city roads have to be filtered out for dangerous damage to the infrastructure or people itself and that is it, otherwise you get into really bad political and discriminatory territory when you say black people or people of opinion Y can't buy housing here or drive on the roads.
In this future system, you can also have alternative tagging systems, credit ratings for moderators, published AI models that tag client side, etc. I think it would be a very interesting marketplace of services!
It’s a disturbing, self-destructive act and I’d rather I’d never had seen it.
It’s just very different, but posed in a way that is simultaneously provocative due to the sexual nature yet mundane given how different the sexual act is.
I also don’t see any artistic merit in the work whatsoever, but you’re welcome to your own views and predilections. I just wish there’d been a more accurate warning so I could avoid it.
Specifically addressing my argument it’s not self mutilation is that this is the sort of thing that if they stoped doing it today would “heal”. Elasticity would change and the body would slowly return to normal. Which is why i specifically called out the gauged ear piercings. When you turn a 1mm hole in your ear to say, a 25mm hole (large but not extreme) that’s not really “self mutilation” when compared to people that deliberately undergo processes for artistic scar patterns or to use another ear example, have shapes cut out of the ear cartilage. The gauge piercing will heal, so would this and there are vastly more shocking things people do in the body modification world… many of which id say it was entirely fair to judge as self mutilation… like sclera eye “tattooing”… having something done to you that has a high potential to impart your vision or even blind you for the sake of achieving a specific artistic look on your face feels like a much more clear cut case of self mutilation than this particular photographed act.
Not Safe For the Limp is just variant of Not Safe for the Weak. They both communicate that if you aren't strong, stay away.
> Or, perhaps we need a “4chan and similarly tasteless degenerates” tag.
It's just an arbitrary grid of pixels. Either you are strong enough to handle viewing arrangements of pixels or you aren't. The weak have been warned, the strong have no concerns. No need to overcomplicating things.
As all things, this should be determined by subscription to content. I mean if i subscribe to CNN news and CNN stars posting dick picks or gore. I unsubscribe and thats the end of dick picks for me.
I should not see things that i don't subscribe to same way as i should not be presented with web pages that i do not explicitly go to by clicking on a link or entering url address.
It's on me if i go to 4chan. And same thing goes i opposite direction. If i like dick pics i should be able to subscribe to all of them and not be judged for that.
Source for the curious?
Correspondingly, I think your argument is a strawman.
The problem isn't technical, it's legal - you can't build a system that operates one way and then publicly misrepresent it.
And any system which involves reporting from the public will need to build a "this account gets lots of false reports of category X, to save time assume they're all false" system. As well as "all the reports made by this user are bogus".
It’s not likely to be more than a nuisance to defend and not terribly expensive to settle, but an enterprising plaintiff’s lawyer could probably find a cause of action that passes summary judgment.
Isn't that literally what Twitter does as well? Making exceptions for Politicians and Government Representatives? Or does HN's bias towards Twitter exempt it from any form of scrutiny?
What a completely bizarre comment.
No one mentioned Twitter, there is no "HN" general viewpoint, and if I had to say I'd say most comments on HN about Twitter are negative.
I'd say this could be the worst case of "Whataboutism" I've ever seen, but it is such a weird thing to use "what about" regarding.
Let me put it this way: what you feel Meta is doing in the West, is what many in India (like me) feel Twitter is doing here. And the sentiment I see is mostly anti Meta and mostly pro Twitter here.
After all it is my perspective and I could be wrong (as I obviously don't have statistics to say if HN definitely has a Twitter bias or not). But I believe I have a right to express my opinion on what I feel is HN sentiment towards big tech censorship (which mostly circles around Meta but rarely around Twitter).
The OP is a very strong BJP supporter (see comment history), but even so it's a weird take.
Vice is part of the same Leftist Media cabal.
> The OP is a very strong BJP supporter (see comment history), but even so it's a weird take.
LMFAO no. It is mostly about Twitter censoring accounts that are pro-Hindu: https://twitter.com/aranganathan72/status/132892246447524659...
And the ban was done after Opposition Party supporting Bureaucrat (from the IPS cadre) threatened to get the account banned. Basically Twitter acted on behalf of the Opposition not the Ruling Party. You can read the full story here [1].
If Twitter is truly unbiased, it wouldn't be siding so openly with Opposition Parties in India.
Quoting the user (TrueIndology) who was banned:
`She asked me for my personal details. I refused to divulge those details. She then said "Your time is up". And boom. My account was suspended within 5 minutes. Twitter sent no mail. Gave no reason. Simply suspended my account`.
The Government of India has authority to regulate Law and pass Executive Orders to entities operating within the Country. The Opposition of India has no such powers. Yet Twitter defied the Government's Executive Orders but did not defy the Opposition Party Bureaucrat's diktat. That should tell you how openly biased Twitter is.
Anyways, being a "strong BJP supporter" is not a crime on HN I presume. Where propaganda thrives it is better to be a strong, vocal supporter. Even if all alone and in minority.
[1]: https://www.opindia.com/2020/11/twitter-suspends-trueindolog...
> The TrueIndology account is very popular and has thousands of followers. The person behind the account is a meticulous fact-checker who corrects wrong and distorted claims often pushed as history mostly by left-liberals, and cites actual sources to debunk myths and leftist propaganda on Indian history. Though the reasons for the accoun[sic] suspension are not clear, it comes just a day after he had ‘fact-checked’ and had idulged[sic] in an online arguement[sic] with senior IPS officer D Roopa.
Spelling errors are reproduced as is.
It's surprising how closely the "left-liberals"/"leftist propaganda" rhetoric matches "Leftist Media cabal"/"Western media propaganda outlets" language of the OP.
I'll explain the context behind this because Vice (as is usual with Western media propaganda outlets) does not provide any details.
The "journalist" in question, who goes by the handle @zoo_bear, tweeted out a clip from a heated National TV debate between a ruling party spokesperson and another Islamic scholar. Some unsavory remarks were made by the Islamic scholar on the ruling party spokesperson's Religious beliefs (she is a Hindu). She retaliated in the TV debate with her own unsavory remarks on his Islamic beliefs.
Now this "journalist", instead of putting out the entire clip, decided to cut the clip to only show the spokesperson's remarks which went viral not just in India but across the World. She not only received beheading/death threats but was also forced to apologize, tender resignation from the Party and go into hiding. Then after that, 4 other Hindus (completely unconnected to this TV debate) were beheaded, by Islamic terrorists, as a "revenge" for what she said. Only later did the entire clip surface and things cooled down. But by then, the damage was done. Was any of this reported by Vice? Nope. This is the kind of propaganda that Western media outlets indulge in.
Now the Government of India wanted to take this out-of-context Tweet down (as well as suspend the "journalist"s account) as he continued to put out partial information just to keep the communal pot boiling.
India is a multi-cultural, multi-religious democracy with a billion+ people. Any riots that break out has potential to turn into communal clashes on a pan-India scale. The Government, unlike Western Governments, has extra responsibility to take care of the social fabric of the Nation apart from protecting the country from adversaries on our borders.
So the Government of India is perfectly justified in requesting take down. Twitter not taking it down is purely politically motivated. It is not like it hasn't taken down accounts/tweets at all. It has done so multiple times on behest of opposition party members.
This quote summarises the overall situation reasonably well:
> A 2021 transparency report released by Twitter revealed that India was the single largest source of government takedown requests in the second half of 2020—accounting for 25 percent of the global volume. The compliance rate for these requests was 0.6 percent in India, as opposed to 30 percent globally.
[1] https://indianexpress.com/article/cities/delhi/altnews-co-fo...
Where is this explanation? There is not even one mention of Nupur Sharma in the Vice article. Quote from the article.
> the tweet was over 4 years old when he was arrested for it
Nice deflection. This is not the tweet I was talking about.
I was talking about this: https://www.opindia.com/2022/06/islamist-mobs-nupur-sharma-p...
and this https://www.opindia.com/2022/06/alt-news-mohammed-zubair-del...
> The Indian article[1] about the arrest doesn't make any claims about the tweet going viral or being responsible for the things the OP is claiming here.
Of course the Indian Authorities are going to charge him for the most heinous/offensive tweet. And they will choose the one that will stick when they prosecute him in a Court of Law. That does not mean what he did, did not lead to beheading of 4 innocent Hindus. This is well documented fact that the communal tension was result of him sharing a doctored clip that went viral.
> The Indian article[1] about the arrest doesn't make any claims about the tweet going viral or being responsible for the things the OP is claiming here
LMFAO from Indian Express of all the papers? The ones who falsely published a report that Indian Army had moved towards Delhi to do a coup on the previous Government of India? [1] and [2].
Please refer to better sources.
> A 2021 transparency report released by Twitter revealed that India was the single largest source of government takedown requests in the second half of 2020—accounting for 25 percent of the global volume
Ah yes. And we should take Twitter at its word. Twitter can also reveal exactly what those tweets were which the GoI wanted taken down. I bet at least half of them involve some fabrication (like The Wire BS that is currently being propagated) or something communal to stoke riots and unrest in India.
[1]: https://www.businessinsider.com/indian-newspaper-slammed-for...
[2]: https://www.sundayguardianlive.com/news/upa-2-top-leaders-wa...
>> the tweet was over 4 years old when he was arrested for it
> Nice deflection. This is not the tweet I was talking about.
Good for you? How exactly was I supposed to know this when the tweet I was talking about was the one mentioned in the article you were so keen to dispute?
> LMFAO
Really?
“Senator, thank you for the question. We treat everyone equally. To be clear this means every single user on our platform is subject to the same terms of service they agreed to when signing up.”
So fine, if you want to define the words as meaning something they clearly don't, and want to try and fight that battle in the media... ok. But almost certainly the first thing that happens is something exactly like this - some nerdy engineering comes out and goes "Well no actually, we manually intervene for the people we like". And then you get to go back to Congress and explain to them why you clearly lied to them, and you're not going to get out of that by using Mark Zuckerberg's pure eye-watering levels of charisma.
That's opposed to an Id or different account types, where if you didn't register with a specific type, you cannot be granted that same exemption
Which (and I hate to ever sound likey I'm defending Facebook) is quite common in legal systems.
"Unequal treatment" would mean: these users are not subject to our policies. That's not the case here. XCheck just turns off the "we receive 100 reports, an algo shuts that account down" phenomenon. Humans review those cases manually, and the account can still be suspended or banned. How in the world could that be contentious?
* if your post gets reported 10 times, it gets taken down
* if your post gets reported (# of followers / 10) times, it gets taken down
* if your post gets reported (# of followers * 100) times, it gets taken down
Which of these are fair in your opinion? For an account with a huge # of followers, the last one effectively means their posts can't be taken down.
It seems a system like XCheck is a step function where at a certain point, you get exempt from certain checks altogether.
The "equality" here could refer to each account's potential to go through the vetting that would give them exempt status.
==============
Maybe this deserves a separate response, but another way to think of this is to compare it to income taxes. There are different income brackets that affect your marginal tax rate in the US. Is it equal for everyone to pay the same dollar amount in income taxes? The same rate? A progressive rate? Are people treated equally under the law in all these cases? In none of them?
Who said anything about fair? The only one that is "equal treatment" is the one where 10 reports takes them down. It's shitty policy, but that's a different question isn't it? Number of followers playing a role is where treatment stops being equal. I'm not arguing what's the right way to handle this, I'm just contradicting rhetoric aiming to justify their policy by fallacious argument.
Edited to add: I don't think tax code is meant to or treats people equally, even though they're all subject to the same rules. The rules explicitly divide taxpayers into categories with different treatment.
Xcheck is, by definition, at its very core, not "equal treatment". Equal treatment is easy and uncontroversial to define.
It may or may not be "fair treatment". That is always subjective and depends on goals and objectives and frameworks.
Something like Xcheck may or may not be necessary. It's interesting to read about problem it's trying to solve and discuss whether and how much it succeeds and at what price and what limitations.
What Xcheck does not get to do, however, is present / defend itself as "equal treatment",and in any meaningful discussion we should not mix and match or treat those as synonyms - that's confusing and unproductive at best, deceptive at worst.
What it empathically is not, by definition, is "equal treatment". It's existence is epitome of unequal treatment.
Same but different here.
"Equality" is an inherently unfair concept because not all humans come into the system with equal footing, or in equal conditions.
And so to simply, and blindly, treat everyone identically - what you mean to say is "equal" here - would be a bad approach.
Pedantically speaking, you can certainly argue that "equal" is not grammatically correct, but I think everyone else understands what's being said.
We may disagree on the last one - I find "everybody understands..." assumptions can range from mostly happen to be true to tragically incorrect to manipulatively deceptive. Based on other discussions in this thread, I'm not alone. The fact that you indicate diversity and inclusion has moved away from equal treatment phrase, that Equal vs Fair is now being taught as basic differentiation just like Pretend vs Real, and Opinion vs Fact in school, also I believe supports my claim these are meaningfully, substantially, importantly different concepts. Again, I think they get confused enough, intentionally or unintentionally, that "everybody understands" is just too dangerous of an approach, even if well meant.
XCheck is a system that supports tagging specific accounts with tags that exempt that account from certain integrity enforcement. In the post I give an example of a tag that used to exempt accounts from the fake account checkpoint. Applying these tags is usually a manual process and usually carefully vetted.
As of June 2021 no XCheck tag I knew of allowed a user to take down content at will.
To clarify, one of the requirement of this regulation is:
Under the new Intermediaries Rules, intermediaries must *complete* the takedown process under Section 79(3) of the IT Act, within 36 hours.
Very simply put, large social media platforms are legally required to respond and delete content when requested by government representatives.
Under such a scenario, can you confidently say that XCheck hasn't been extended with new features, like a tag that allows a user to immediately flag posts that must be compulsorily taken down when the particular user reports it? It seems a very obvious extension when you consider the political demands that are placed on Meta from around the world, and in India in particular.
At any sufficiently large scale - especially with a product you're not paying for - this is the best you're likely to get. Once you fully understand and embrace this, your stress levels about imperfection tend to go way down.
I was initially extremely sympathetic to the story presented by The Wire, because it's quite believable that Meta/FB would go to extreme lengths to try to distance themselves from such a situation, but the facts ... just ... don't add up. As Alex Stamos has noted, there is little debate regarding collusion between Meta's "government relations" people and policy groups, so that's not really much of a scandal -- it's not like Meta will deny that Modi was treated like a divine being when he visited their campus, so the idea that his people can easily call in a favor to crush a social media post won't really surprise anyone.
The bulletproof evidence with DKIM authentication and a video of a logged-in admin instance doesn't look so bulletproof after all, based on the credible reports from those who know how Meta's admin tooling actually looks and functions, and those with other DKIM authenticated emails from the fb.com domain.
So, the question is, what's the agenda here? Why would someone go through all of this effort, to create a scandal out of something that is not very far from the truth? What's the point of this entire thing? Maybe this is like Nick Denton and Gawker losing their entire business over the stupidest sex tape story ever; or maybe this is part of something else that requires domain knowledge regarding Indian politics and media to understand (do page views monetize so well that this mini-scandal is going to be super-profitable to The Wire? Highly doubtful, right? What could they possibly get out of this?).
The whole thing is just really weird. It's also a major distraction from the very real problems that Meta doesn't even try to hide. Meta's relationship with the governments in various countries -- including the United States -- is way too close for comfort, and absolutely toxic on multiple levels. If this story turns out to be fake news, it'll do a lot to help the company deny, deflect and discredit the next real scandal. I think this is what the wacky conspiracy theorists call "4D chess," but I don't think that's what's happening here.
The true story behind all of this is bound to be very strange, and very stupid.
The extreme lengths that these people seem to have gone to is shocking. Whatever it is, FB has the evidence on their servers as Alex Stamos points out since they created a Workspace instance.
My read is that he wants this story to be true so much that he’s ignoring evidence to the contrary.
As for why the source is doing this, I couldn’t say.
It also takes a special level of suspension of disbelief to think that Andy Stone somehow writes emails in Indian English and that XCheck gives privileges to take down content from other users.
There’s plenty Gawker did that I didn’t like or didn’t care for, but as a proponent of free speech, I see nothing positive about them losing that shit.
Edit: lol, no wonder parent & gp were circumspect on names - I'm getting downvoted :'-D Anyhow, here are my sources:
https://www.theatlantic.com/business/archive/2018/02/hogan-t...
https://www.theatlantic.com/technology/archive/2016/05/the-b...
At the time there was a story about the Saudi's being involved and some link to Trump.
Years later it came out that Bezos's investigator knew pretty early on that it was likely the brother of Bezos's girlfriend, however the National Enquirer was already in trouble for allegedly acting as a foreign agent for the Saudi's. By seeding the fake Saudi story they were able to apply huge pressure on the Enquirer.
Maybe something similar is happening here, Meta is under legal pressure somewhere on this topic. It doesn't have to be true, it just has to annoy the wrong politician/judge.
Different but related, I would say. For example abuse of a service and undesirable behaviour on a service could be thought of as different points on a spectrum.
Isn't Reddit the counterexample?
A dozen of us moderated a controversial sub with 2M users, no problem.
The equivalent situation assuming mods review every comment would be a subreddit with 1 billion users. If mods are only reviewing reports which is some small percentage of comments then adjust accordingly.
Moderation workflow starts with user-reports on posts/comments. At scale, you can set a threshold as well, which dramatically reduces the number of items to review.
Obviously, scaleup the number of reviewers. Obviously, add some obvious algorithms: warnings, temporary/permanent bans, etc. ...
No problem except for people who may have been censored due to bias and error on behalf of moderators.
When you report content for the first time your reputation score is 0.5, if the content is of a celebrity or public figure, your reputation will invisibly drop, enough of these and your score will be so low as to not be a useful flag when evaluating whether content is objectionable.
On the other side, if the content is reviewed as violating ToS, your rep will increase.
With a system like this the fake reports and brigading will quickly lose its power, eventually all users who abuse reports will be ranked so low as to not matter.
A threshold of rep score can be set before the content is auto removed. Maybe something like 25pts in order for it to be removed, and that's after adding together all the rep of the reporting users.
That being said, the above can also be abused by creating a bunch of high rep accounts or opening up a black market of high rep accounts for sale. It's definitely a higher bar though.
Then they use the high rep accounts to brigade and fake report. Or sell them.
They are patient, so even if you make reputation go up slowly, they will just build a pipeline of account harvesting.
This explains why so much "moderation" on sites like Twitter seems to lack any rhyme or reason.
What about human employees reviewing users reports, especially randomly picked samples from those that resulted in successful takedowns. If the content was found to be sound under the rules, penalise the reporters. Suspend their ability to make reports (probably in the form of silently ignoring them).
Ie some human moderation should still be done at this scale, but using a combo of random sampling and system-flagged-suspicious.
Of course that could be abused as well,but you'd have to make a group of people that first got the good rating then reported same people and that would be significantly harder
Clearly the actions were reviewed so people’s percentage change could be adjusted based on seeming fairness.
I’m also very pro a pay for final review, where eg you pay $50-100 or something around that level of inconvenience to have a multi-employee review (each employee reviewing it independently, unknownst to each other, and taking the majority decision).
1) A "revolution" and promises of new fairness
2) A Nomenklatura
3) The rise of unassailable systems and faith in them
4) Widespread failure and instability of those systems
5) Attempts to stabilise by handing more power to privateers
6) Abuse and corruption run wild amongst new oligarchs
When I started, my personal account was banned twice on the condition of adjusting 2FA. It was sorted out. I had a past account that I could no longer access as email domains changed, so it looked like I was trying to maintain duplicate accounts. Why it triggered on adjusting 2FA didn't make sense. Then, it wanted me to verify my identity in a way that created a Catch-22 of wanting a login from itself or to a device that didn't have a valid login session. I set the options for better security and left it, rolled all of my personal passwords, eliminated public data, canceled unnecessary social media accounts, and enrolled in security features like Google's Advanced Protection Program. I'm hesitant to make changes risking getting locked-out again.
I recently spoke with someone in Readiness who works with hiring human verifier contractors around the world. A primary issue is scale. You would need to hire the entire world's population to moderate the content being produced. Still there will be bias and misinterpretation of sarcasm, and varying standards of acceptability and decorum. AI is a force-multiplier to an extent, but it takes human judgement to rectify mistakes and data to identify brigades, scammers, terrorists, and political manipulators seeking to exploit an imperfect system. Sadly, the humans with the best judgement typically have better career options that they couldn't be paid to do social media moderation. It can be made better, but the ultimate realization is with even great care, good intentions, and attempts at making things sensible and fair, there are always going to be mistakes. It's trying to minimize mistakes and not enable genocides, election sentiment manipulation, or product scams. It's doable to minimize mistakes, but it take persistent vigilance, wisdom about human nature, and creative solutions to deter and prevent harm while avoiding harming innocent persons. Mistakes are bad and disappointing and it feels bad when they happen.
In case anyone were wondering, the security is the inverse of Twitter's. Everything is logged and access requires a business purpose for a limited time, narrow scope, and approval to get that access. Almost no one has access to production data. PII is taken very seriously. There are no laptops with copies of user data. All laptops are encrypted, just in case, and for general principles. Password complexity requirements are insane. I can see my work/personal FBID user object in the graph, but as soon as I try to prod any links to other users, big warnings appear. There's an army of insane genius security researchers and practitioners who create and deploy defense-in-depth tools for broad and specific solutions to prod, corp, and endpoints that reduce our risks to being compromised, data being exfil'd, and security "oops"es from happening.
Work users who transit through certain "hostile" countries lose some security credentials and access. I'm actually wondering why laptops aren't spot-checked for malware implants and hardware/firmware modifications. I would assume employees with critical access who travel internationally with their work laptops and phones are prime targets.
PS: I wonder if people would pay $X / month (say $199) to have a high-signal social media service that requires a level of "vouching" invitation, names with faces profiles (not visible to the wider internet), sensible/proportional mediation and civilized feedback, politically-neutral, and free speech-loving to increase the sense of community and reduce the potential of anonymous bad actors. 37signals/basecamp accumulated research that showed that smaller communities with faces and real profile names lead to nicer interactions. I don't recall the source, but communities that are defended in terms of politeness and boundaries tend to endure while undefended communities drive away users and tend to disperse.
You mean to tell me in your country there is nothing to prevent you from quitting your job and then next day writing up and publishing full detail reports about internal operations?
[edit] I have no knowledge or opinion on this story, never read the Wire, and never use Meta or any of its products. It does, however, seem rather clear that governments and corporations concerned with controlling message and image do view social media platforms as the most important battleground in the information wars these days.
I can't be the only one who sees a problem with this. No content should ever be taken down automatically just because a bunch of random people report it.
Serious question, why not?
Your question was Why should no content should ever be taken down automatically just because a bunch of random people report it.
It's because the random people reporting it can't be trusted to be acting honestly. Without a human in the loop, the automated system becomes a tool for cyberbullies, and harms the very users you intend to protect. The foregone conclusion, thus, is that a fully-automated system will do more harm for the user-base than good.
> The foregone conclusion, thus, is that a fully-automated system will do more harm for the user-base than good.
This is not an answer, it is just lazy circular reasoning. "Automated take downs should not be used because they do more harm than good." Yes we have already established that is your assertion, I am asking for how you were able to conclude that.
Your options are:
1. Leave up child porn, revenge porn, hate speech, gore, fake and misleading stuff, etc for a long time, possibly forever, since a machine can no longer automatically take down mass-reported content
OR
2. Hire so many human moderators that facebook goes bankrupt
OR
3. Use untrained and unpaid/underpaid human moderators, such as strangers (the reddit model); go back to square 1 where volunteer moderators take down stuff they don't like with minimal oversight
OR
4. Have so few users that a small number of human moderators can actually review every report, i.e. the hacker news model. Or actually, no, hacker news "dead"s posts based on just number of reports without a human seeing it, I take that back. I guess this is the approach forums, most smaller blog comments, and other quite small websites use.
Do you have another solution that isn't one of those? Do any of those solutions sound good for facebook? Better than what they have now?
If almost all of your reports are in the first category, then you're considered trusted, and if enough trusted users report a post, then it can be automatically removed before a human moderator sees it. If you haven't reported anything before, or too many of your reports are in the second category, then your report only helps to get the submission in front of a human moderator and doesn't directly contribute to it being removed. If more than a handful of your reports ever end up in the third category, you get banned for abusing the report system.
That still seems like a violation of "No content should ever be taken down automatically just because a bunch of random people report it", as you wrote above.