In the end you can be profane and very positive. You can also have strike language while being incredibly vile and negative.
My wife's cousin recently had a baby, and posted a photo on Facebook. My wife commented (something like) "She's so cute I'll have to kidnap her". Facebook locked her account for 24 hours for making a "violent threat". I am sure the mother wasn't threatened by the remark in the slightest. It just added to my wife's anger at Facebook for repeatedly giving her "warnings" over trivial or innocent things.
Not online, in person, but one of the teachers at our son's school sometimes "threatens" to "steal him" from us – her remarks don't worry us, because we know she would never actually do that, it is just a colloquial way of expressing affection.
This is due to years of old media digging through Facebook posts to write stories like "Child kidnapped after threat on Facebook and Facebook DID NOTHING". This led to more and more calls for FB to "do something" to "keep people safe online". So they started to run all posts through the AI prescreener and well, these are the results.
But hey, they get to at least say that they take down X billions of posts before any other human sees it.
I don’t think the mother would be as accepting then, so it also matters who posts the comment to who
Beyond this, by hiding the comment/post, the mother is now unaware of the threat and Facebook didn't contact local law enforcement in fact worse than doing nothing at all.
Which is to say: Often it's not reasonable to act without first knowing if the recipient actually took offence or felt threatened, before even considering anything else, because you can't know. A model aimed at judging utterances needs to also take into account "how close is this group? could this be in-group language?" and it needs sufficient context to judge that. Even then it'll have huge potential for error.
Incidentally, a lot of this wouldn't be nearly as much of an issue if we could trust that these kinds of tools would be used appropriately. E.g. instead of banning things, even just asking a "are you sure this will be taken the right way?" or similar is a whole lot more benign. The dating app Bumble uses a filter that checks for possibly explicit pictures, for example, but instead of blocking them it shows you a short message asking if you're sure the recipient wants to receive what you're sending, and lets you choose. Depending on use there might well be several levels of severity in messaging justified, from just a mild "are you sure?" marker to forcing you to explicitly acknowledge that you're on notice that you're responsible for the content.
If a male relative or friend had made the same remark, would my wife’s cousin have felt threatened? My wife tells me she would not have been bothered by such a remark, in that context, from one of her male relatives or friends.
And, if someone was really bothered by it, they could report it. But, it doesn’t seem like that is what has happened here, it appears to just be the work of some automated system. I’m not saying that if the mother reports the remark Facebook shouldn’t take the report seriously - cultures differ after all, in some cultures those kind of remarks are normal in that context, in others they would seem bizarre and offensive - but it shouldn’t just assume the remark is threatening because it contains the word “kidnap”
I was more interested in hypothetical "can we conclude it is ok from friend status on Facebook without actually knowing involve people" question.
I've noticed a lot of humans can't detect sarcasm. And that isn't correlated with traditional smarts either. For AI it's a hopeless task.
> You can also have strike language while being incredibly vile and negative.
I'm reminds of a group of people I know. They love love using genteel language to throw vile insults at each other. Using profane language is a automatic foul.
Now consider a large group of people upset about censorship and motivated to adjust their language to circumvent it...
The specific words don't matter, as any reaction would just shift the insult to another term. Ban cheeses? Start saying "I love you" ("mom, he told me he loves me! Tell him to stop it!"...). Or call him awesome. If anything, I think the brother missed a trick by not going straight to compliments - few things can be more cutting.
Given the right context it'd be clear to the brother what the intent was no matter the term, but increasingly impossible for the parents to distinguish "legitimate" communications from the insults. For that matter, no words are necessary to achieve that once both sides know. Just a look is enough.
We don't know how to fix this social issue with human censors, so it's much worse than just the state of the tech - it's an issue that's basically unsolvable without addressing why the speaker is intending to insult (if they are) and why the receiver takes offence (if they are).
The best you can do is address a small proportion of the most blatant cases (and that might well be ok and enough if you can do it with few enough false positives, but getting the false positives to an acceptable level is in itself a momentous challenge).
There of course isn’t a perfect solution and likely never will be.
The problem is that unless you observe the behaviour yourself as it happens, with substantial context, the idea that you can reliably know whether the behaviour was unacceptable is fundamentally flawed.
This is exactly what the choice of innocuous terms is for; to create plausible deniability. E.g. the word "camembert" thrown at someone out of the blue might be obvious because it stands out. But that's not the story you're going to get from the person who has used it as an insult. You're going to get a story about how they just talked about what they like to eat, or something like that.
You might see through that, but what quickly happens - what indeed happened with the brothers in my example - was that the person making the insults quickly calibrates to know exactly what results in a reaction and what creates enough plausible deniability that the person deciding if this is sanctionable is unable to determine with any reliability whether what they're hearing genuinely is an insult or the "target" being oversensitive and misinterpreting things. I was bullied at school at times, and saw this first hand as a target myself - the bullies quickly learned which ways they could word themselves to make the insults clear to me while creating enough plausible deniability even when there was no conflicting claims about what exactly they'd said.
Point being being that humans even with context and in the position to interrogate the people involved can't solve this with any kind of reliability. Then the notion that an AI at current state of the art without access to context or the ability to interrogate those involved doesn't stand a chance.
Yes but we're not trying to be perfect, we have to live and deal with this grey area.
> Point being being that humans even with context and in the position to interrogate the people involved can't solve this with any kind of reliability.
I disagree here, I think we can quite reliably do this a great amount of the time. There are absolutely bad actors who will get away with things in the short term but because context builds up over time it never lasts. And once you are aware they're trying to be subversive its much easier to spot earlier. It's not an attack with legs.
> Then the notion that an AI at current state of the art without access to context or the ability to interrogate those involved doesn't stand a chance.
I absolutely agree here.
And I'm sorry the people in charge were rubbish at protecting you from bullying.