Initiative from Google, OpenAI, Discord, others could transform trust and safety

I would think safety would be maximised by keeping OpenAI and Google as far away from your data as possible

We're all less safe when censorship puts their foothold in and we've been losing battles since things like hate speech eroded incredibly important concepts for democracy such as free speech.

As other commenters say, every time those with power get to control information flow "to keep us safe" they'll invariably use it to protect their own interest in detriment of ours. It does not matter if it's governments, corporations, or one at the request of the other.

No matter how the "vocal online groups" may appear to be in favor of constant censorship, reality differs significantly.

The good thing is that open source and controlling your own computer will still be what many users want and fight for instead of corporate control.

The only software that "keeps real people safe" is Tails+TOR, certainly not anything that has "Google, OpenAI, Discord" in the title.

I really hope this does not lead to every single popular AI platform or product implementing censorship or various moral guidelines like we saw in the last ten years of social media. This type of sharing can sound good, but often leads political capture. I find safetyism based statements like this disturbing not exciting:

> ROOST is here to develop and deliver tangible products, to produce and deploy software, and to keep real people safe

Nice write up. I like the comparison to the cybersecurity field of the 1980s & 90s. Seems a good way to look at it.

What does trust and safety mean in this context ? Who is trusting who; and who/what is safe ?

I think this is a very good initiative actually, especially for small businesses.

Building these tools in-house is not easy, and frankly quite stressful.

And while there are commercial solutions, they have questionable privacy profile and are extremely overpriced.

Mh. Is a comparison with OpenSSL or Snort really valid?

OpenSSL implements crypto primitives, which, as far as we currently know, are mathematically safe. It's clear cut, black and white, it either is implemented correctly and (as far as we know) safe, or it isn't.

Snort works based on fixed rules. It's black and white, either traffic is logged and / or blocked, or it isn't. Basic rules log entire classes of traffic, the ability to engineer around them is very limited. Specialized Snort rules are a cat and mouse game, though.

At least two of the three things ROOST wants to do are machine learning models. They are not black and white, they are fuzzy. Minor changes can be enough to fool a classifier without changing the real semantic meaning for humans.

I have very, very hard doubts Google, OpenAI et al. will release Open Source classifiers which they will actually also use themselves.

Either they will try to get perfect recall, then it certainly will have (probably massive) overreach. Or there will always be false negatives. In the latter case, ML classifiers are IMO the one thing where security through obscurity not only actually works, but is the only thing which keeps them working at all. If you can freely train on an imperfect classifier, you can fool it, no?

I would love this, I like the Open Source approach a lot, but I cannot imagine how this should work in practice.

Also: How should third-party companies be able to reliably "set their own rules" on a probabilistic ML model, anyway, without training it themselves? The thing Roblox published[1] is a classifier for 6 categories. Sure, it's nice that you can e.g. say "I only want to filter profanity" this way - but what constitutes profanity is not controlled by you, but by the whims of a US company, even though this very much is not a globally homogeneous standard.

[1] https://github.com/Roblox/voice-safety-classifier

Yeah, "trust and safety" in the same sentence with "content moderation" in a post COVID/Twitterfiles/Russiagate/Biden Laptop world, good luck trying to make that happen.

I'll stick with nostr.