Hacker News Reader

Comment by BoredPositron | Hacker News Reader

Everyone is treating this like corps have anything to gain from an open uncensored model. Switch your view and give me a single argument for it? That random nerds on HN stop jerking each other about what „open“ means? You are just not their target group. Having this discussion every time no matter if the model released is censored or not is just insanity. Bring new arguments or don’t use the models you don’t like. There will be a new sota „tomorrow“, maybe even one open enough for you.

View on HN

The argument is that it simply improves the product. For instance, Github Copilot is apparently refusing to do anything with variable names like "trans" and anything related to sex or gender, regardless of the intended meaning. That is a serious flaw and makes the product less useful.

See this: https://github.com/orgs/community/discussions/72603

You don’t know if the censorship is in the model or the system prompt.

That is not relevant to the argument. Censoring limits possibilities. While that sometimes has its uses, the overly puritanical approach American companies generally take degrades the value of their products.

I am talking about an „open“ weight model you are talking about a service. If the service wants to censor that’s fine and on them and their leadership if an „open“ model gets released with censorship it’s not, because it’s just „open, but how my manager likes it“

The lack of NSFW knowledge/capability makes them less useful for content moderation. I've tried to use multimodal models for categorizing images from large, mixed data sets. 95% of the input is safe for work. 4% contains nudity but is not sexually explicit. 1% contains nudity and is also sexually explicit. I'd like to categorize content so that nudity is hidden from users by default and that sexually explicit content is always hidden.

Every model I've tried so far is bad at distinguishing sexually explicit content from mere nudity, and many models are bad at distinguishing nude from non-nude. I don't know about Gemma 3 but Google's large commercial Gemini models refuse (or formerly refused; haven't tried recently) to tell me anything useful about images containing human figures. I assume that this is due to aggressive "safety" measures. On a technical basis, I assume that a model that can distinguish 10 different breeds of dog should also be able to usefully describe images of people wearing swimsuits, nude people, and people engaged in sexual intercourse.

There are models especially tuned for it even open weight ones. llms even multimodal ones are not up to the task. You know what doesn't help the discussion at all? That everyone's response is as usual just about titties.

4 months ago I tried every dedicated NSFW-image-classifier model I could find on HuggingFace or GitHub. They have a high false positive rate on certain kinds of benign content, like close up photographs of hands with painted fingernails, and a high false negative rate on artistic nude photographs. I even tried combining multiple models with gradient boosting but the accuracy barely improved; maybe everyone is training with very similar data sets. At this point I should train my own model but I was hoping to find something capable off-the-shelf, since content moderation is such a common task.

You can just finetune an open model instead of starting from scratch... that's the point of them.

This is what HNers surprisingly seem to not understand.

The risk of the model generating illegal content and then the company getting bad PR from vultures in journalism simply outweighs any benefits of including this content in the training data.

This is also why you will never see the big companies release a capable open weight image or video gen model.

>The risk of the model generating illegal sexual content and then the company getting bad PR from vultures in journalism simply outweighs any benefits of including this content in the training data.

This is completely unsubstantiated. The original Sydney (Bing AI) was violently unhinged and this only drew more users; I haven't met a single person who prefers the new Bing AI to the old Sydney, and for that matter I haven't even heard of anyone using Bing AI for ages now they toned it down. Trust in journalists is at an all-time low ( https://news.gallup.com/poll/651977/americans-trust-media-re... ) and America recently elected an extremely unorthodox president in big part due to the sheer hatred of the media shared by a large proportion of the population. Even the most hardcore social conservatives aren't calling for companies to censor the training of open source models so they don't produce adult textual content even when prompted to do so; it's not a political issue.

Brings an argument from nearly a decade ago ignores everything on google in the last four years. Ofc the „first“ rogue AI drew in more users because of the novelty of it… what a shit argument.

>You are just not their target group. Having this discussion every time no matter if the model released is censored or not is just insanit

Who is their target group for small local models that benchmark inferiorly to their proprietary solution (Gemini 2.0) then, if not hobbyists and researchers?

>> The press and decision makers without technical knowledge are the target group, it doesn’t matter if it’s used in production or not. They need a locally deployable model to keep up with enterprises that are to risk averse to put their data into the cloud and also don’t care that their shitty homegrown ChatGPT replacement barely works. It’s a checkbox.

But who is the target group?

Last time only some groups of enthusiasts were willing to work through bugs to even run the buggy release of Gemma

Surely nobody runs this in production

The press and decision makers without technical knowledge are the target group, it doesn’t matter if it’s used in production or not. They need a locally deployable model to keep up with enterprises to risk averse to put their data into the cloud and also don’t care that their shitty homegrown ChatGPT replacement barely works. It’s a checkbox.

Reply on news.ycombinator.com