Hacker News Reader

Comment by p-e-w | Hacker News Reader

> And yes, StableDiffusion from the original repo will rick roll you if you try to generate something that triggers its NSFW filter.

It goes without saying that the authors of a piece of software have the right to make the software do whatever they want, but that shouldn't stop us from recognizing that AI engineers are starting to act like megalomaniac overseers who consider it part of their mission to steer humanity onto the "right" path.

Who exactly do these people think they are?

Imagine this behavior from a web browser. "The URL of the file you were trying to download triggered my NSFW classifier, so I'm going to replace the file with this funny image."

This isn't funny, it's creepy.

View on HN

Nah it’s just basic due diligence for releasing an open source app of this nature. Turning it off is a simple one line change, because everything is obviously named. This is not some high wall to scale.

As someone that manages nsfw open source projects, this move seems fine to me.

And actually kinda hilarious.

If what StableDiffusion did was ask the user something like "The prompt you entered may result in the generation of content some people find objectionable. Are you sure you want to proceed?", then I would buy your argument.

As currently implemented (and the implementation took more work than a confirmation prompt would have!), it's an obvious attempt to control, rather than protect, users. They try their best to dress it up like a joke, but it clearly isn't.

It's open-source. Patch it out yourself. It's not hidden.

That's not the point at all.

It's the point of open source.

You have no point whatsoever. You're fighting windmills.

I disagree that the rick roll is inappropriate, but i also hate the argument that just because its open source, the programmers are above criticism. I think its reasonable people can dislike or even morally object to the decisions open source maintainers make (even if i dont particularly object to this specific instance)

They're not above criticism, but what they've done is a very small layer that's easily removed. You can do it, and release a version without it, in minutes. You can do it in less time than it takes to write a long comment decrying it.

It's not fundamental to SD in any way, and their suggestion wouldn't have worked.

I don’t read it that way. I think their intent is open to interpretation.

Fork it and fix it, see if your branch gains traction.

As currently implemented, it’s a hilarious Easter egg. I wouldn’t read too much into it.

They aren’t trying to “control” users, they’re trying to minimize the amount of bad press these AIs get. The approach they took was easier than your suggestion, because their system detects NSFW content on a per output image basis… so it’s easier to output some image (rickroll) rather than put up a prompt or even skip it completely (since they also generate a grid of the resulting images).

I should add that it’s also incredibly easy to comment it out.

Yeah, it's basic survival. Technology is inherently free and driven by porn, but sometimes that can be shocking to those uninitiated, so to "protect the children", restrictions are often placed to avoid alerting the horde and reverse progresses. I think those "enlightened" should construct and share such understanding that these are the actual reasons, so to avoid us all ending up in an actual dystopia where "protections" are enforced stricter than intended.

The content it can produce is illegal in many countries. It's not something they just decided to do one day on their own.

That would absolutely be more work than just subbing in an image and would neither have worked in the original implementation nor the various frontends it's been used in.

This is quite a classic HN kind of comment. Immediately assumes specific problematic intent and proposes a solution that doesn't fit the API.

Now instead users have to spend hours upon hours editing docker images to remove it. Much easier.

That's neither true not would it have been helped by making the original code require input during generation.

They put a simple check in, that tries to avoid returning nsfw images so that you don't get that back despite a more 'innocent' prompt. It's trivial to remove and is only part of the demonstration scripts, it is not part of the model or anything fundamental to it's workings.

From your previous comment:

> AI engineers starting to act like megalomaniac overseers who consider it part of their mission to steer humanity onto the "right" path.

You outdid your comment. You have an idea of how this should work. And you're trying to supposedly "steer" in a path of humanity. It's just projection.

This kind of preachery righteousness control tactics should sit elsewhere (fork it). The devs want to convey something is nsfw. They will do it however they want.

The implementation as it stands is actually easier the way they do it. They replace the image by scaling Rick to the image size. That means that they can still automatically put your output images in a grid without any extra code. They could have zeroed the pixels, but that could have led to confusion about if things were working or not.

It's silly and you're able to patch it out if you so desire. But most importantly, they released the model to everyone so it's way more open than OpenAI's DALL-E 2 (for better or worse). I don't think the argument that they're trying to control you really makes a lot of sense given how widely accessible this model is.

Absolutely. The limit isn't in what you ask for, it's in what's returned. The whole point is to try and stop the user being shown nsfw images despite sfw prompts, and since you can generate multiple results per prompt adding an interactive check on each one doesn't make sense.

What would the average 11 year old answer?

You know you can download the code yourself, right? You need not be reliant on their web interface if you're having these control issues.

It's also clearly in place because they don't have an age disclaimer, which would likely open them up to some sort of liability (or additional liability). Again, it's open source, and freely available, so like. . . this feels like a you problem, sorry.

Yeah, but, when it comes to generating NSFW contents, it is illegal to distribute pornographic software in some countries, and doing so can lead to domain blockage and such (i.e. South Korea, believe or not). This is something you gotta understand.

Also, if the author wanted to block NSFW contents at all, I'm pretty sure one can actually make the filter inseparable from the main network. This isn't the case here AFAIK.

That makes no sense. You can draw whatever you like with Photoshop. You can search for whatever you want on Google. That doesn't make the software "NSFW", and no country will block Photoshop's domain just because someone used Photoshop to create NSFW content.

Don't let people who are clearly motivated by a moralistic desire to control others hide behind BS pseudo-legal excuses.

Explain this to insert regime when they block your GitHub repo or make using your software a felony. You are proposing to act on ideology, while such mitigations are rooted in pragmatism. You often can’t have both.

1. Google do have a stuff called "SafeSearch"[1] to filter explicit contents, and they do have a DMCA-like system for processing violation reports from governments and legal entities. That is, Google is already cooperating.

2. Photoshop's sole function is to let people draw, so drawing a porn w/ Photoshop is 100% users' responsibility. In case of image generators, the AI itself has the capabilities to create NSFW contents. If you're talking about the interactive nature of "prompt", that logic also makes any adult games SFW - they are safe until the user clicks something.

[1]: https://support.google.com/websearch/answer/510

You'd be right about Photoshop, but...

https://helpx.adobe.com/photoshop/cds.html

SK and Germany are notorious for that. They don't care how the pixel arrays came to existence, any shapes and forms and representations, even links to such information, that are interpretable as belonging to certain categories are plain illegal there. I believe SK don't even recognize copyright laws for such "illegal" information.

Yes, the megalomania of someone who makes their code and models freely available.

StableDiffusion isn't freely available, in the "Free Software" sense. They use the highly uncommon "CreativeML Open RAIL-M License" which is a wall of text composed of weasel words describing how the software is so incredibly advanced and dangerous that despite the authors' earnest wish to do so, they cannot in good conscience make it genuinely Free Software.

These people wrote a bunch of Python code that pipes images and text through a GPU, and they're acting as if they had created a secret weapon that somehow humanity must be protected from. If that's not megalomania, I don't know what is.

Oh please, all the creators of these image AIs (OpenAI, Google, Midjourney, SD, etc) are being very very cautious with this stuff. It’s not going to end humanity, but it could easily lead to some really gross content that would paint the responsible organization in a bad light.

True, they are all very cautious not to let bad actors generate bad content.

Where "bad actors" are defined as "people who disagree with us", and "bad content" is defined as "things we don't want to see".

Needless to say, the list of bad actors never includes the authors themselves, and the list of unacceptable applications never includes anything the authors had in mind.

They are not cautious at all to prevent bad actors. It is very very easy to bypass their filters. They are just doing the basic due diligence to make sure that the average casual user doesn’t make something that grosses themself out.

Can you elaborate? Would Adobe be painted in a bad light if someone committed a crime using Photoshop?

Honestly it just feels weird. I've read the license restrictions and I can't see why they are there out what they're preventing.

Photoshop doesn't have an automatic "Generate image of Obama selling weapons to Putin" function.

But anyone with enough time to learn (or money) can do it anyways without much effort.

(I'm trying to understand the exact difference here.) So it boils down to democratization? That anyone can do it regardless of skills acquired?

It boils down to who's doing it. An artist in photoshop the artist is responsible. An AI asked to do it likely the people that made the AI is responsible (until AI can think for itself).

Take it out of drawing. If you write a program to control elevators and it breaks the elevators aren't you, the person that wrote the program, responsible? Why would Adobe be responsible for something someone else draws?

Or let's take an easier more close example. If you just made a character generator. Here's one

https://www.numuki.com/game/mii-creator/

And there was a "randomize" button that one out of 100% made a very pornographic image. Who would get the blame? The person that pushed the button or the person that created the project?

Perhaps it's the use of the AI moniker but this thing is a computer program under the control of a human, who is the person responsible. It takes time and effort to use and is very much not like automated elevator control. In order of difficulty:

1) figuring out how to rejig the prompt to get what you'd like, adjusting seeds and tuning configuration options. The more control you want, the more complex and manual the pipeline of backing software will be. All it does is amplify everyone's innate artistic talent.

2) Coming up with a good prompt. This relies on the person's imagination, facility with words and familiarity of limitations of the image generating software and its training datasets.

3) Selection. This can be a frustrating experience that tries one's patience.

> made a very pornographic image. Who would get the blame?

You would, it's not like the software automatically distributes all its generations. The vast majority of images these software generate are not good enough to be shared and aren't. You made the conscious decision to share it.

Even if it were an AGI, you would be responsible. It's very much possible to commission a ninja from a human artist and get something very pornographic on a famous celebrity and you would be held responsible for choosing to share it, you had the choice not to.

If they think it's a secret weapon humanity must be prevented from using, giving it away for free with examples on how to use it with a note on the side saying "pls don't misuse thanks" seems like a very odd thing to do.

You may be right in the larger sense, but the word you're looking for is conceited, maybe.

Megalomania: obsession with the exercise of power, especially in the domination of others

Devil's advocate, this output could have unexpected results, making a filter possibly desirable. Browsers do filter by default with safe browsing URL lists for malware, and they gate access to sites with bad certificates. But agreed users should have control, leaving such filters as safe defaults only.

> Browsers do filter by default with safe browsing URL lists for malware

Yes, for malware. As in, software created by criminals to damage your computer and/or steal your personal information.

That's not even remotely comparable to what StableDiffusion does here. What they do is refuse to generate content you requested, based on opaque criteria that no doubt are ultimately influenced by quasi-religious sentiments retained from the Bronze Age.

The concept is quite similar to the NSFW filter found on Google and other image searches for many years. Like the “mature” search settings, this filter is active by default but simple to disable.

> Who exactly do these people think they are?

Software engineers who took the time to think about the ways that their work could be used.

How many times have we seen on HN people pleading with developers to think about the ethical dimensions of their work?

What's creepy is the very real world that we're approaching where anyone can create porn of anyone else given a few pictures without their consent. That can cause all sorts of lasting psychological harm and other societal negative consequences. Even if this probably can't be entirely avoided, that doesn't mean that the folks developing these models are obligated to make it easy or are incorrect to take steps to avoid it.

> What's creepy is the very real world that we're approaching where anyone can create porn of anyone else given a few pictures without their consent.

Why is that a problem, exactly?

> That can cause all sorts of lasting psychological harm and other societal negative consequences.

Like what? To me this sounds like a problem with how some people want to have absolute control over what others do and want everyone else to play along. That's what I find really creepy.

Someone wants to generate porn of me fucking a donkey? Let them. They already could do that with ms-paint and I don't get magically harmed if they do.

This reminds me of the old saying: "Sticks and stones may break my bones, but AI-generated/photoshopped/glued-together pictures of me fucking a donkey will never hurt me" :)

That proverb isn’t true and stems from a generation that didn’t understand mental health

The proverb describes a state of mind one should aspire to. "Words will never hurt me" not because saying mean things can never do harm (it obviously can), but because I make an effort to control my emotions and reactions. Just as physical health is improved by a nutritious diet and regular exercise, mental health is improved by proper habits of mind.

Refusing to acknowledge the affect of hurtful words on yourself, is pretty much the opposite of "proper habits of the mind".

Denial does not make healthy people.

It's not denial so much as it is not caring to the point where you get upset. I think that's the strength of mind that GP is referring to.

I agree. Just because we have more advanced diagnosis and detection tools these days doesn't mean everything has to be a problem. The internet has really shaken up social norms and signaling and the kids are the first ones to enter the new world. I believe once we have a generation of fully formed 30-50 year olds who grew up with this strange social media, then we'll have the proper understandings and social knowledge to handle it well. Kinda like how parents these days can attempt to relate to their kids in high school. We need the elder wisdom there.

But I do believe that right now we're kinda off track. We almost venerate the act of being hurt. Everyone likes attention and nothing gets such protection by certain classes as having been offended or wronged by some other class. Social signals are currently built to display virtue and so people will go out of their way to display their support of the wronged. I _do_ believe that this is the correct direction to move from where we were, but I think it's gone a little too far and needs time to rebound.

Being a victim is the fastest way to go from zero to hero (reach millions of people) these days and it's also seemingly the least likely way to backfire. People are much more hesitant to bring up the wrongs of someone who's currently being defended for fear of ending up being placed in the out group and ostracized from the signaling group.

This all contributes to a social custom of looking to be wronged, so that you may point out the evil of the person/generation/world. However, people aren't idiots and can _sometimes_ tell if you're just faking for clout. This leads to a feedback loop of needing to be really hurt, seeing other people be really hurt, believing your really hurt, and finally internalizing that pain and trauma. Things do hurt, bad words are called bad for a reason. Society might be better if we were all nice. But every bad word and every microaggression does not need to become such a large roadblock to personal freedom. People are chaining themselves to the road with this stuff. Dieing on hills that require them to have been personally wronged, using their own pain as a way to shut down criticism. Yes, people hurt, and things can be bad. But it's also entirely possible to see something hurtful and continue life without it hurting you. It's 100% doable to actually not be hurt, not just ignore it, but to construct a self esteem and understanding that allows you to not be shanked by every half difficult social interaction that occurs.

Mental health awareness is good. But social signals lead individuals to believing they must be hurt to be a part of the in group. Virtue through suffering is an incredibly effective signaling mechanism

Im not saying you should broadcast your woes to the world for fake internet points. Just that people should be ok with having a bad day sometimes and that people who think they are too "strong" to ever be sad, usually are the most screwed up mentally.

I would suggest that anyone who cares so little that there is literally nobody who can say anything that hurts them is basically dead inside.

Part of living is opening yourself up to people. That includes the risk of getting hurt. That's normal and part of the human experience.

I don't think anyone in this thread is denying this.

Revenge porn, rumors, and defamation are already huge problems in society. Being able to generate such things on demand with AI models will make it significantly worse.

https://www.theguardian.com/lifeandstyle/2018/may/07/many-re...

To play devil's advocate being able to generate this kind of content on demand with AI may actually make things significantly worse.

The reason why revenge porn, rumours, and defamation are all possible online is because there's a level of plausibility to these things -- especially revenge porn.

Now that plausibility goes out the window because of AI generated content. Someone posts your actual nudes? Give a nervous laugh and say that AI made them.

Someone says they heard a rumour about you and asks if it's true? Say 'don't believe everything you read on the internet, you know they have AI that writes anything you want now, right?"

Perhaps, but there comes a point where the technology is out there whether we like it or not. We should criminalize bad uses, not engage in some futile attempt to try to stuff the proverbial cat back in the bag

Don't let perfection be the enemy of good. Anyone with a basic machine shop can modify an AR15 for fully automatic fire, and a bump stock can be fabricated even easier. That doesn't mean we should just give up and make these things legal and easily accessible to everyone.

An automatic rifle has almost no positive use to it. This software is not like that and more like something that makes everyone better at a skill, uniformly raising the artistic talent floor. The large majority benefit but some would use their new earned skill for bad. It would be like denying literacy and the printing press because there will always be some people who use it to write very bad distressing things, manipulate more easily en masse and spread misinformation.

They're not refusing to distribute the software, they're restricting it for NSFW uses, of which the danger is more significant than any positive uses - much like automatic fire.

"restricting it for NSFW", according to whose standard? "which the danger", like what?

I'm glad this technology is finally out in the open where people like you no longer have a say in how it is used. The sooner people accept its existence (like they have with Photoshop), the better off and healthier we'll be as a society.

>Why is that a problem, exactly?

Do you really need someone to spell out to you why being able to create realistic-looking porn of anyone might lead to some issues...?

>Someone wants to generate porn of me fucking a donkey? Let them. They already could do that with ms-paint and I don't get magically harmed if they do.

What an absolutely wild take. Do you expect this line of thinking to be convincing when you're comparing AI-generated images to what someone can whip together in MS Paint? Come on. You, me, and everyone else reading here knows that's not even close to a valid comparison.

Do you really need someone to spell out to you why being able to create realistic-looking porn of anyone might lead to some issues...?

A less creative next generation as they don't have to close there eyes and imagine generated porn of anyone.

The second problem will be finding people that you would even want generated porn for.

> Do you really need someone to spell out to you why being able to create realistic-looking porn of anyone might lead to some issues...?

Yes. I honestly have no

> What an absolutely wild take. Do you expect this line of thinking to be convincing when you're comparing AI-generated images to what someone can whip together in MS Paint? Come on. You, me, and everyone else reading here knows that's not even close to a valid comparison.

Wild or not, my point stands. With minimal skills, you can photoshop anyone's face on any pornographic image out there. It's a spot on comparison.

It is almost like a human able to imagine. If you think human mind is creepy because it can create any thought and start worrying about some psychological harm then you have created a false prison within your own mind where scary thoughts will creep in.

Deepfakes have already existed for several years, and let you paste any face onto any video, in a much more convincing manner than Stable Diffusion, which is very limited in video2video domains.

Photoshop doesn’t let you work on an image of a dollar bill.

I would argue the authors went out of their way to actually make it very easy to “decensor” their model in a way that lets them wash their hands of things. So regardless of the outrage over their Puritanism, they actually have made it easier to make pornography than ever before. I don’t think applying the tiny speedbump of their model being censored by default is unreasonable in the interests of conservativism.

In the long run these men will go down in the history of pornography.

Read the tweets of the Stability AI, company behind Stable Diffusion, founder. He agrees with you. The image is purely to protect themselves against criticism, takedowns, possible legal action, stuff like that.

https://mobile.twitter.com/emostaque

It's literally one line of code so you don't generate unexpected boobs. Save your ire for the nannies that won't let you generate NSFW images at all.

I’m sure you’re great at parties. A number of people have explained how easy this is to turn off (I did it myself in minutes without outside help by literally commenting out a line of code and changing a variable name). Getting offended at everything doesn’t solve anything and just makes half the US think we’re all snowflakes.

Remember the Terminator movie? It’s about an AI that went too far.

Don’t worry, intentionally warped faces and NSFW filters have like a month, tops

There's no intentionally warped faces on any of these models, SD is just the first model technically advanced enough to be able to generate decent faces. People have very high standards for those.

(What is silly is that DALLE2 won't let you edit a picture with a face in it, so you can't outpaint one of its own generations. But actually you can, if you crop it carefully.)

Reply on news.ycombinator.com