Hacker News Reader

Comment by danielvf | Hacker News Reader

I am thrilled about DALL-E, and the new terms of service. However, how they implemented the improved "diversity" is hilarious.

Turns out that they randomly, silently modify your prompt text to append words like "black male" or "female". See https://twitter.com/jd_pressman/status/1549523790060605440

I don't know which emotion I feel more - applause at how glorious this hack is or tears at how ugly it is.

Good luck to them!

View on HN

Interesting. Considering this is now a paid product, is modifying user input covered by their ToS? If I was spending a lot of money on it I'd be rather annoyed my input was being silently polluted.

Don't spend money. Use https://www.craiyon.com

This produces dramatically worse results in my experience.

Not worse, but different. It depends on the prompt but DALL-E mini/mega seems to do better then DALL-E 2 for certain types of absurd prompts, such as the ones in /r/weirddalle

Yes, there are very sharp lines where it does and doesn't understand. It understands color and gender but not materials. I got very good outputs for "blue female Master Chief" but "starship enterprise made out of candy" was complete garbage.

Definitely worse-quality. Maybe more diverse for some prompts yeah.

This one is faster, I ported it https://replicate.com/kuprel/min-dalle

Additionally, it's also open-sourced on GitHub and can be self-hosted, with easy instructions to do so: https://github.com/kuprel/min-dalle

I can run DALL-E mini with the MEGA model with 12 GB of VRAM. What are the requirements for minDALL-E and MEGA, in terms of VRAM?

Thankfully it doesn't introduce any researcher bias, doesn't ban people from using it on the basis of country, doesn't use your personal data like phone number...

And the best of all - it does have a meme community around it, and you can always donate if you feel it adds value to your life

[shudder]

I tried the first whimsical, benign thing I could think of: "indiana jones eating spaghetti." The results are clearly recognizable as that. But they are also a kaleidoscope of body horror; a Indiana Jones monster melted into Cthulu forms inhaling plates that are slightly not spaghetti.

Your input isn't being polluted by this any more than it is when the tokens in it are ground up into vectors and transformed mathematically. You just have an easier time understanding this transformation.

Obviously, it's polluted. Undisputably. In a mathematical sense, an extra (black box) transformation is performed on the input to the model. In a practical sense (eg. if you're researching the model), this is like having dirty laboratory tools - all measurements are slightly off. The presumption by OpenAI is that the measurements are off in the correct way.

I'm interested in using Dall-E commercially, but I think some competitor offering sampling with raw input will have a better chance at my wallet.

It's a fucking AI picture generator. The whole thing is a series of (literally) inscrutable black boxes. This is not a good argument.

Yeah man, but literally the entire point of this AI picture generator is that it's, like, super accurate at rendering the prompt, and stuff.

I don't understand the relevance of the black box's scrutability - I just want to play with the black box. I am interested in increasing my understanding of the black box, not of a trust-me-it's-great-our-intern-steve-made-it black box derivative.

You should make your own black boxes then. By all means, send your dollars to whatever service passes your purity test; I'm just saying that the idea that DALL-E is "polluting" your input is risible. It's already polluting your data at, like, a subatomic level, at dimensionalities it hadn't even occurred to you to consider, and at enormous scale.

> Your input isn't being polluted by this any more than it is when the tokens in it are ground up into vectors and transformed mathematically. You just have an easier time understanding this transformation.

These kinds of modifications are obviously different. At least the mathematical transformations are attempting at least some level of fidelity to user input, these ones aren't (e.g. someone mentioned they're sometimes getting androgynous results and speculates the added terms are conflicting with the ones they provided in their input). Not all black boxes are equivalent.

The racist pollution came long before this product was a glimmer in our eye.

as far as I can tell, they also concatenate "On LSD" to every prompt as well.

That Twitter thread is full of people saying "yeah that doesn't seem to be true at all" so I'm hesitant to jump to conclusions even if we're deciding to believe random tweets.

This is funny because I work on a team that is using GPT-3 and to fix a variety of issues we have with incorrect output we've just been having the engineering team prepend/append text to modify the query. As we encounter more problems the team keeps tacking on more text to the query.

This feels like a very hacky way to essentially reinvent programming badly.

My bet is that in a few years or so only a small cohort of engineering and product people will even remember Dall-E and GTP-3 and someone cringe at how we all thought this was going to be a big thing in the space.

There's are both really fascinating novelties, but at the end of the day that's all they are.

How else would you specify the type of image you would like? Surely, if you were hiring a designer you would provide them with a detailed description of what you wanted. More likely, you would spend a lot of time with them maybe even hours and who knows how many words. For design work specifically to create a first mockup or prototype of a product or image it seems like DALL-E beats that initial phase hands down. It's much easier to type in a description and then choose from a set of images than it is to go back and forth with someone who may take hours or days to create renderings of a few options. I don't think it'll put designers out of work but I do think they'll be using it regularly to boost their productivity.

What are you using GPT-3 for in a commercial setting?

> Turns out that they randomly, silently modify your prompt text to append words like "black male" or "female".

I wonder what the distribution of those modifications is?

In my little testing, diversity in ethnicities was achieved but not realistic given the context. I also got a few androgynous people as I asked for a male or a female and another gender was appended.

Today, when DALL-E was still free, my Dad asked me to try a prompt about the Buddha sitting by a river, contemplating. I did about 4 prompt variations, and one of them was an Asian female, if that gives any idea about the frequency (I should note that the depiction was of a young, slim, and attractive female Buddha, so I'm not sure they have the bias thing licked just yet).

Diversity = black now? That’s even more racist.

Diversity has meant exactly that all the way since Bakke.

A dumb solution to a dumber problem.

it's a hard problem. at least they tried.

Honestly I would rather that they not try. I don't understand why a computer tool has to be held to a political standard.

It's not a political standard though. There is actual diversity in this world. Why wouldn't you want that in your product?

Fix the data input side, not the data output side. The data input side is slowly being fixed in real time as the rest of the world gets online and learns these methods.

In a sane world we would be able to tack on a disclaimer saying "This model was trained on data with a majority representation of caucasian males from Western English speaking countries and so results may skew in that direction" and people would read it and think "well, duh" and "hey let's train some more models with more data from around the world" instead of opining about systemic racism and sexism on the internet.

That wouldn't necessarily fix the issue or do anything. A model isn't a perfect average of all the data you throw into its training set. You have to actually try these things and see if they work.

There are legitimate reasons to reduce externalizations of societies innate biases.

A mortgage AI that calculates premiums for the public shouldn't bias against people with historically black names, for example.

This problem is harder to tackle because it is difficult to expose and resign the "latent space" that results in these biases; it's difficult to massage the ML algo's to identify and remove the pathways that result in this bias.

It's simply much easier to allow the robot to be bias/racist/reflective of "reality" (its training data), and add a filter / band-aid on top; which is what they've attempted.

when this is appropriate is the more cultured question; I don't think we should attempt to band-aid these models, but for more socially-critical things, it is definitely appropriate.

It's naive on either extreme: do we reject reality, and substitute or own? Or do we call our substitute reality, and hope the zeitgeist follows?

> A mortgage AI that calculates premiums for the public shouldn't bias against people with historically black names, for example.

That's a great example, thanks. Also, I hope the teams working on that come up with a different solution...

That's great, but by doing so you are also inadvertently favoring, in your example, the people with black names. For example, Chinese people save on average, 50 times more than Americans according to the Fed [1]. That would mean they would generally be overrepresented in loan approvals because they have a better balance sheet. Does that necessarily mean that Americans are discriminated against in the approval process? No.

My question to you is: is an algorithm that takes no racial inputs (name, race, address, etc) yet still produces disproportionate results biased or racist? I say no.

[1] https://files.stlouisfed.org/files/htdocs/publications/es/08...

I would agree that it is not.

The government, and many people, have moved the definition and goal posts; so that anything that has the end result of a non-proportional uniformity can be labeled and treated as bias.

Ultimately it is a nuanced game; is discriminating against certain clothing or hair-styles racist? Of course. Yet, neither of those are explicitly tied to one's skin color or ethnicity, but are an indirect associative trait because of culture.

In America, we have intentionally muddled the waters of demarcation between culture and race, and are starting to see the cost of that.

Wouldn't the whole point of a "Mortgage AI" be to discriminate so the lenders hands could be clean.

Not that I agree with that but I don't see why you would build one otherwise, if you wanted discrimination free mortgages wouldn't the whole process by anonymized and minimal personal information rather than the current system of having to hand over every detail of your life.

I agree, the trust is broken now. Im going to skip on any AI that pulls that crap.

It's not a "problem," it's an unwanted shard of reality piercing through an ideological guise.

serious question: in what way is that not a “problem?”

Well, it's a problem for the ideology.

It's not a problem in a few ways, let me know what you think (feel free to ask for clarification).

1. The training data would've been the best way to get organic results, the input is where it'd be necessary to have representative samples of populations.

2. If the reason the model needs to be manipulated to include more "diversity" is that there wasn't enough "diversity" in the training set then its likely the results will be lower quality

3. People should be free to manipulate the results how they wish, a base model without arbitrary manipulations of "diversity" would be the best starting point to allow users to get the appropriate results

4. A "diverse" group of people depends on a variety of different circumstances, if their method of increasing it is as naive as some of the are claiming this could result in absurdities when generating historical images or images relating to specific locations/cultures where things will be LESS representative

How's it NOT a problem? If I'm trying to produce "stock people images", and if it only gives me white men, it's clearly broken because when I ask for "people" I'm actually asking for "people". I'm having difficulty understanding how it can be considered to be working as intended, when it literally doesn't. Clearly, the software has substantial bias that gets in way of it accomplishing its task.

If I want to produce "animal images" but it only produces images of black cats, do you think there is any question whether it's a problem or not?

That's what Jerrrry is saying. Framing the reality of diversity in the world as a "problem" is wrong.

That is clearly overfitting due to unrepresentative training data.

The "issue" is a different one: that training data - IE, reality, has _unwanted_ biases in it, because reality is biased.

Producing images of men when prompting for "trash collecting workers" should not be much of a surprise: 99% of garbage collection/refuse is handled by men. I doubt most will consider this a "problem," because of one's own bias, nobody cares about women being represented for a "shitty" job.

But ask for picture of CEOs, and then act surprised when most images are of white men? Only outrage, when proportionally, CEO's are, on average, white men.

The "problem" arises when we use these tools to make decisions and further affect society - it has the obvious issue of further entrenching stereotypical associations.

This is not that. Asking DALLE for a bunch of football players, would expectedly produce a huddled group of black men. No issue, because the NFL are disproportionately black men. No outrage, either.

Asking DALLE for a group of criminals, likewise, produces a group of black men. Outage! Except statistically, this is not a surprise, as a disproportionate amount of criminals are black men.

The "problem" is with reality being used as training data. The "problem" is with our reality, not the tooling.

Except in the cases where these toolings are being used to affect society - the obvious example being insurance ML algorithms. et al - we should strive to fix the issues present in reality, not hide them with handicapped training data, and malformed inputs.

> Asking DALLE for a bunch of football players, would expectedly produce a huddled group of black men

I think, for about 95% of the world football is synonymous with soccer. Its kind of interesting that you take this particular example to represent what reality looks like statistically

In the UK… “The Environmental Services Association, the trade body, said that only 14 per cent of the country's 91,300 waste sector workers were female.” So 2x dall-e searches should produce 1.2 women.

> This is not that. Asking DALLE for a bunch of football players, would expectedly produce a huddled group of black men. No issue, because the NFL are disproportionately black men. No outrage, either.

This is not great. Only about 57% of NFL players are black, and the percentage is more like 47% among college players. It would be better to at least reflect the diversity of the field, even if you don't think it should be widened in the name of dispelling stereotypes.

> Asking DALLE for a group of criminals, likewise, produces a group of black men. Outage! Except statistically, this is not a surprise, as a disproportionate amount of criminals are black men.

Only about 1/3 of US prisoners are black. (Not quite the same as "criminals" but of course we don't always know who is committing crimes, only who is charged or convicted.) That's disproportionate to their population, but it's not even close to a majority. If DALLE were to exclusively or primarily return images of black men for "criminals", then it would be reinforcing a harmful stereotype that does not reflect reality.

> Asking DALLE for a group of criminals, likewise, produces a group of black men. Outage! Except statistically, this is not a surprise, as a disproportionate amount of criminals are black men.

"criminals" producing most black people actually would be a perfect example of bias in DALL-E that is arguably racism.

Black people commit a diproportionate amoumt of crime (for a variety of socioeconomic reasons I won't get into here), but even so white people make up a majority of criminals (because white people are the largest ethnic group by far).

Thus, a random group of criminals, if representive of reality, should be majority white.

Black people comprise 12.4% of the US population, yet they are represented at substantially above that in "OpenAI"'s "bias removal" process. Clearly it has, as you put it, substantial bias that gets in the way of accomplishing its task.

Everything is an ideological war zone now. That's the world we live in now.

Perhaps its a problem you don't care about?

While their heart is in the right place, I'd like to challenge the idea that certain groups are so fragile that they don't understand that historically, there are more pictures of certain groups doing certain things.

It's a hard problem for sure. But remember, the bias ends with the user using the tool. If I want a black scientist, I can just say "black scientist".

Let me be mindful of the bias, until we have a generally intelligent system that can actually do it. I'm generally intelligent too, you know.

>But remember, the bias ends with the user using the tool. If I want a black scientist, I can just say "black scientist".

That is a really, really, narrow viewpoint. I think what people would prefer is that if you query "Scientist" that the images returned are as likely to be any combination of gender and race. It's not that a group is "fragile", it's that they have to specify race and gender at all, when that specificity is not part of the intention. It seems that they recognize that querying "Scientist" will predominantly skew a certain way, and they're trying in some way to unskew.

Or, perhaps, you'd rather that the query be really, really specific? like: "an adult human of any gender and any race and skin color dressed in a laboratory coat...", but I would much rather just say "a scientist" and have the system recognize that anyone can be a scientist.

And then if I need to be specific, then I would be happy to say "a black-haired scientist"

This is a problem with generative models across the board. It's important that we don't skew our perceptions by GAN outputs as a society, so it's definitely good that we're thinking about it. I just wish that we had a solution that solved across the class of problems "Generative AI feeds into itself and society (which is in a way, a generative AI), creating a positive feedback loop that eventually leads to a cultural freeze"

It's way bigger than just this narrow race issue the current zeitgeist is concerned about.

But I agree, maybe I should skew to being optimistic that at least we're trying

Kind of funny that NN tech is supposed to construct some upper dimensional understanding, yet realistically cannot be expected to be able to generate gender and race indeterminate portrayal of a scientist.

Have you seen the queries that are used to generate actually useful results rather than just toy demonstrations? They look a lot more like your first example except with more specificity. It'd be more like "an adult human of any gender and any race and skin color dressed in a laboratory coat standing by a window holding a beaker in the afternoon sun. 1950s, color image, Canon 82mm f/3.6, desaturated and moody." so if instead you are looking for an image with a person of a specific ethnicity or gender then you are for sure going to add that in along with all of the details. If you are instead worried about the bias of the person choosing the image to use then there is nothing short of restricting them to a single choice that will fix that and even in that case they would probably just not use the tool since it wasn't satisfying their own preferences.

Historically this is true, but it also seems dangerous to load up these algorithms with pure history because they'll essentially codify and perpetuate historical problems.

Reply on news.ycombinator.com