This tweet provides important context: https://twitter.com/nickcammarata/status/1512119623315075081...
They weren’t just copying/pasting prompts there was human creativity involved as well
They weren’t just copying/pasting prompts there was human creativity involved as well
I think the primary reason people are wowed by this thread isn't attributable mainly to the subtle effect of the cherry-picking he did, but in fact to the overall quality of any image generated by DALL-E 2.
Gather round, gather round, give me a text, any text at all and I will produce you an image of some kind. And you will call it "good" if it looks like anything at all.
Because all art is subjective and your mind will work overtime to connect it back to the text you provided.
Practical and epistemically relevant knowledge to anyone deciding how interesting these results, presented originally without mentioning they were cherry-picked, are. I’m doing a favor to provide it, as doing so isn’t exactly something that makes me look great, but is very much worth knowing for anyone following him.
Side note - there’s this SF club of effete intellectualistas who fashion themselves as modern day florentines during a de novo renaissance. They do a lot of back-patting. They have the exactly mentality of your reply - be kind, love is all you need, etc.
It’s sort of the exact opposite of the east coast mentality that willingly sacrifices looking good and “getting along” in favor of finding the truth despite some discomfort. Discomfort to this group is very taboo.
Of course, this don’t-rock-the-boat mentality is very much intentional as it gives said club the ability to instantly shun anyone who deigns to critique it, allowing them to continue building their following.
Your second critique: assumes the original presentation had to be accompanied by methodology and proof to be of value; derives an implicit attempt-to-distort from your perspective of the scene at hand; devolves into paternalism to end in unsubstantiated moralism.
I might even agree with the spirit underlying your words—given, say, the meaning-loss of the company’s name—this just isn’t the way to convey it.
You're absolutely right, here he displays the full set for a given prompt. They all look fantastic!
Would you have names or links for some other projects you're aware of? Would love to check them out.
I don't think that is terribly surprising, nor a very cogent detraction from the model.
DALL-E 2 is different from what I've seen. The things it produces seem to actually make sense the majority of the time. The outputs are strikingly similar to what a competent human might output as opposed to one with a severe mental illness.
I'm sure part of this is an inherent advantage that DALL-E enjoys regarding context. Art is supposed to be artistic whereas text is expected to maintain long distance logical consistency of abstract concepts across a stream of output and also to communicate something concrete. So in a sense the bar for art is probably lower in many ways.
GPT-3 amazes me and occasionally disappoints me. But it's still something I never thought I'd see in my lifetime. I suppose I'm still putting GPT-2 and Dall-E 2 in the same ball park because they are both so far beyond what I thought would be possible from what are essentially brute force methods.
The rejections were most commonly
1. Kind of just slightly boring or literally drawing the thing rather than being cool and artistic
2. Cool but similar to the artistic style of bios near it in the thread, whereas I wanted to keep it diverse (surreal followed by literal, oil followed by sharp lines etc) so it's more fun to scroll through
Whereas a few years ago generative models (GANs etc) would often render like static noise sometimes or completely wrong things. I've only seen that problem once with DALL-E across hundreds or thousands of images now (it generated a fully white image)
The stuff on deviantart is pretty good too and neatly tagged and classified by art style.
Has anyone compiled a list of the styles and artists Dall-E "knows"? How niche does it get? Decorative Initial Caps? Florid Victorian Ornaments? Googie Architecture? SFF artists like Michael Whelan, Vincent Di Fate, Jeffrey Catherine Jones, or Jim Burns? Banksy? Sculptors like Bathsheba Grossman or Markus Pierson? Early animation artists like Ub Iwerks or E. C. Segar?
I was experimenting with one of the VQGAN+Clip notebooks a while ago, and it did pretty well with some styles, but not so much with "Heroic Realism" or "Soviet Propaganda Poster" or "Sheppard Fairey", and even worse when I was trying to get it to draw in that style an object that could be construed as implying a style itself like "retro robot" or "50s raygun" (eg. "A retro robot drawn in a heroic realism style" or "A cubist painting of a steampunk pistol"). Is that kind of dissonance a problem for Dall-E?
How many people would then be defending the AI, that actually it wasn't just the human, the AI was playing a critical role in the creative process, ne'er to be replaced? I venture zero people would say that.
Jiggle the imagination just a little bit, dangle some progress, and we're off to the races.
This is "I'm feeling lucky" on google image search + style transfer + trial and error.
If you think I am being dismissive try a few of these twitter bios as searches and see for yourselves.
I guess it fits with the times we live in. Reward shallow plagarism. Outsource your mind.
It isn't theft if you can automate it.
Autotune for the deaf, Dall-E for the blind.
https://twitter.com/nickcammarata/status/1512123067803344899...
Prompt: "expressive painting of a man shining rays of justice and transparency on a blue bird twitter logo"
You have to break the concepts up apart (which is one of the things Dall-E improved on).
As such: "expressive blue bird"
In google image search, type clipart, and I even get pill tags to further narrow it down to illustrations for animal paintings and so forth. Google's classifier knows the concept of a "blue bird" and expressionism too.
https://www.google.com/search?q=expressive+blue+bird&tbm=isc...
The same for "ray of light". In fact the top results there I get pngs of sun beams on a transparent background. Which is perfect.
Neither the birds nor the rays of light in the pictures it produced are truly its own creations but lifted from bits of pictures in its training set. I bet you could find the exact bird from the second row online in many places for example. It just won't be blue or stylized.
Composite those things together manually and add a style transfer you'll get similar results to DALL-E as that is what it is doing more or less.
Note that in that example the "twitter bird logo" is actually expressed in 6 out of all of those images. Look for the small bird, that looks like the Twitter logo. It's there. It's doing the thing.
Nothing is expressed. Find yourself a blue bird in an expressionistic style, go to google image search and give it the url. Click on tools -> visually similar.
Enjoy an endless supply of things to plagiarize. In the middle picture of the second row you can clearly see how several pre-existing images are sharply cut off before being re-blended.
Tech like CLIP, GPT-3, DALL-E, etc. are indeed nearing the sophistication (w. caveats around outliers and harmful outputs) of Google search.
It took a lot of people to create Google search. It took precisely one training run for DALL-E 2 to create this.
edit: Removed toxic comment.
The tool wielders however.. I think are overyhyping this to say the least. And focusing on the wrong bits. It isn't sentient and it is not making art. But teasing apart how it is deriving these images might shake out serious advancements.
If you try actually doing this it will be trivial to see that this assertion is incorrect.
1. The way in which the elements of the images are integrated together is deeper than the level of style. For instance, see the image in the top row, second column: it has integrated the blue bird wings onto the man, not only simply grafting them on, but giving the appearance of their being draped on like a cloak, partly behind and partly in front of him (+ it's consistent with the man's posture and the rays of light to evoke a certain coherent cultural idea/image). You might be able to integrate multiple images (of man, bird, rays etc.) together and style transfer to arrive at a poor approximation of this—but even then, the decision to place the elements together in such a way would require creativity on your part.
2. The one example set of of trial images (generated from the phrase "expressive painting of a man shining rays of justice and transparency on a blue bird twitter logo") is one of the easiest among the full group to pick its various elements apart; if you try this thought experiment with the others in the thread, you'll see this idea is by far insufficient.
> the decision to place the elements together in such a way would require creativity on your part
I strongly suspect that's because it found similar compositions in its training set. So what exactly is going on here is fascinating.
Did it learn compositing? Is that why the image output is now much more stable? Or is it mearly finding similar artwork and competently recreating/mimicking existing compositions from different building blocks? So now we can not only transfer styles but also transfer compositions. That could be the beginning of something useful. Instead of a text prompt I'd give it my crappy doodle and it will respond with an improved/different one that is comparable (also a great way to steal tho).
And of course I picked the one that is easiest to tease apart where it is most evident so people will see what I mean.
> if you try this thought experiment with the others in the thread, you'll see this idea is by far insufficient
That depends on your imagination and your artistic eye I guess. Even if somebody could do that they certainly couldn't make you believe them. That's the accomplishment.
Neither one of us can prove it one way or the other so long as the model is a black box. And certainly so long as we don't have direct access to openai but just to curated examples.
So I reckon with "happy sisyphus" it breaks it apart into discrete vectors as a first disambiguation step and in this case resulting in two distinct queries.
Happy returns all kinds of image results.
Sisyphus returns the same kind of image results over and over.
A man rolling a boulder up a hill. Thus it can learn the concept of "sisyphus" on the fly as it would return:
man 95% boulder 90% hill 80% etc
Over a range of images.
So it must be Man+Boulder+Hill. That's its scene cue. That's what CLIP doodles initially. That's the "find me similar images step".
Happy is the style cue.
That's how "happy sisyphus" expanded into "a person carries a large ball in a mellow image in the style of a pixar cartoon"
Why specifically the Pixar style? One of several variations it tried, selected by a human.
The thing we don't know is whether the Pixar styled image is composited from the existing images in its training set. In other words whether this can be reversed.
That character looks familiar tho. I think it is plagiarizing.
Here is another observation: the boulder is not round, it reminds me of one of the Platonic solids. I don't think that's a coincidence, heh.
Yeah, weak results. None of the men look anything like Elon Musk. /s
But I can't find anything close to the realism that DALL-E 2 achieved here.
In fact this photo is exactly what you get when you photoshop the face of an ugly chihuahua unto a Pikachu plushie head and add a yellow brushed hamster body. And a cape. Literally that is what you're looking at.
It understood your prompt and amalgamated the right source photos into this nightmare fuel. Jesus wept.
I am pretty familiar with photoshop, and while I'm not an expert, I would find making something like this really difficult. Anything is possible with photoshop, but some things are very hard.
i guess some people are overhyped, but it's cool that this can do that. Previously, it took a trained human.
What are you going to do with this cape wearing realistic Pikachu that is actually a picture of a hamster?
Typically the trained human has something specific in mind. And if the client isn't satisfied they will torture them with countless requests for adjustments. So right now this is of limited use.
To me what is far far far more interesting is that Dall-E possibly understands the concept of what a Pikachu is supposed to be. That is downright creepy, and fascinating. I suspect that this visual aspect to things after people get over the clipart generation might find more functional utility as a way to see through the "model eyes" so to speak. To visualize the model itself. That could unlock a lot of doors in how training is done.
Maybe in the future you could train it on textbooks and prompt it for a picture of a molecule. Now that would be something. Especially if you start feeding it data from experiments.
Confused as to why you think you cannot do this with DALL•E?
This thing will kill art dead.
Well, representational art. I'd like to say post-abstract expressionists are at risk, but they still have their admirers convinced they're wearing clothes, and there's no indication those idiots will ever change their minds.
I'm going to have to write up a piece on this sometime, my argument is a little too involved for an HN post. But the gist is that the heart of what a fully trained painter does is make personal choices. A quick and dirty example of the difference:
Suppose you train an AI on Picasso's pre-1901 pieces. It's not going to decide it's time for a blue period.
Two more papers down the line who knows what Dall-E 4 will be capable of. It is a step in the right direction that the image output is now "stable", which is what this is demonstrating.
But it can't read your mind despite the eerie feeling you get, that is an illusion. Kismet in api form.
The next steps is to open this black box up and actually make its internal pipeline tweak able so it can become a useful tool.
It may end up an amazing super useful tool or a clipart plagiarisor/generator on steroids.
You can't even use it yet and you're already so eager to believe.
I'm simply certain that whatever its capabilities they are short of mind reading. You'd be equally impressed if you asked me to perform a google image search.
That does not mean that Dall-E is unimpressive or the results are fake. What I'm saying is that the hype and mysticism around this is unwarranted.
Elsewhere in the thread somebody else wrote that we are on the cusp of it producing convincing fake footage from the Kennedy assassination from a single text prompt.
The image output now being stable and pleasing to the eye is enough of a result even if it requires trial and error.
You wouldn't lose your mind over a wallpaper generator even though no machine learning is necessary to produce infinite variations of interesting patterns. This thing is spewing out "art" and people are ascribing magical capabilities to it as if it taped a banana to a canvas.
Anything is possible. Maybe Dall-E is capable of even more incredible things. Who knows where this all ends up. Sure. But not quite that much follows from what has been presented so far.
I'd dig it.
Anyway, I too Want To Believe. It is worth thinking about the falsifiability. One way that comes to mind to determine whether it is truly demonstrating creativity is to prove it doesn't have anything remotely similar in its training set (it almost certainly does).
The paper omits important details and they didn't release code, nothing has been reproduced independently. So far all that happened is that a human sent you a cool looking doodle.
We just don't know enough at this stage. Probably the people who made the damn thing can't fully make this claim yet either - it sure is an intriguing result however.
Is yawning at everything astonishing not just exhausting? Everything is "just" made up of less impressive things. But is this really not worthy of a little wonderment?
It is very good to keep an open mind, just make sure your brain doesn't fall out.
They call me the Hip-Hopapotamus
My lyrics are bottomless