The pictures, taken one by one, are impressive.
Is it possible to tell DALL-E to use the same girl in each panel? That is the only detail preventing me from being convinced this isn't a real children's book.
Is it possible to tell DALL-E to use the same girl in each panel? That is the only detail preventing me from being convinced this isn't a real children's book.
So you can generate a picture say 'picture of a boy playing with a ball' then go into edit mode (think this is new) and erase ball, then change prompt to 'picture of a boy playing with toy car'.
It will keep the original elements of boy not erased and put in a toy car. The character is kept the same but doesn't work as well as you might think as the body, face pose are the same.
Still, you can see this getting better over time.
Only got access this morning, hard not be impressed.
If you wanted to work within the current feature set, I think inpainting or 'uncropping' is probably the way to go. For example, generate a character until you get one that you like; generate a bunch of variants of that one face in various positions/angles/emotions (either through the variation feature or by reprompting); now, cut the faces or figures out of each one, and use them for each illustration as inpainting or uncrop with a text prompt describing that scene.
So you'd generate your target girl, get a bunch of her faces (uncertain, sad, facing left, facing right), cut them out for an outline, and then inpaint/uncrop: "Shawna felt sad to be away from home and Mom all day"+[sad-face.jpg]+"a brown-haired young white girl in a gingham dress standing alone in a colorful cozy kindergarten classroom"; and so on for each scene.
Denoising diffusions have been successfully applied to produce coherent videos.
In fact, there are even some benefits to train on video because images share information between them, which make context extraction easier.
Counterintuitively, because of the shared information between frames, their computations can also be partially shared, it can be almost free to generate multiple frames at the same time : A pair of RGB image can be considered as a 6D image, a 5-frame sequence a 15D image, and the neural network can learn to compress the various image channel together. The subsequent neural network layers can be kept 256D (for example) the same size as they were for a single frame.
Of course this trick won't work if the pair of image are too different but you can always stack images side by side to have a bigger film-roll image and you use a standard attention mechanism to attend to various parts.
https://old.reddit.com/r/dalle2/comments/vfi8lj/homer_simpso...
For original characters, it's tricky, you have to use a simple design that can be describe in text, in this example it's the prominent red hoodie and dark pants that make it look like the same character across pictures:
https://www.youtube.com/watch?v=J_laffNOoQw
Another problem is that DALL·E has a hard time telling different characters apart, e.g. requesting Captain America and Iron Man in the same image will give you two characters that have attributes of both Captain America and Iron Man mixed together:
https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a39...
As a comparison, stock photos and video of people are often available as a scene showing the same folks in a bunch of scenarios and angles.
You kind of need this to build something of any complexity out of the art.