I am frustrated with Stable Diffusion
novalis.org
novalis.org
First you need to refine the prompt so it is ultra specific (which OP has not done), then you need to generate a hundred or so images and pick the best ones. From there you can use img2img to refine it more, and once that's done you might want to go to Photoshop to add some finishing touches.
To get good results it is an art form at the moment, but of course as the tools get better eventually it will just be a click of a button.
The popular AUTOMATIC1111 webui implements it: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki...
I am also not worried about AI taking over art, exactly for the reasons you say. Even if alignment was a whole lot better you're never going to be able to perfectly describe an image in your head. Language is too limited. So it will always be a tool. Of course, it can enable fraud. But not many people are putting digital arts on their walls.
The other pet peeve is that people think generative models are only image or text. They so so much more.
I'm not an artist. I have a GCSE grade C or D, from 22 years ago back when that qualification still used letters. But despite that, I have enough of an eye for art to be surprised and disappointed by how many flaws other people are not only willing to accept, but actually prefer. Back in 2000-ish, that was my peers using non-tiling animated gifs as the background of their geocities pages; later it was seeing 72 dpi pixelation and jpeg compression artefacts on food packaging in a supermarket; or a lack of kerning in a clearly not-fixed-width font in a video game.
I've also seen an artist get frustrated because two managers completely disagreed about what an ideal UI should look like, and kept saying they were too busy to talk to each other even though both needed to sign off the design and didn't like what the other wanted the UI to look like.
And another who got a string of unpaid interns (they didn't tell me the interns were unpaid, but given how that contract ended it couldn't have been otherwise) to design a series of changes to the UI and then wondered why it was taking me so long to finish it. But that manager also kept telling me they wanted a button "wider" until I asked them to draw it and then they said "wider but vertically".
I'm starting to think one of the things artists are needed for is to convince everyone else to stop arguing and just do the thing. This fits with the meme of the last two decades of fancy art being "Here is an unmade bed representing my depression" — the convincing justification is more important than the work itself.
haha xkcd has one for everything
There's something called the Gullibility Gap and it is created because we anthropomorphize things as humans. More specifically, we see machines doing things that ONLY humans can do and thus our propensity to anthropomorphize them is even greater. We think that they must be intelligent and thinking because the only other things we've seen that do the tasks that they are doing also are intelligent and thinking. But what we've forgot is that the non-machines are also generalists. The machines can only do their specific tasks.
There's also issues with how machines "think." We know that they do not think like humans, and so this does create issues and ones I doubt we'll solve anytime soon. And like you pointed out with emotion, that's not going to be something that machines can understand. Who knows if AGI will even be able to. But then again, often we have a hard time understanding one another's feelings. But sympathy and empathy were powerful creations for us bio machines. So we'll see, but I do think it is quite easy to over attribute what these machines can do.
Imagine going back to 1960 and showing a programmer modern JavaScript. Surely this new "JavaScript" thing will completely replace programmers! I mean, what use is a programmer when someone can just describe their program in a language that looks almost like English (relative to this stack of punchcards I've been working with!). The user doesn't have to spend time fiddling with memory management. Complex algorithms like search and sort are abstracted away into simple functions. Why, implementing all that was 99% of a programmer's job! Sure, you still have to format the program in a particular way, but any random John should be capable of that with a little tweaking!
And you know, the old programmer would kind of be correct. It takes no less than two weeks to learn React and throw together a decent webapp now (not an enterprise-scale webapp, but a good proof of concept). Despite this 99% reduction to the barrier of entry, there's still a lot of paid work for web developers.
Pretty much this - and that's why both "AI art panic" and "I won't need artists anymore as I can just prompt anything" (which OP seems to try) are based on the wrong premise.
In general, people tend to make 2 false assumptions:
1. That SD is a ready to use product (it's more of a "middleware" model that is designed to build products upon)
2. That text to image is all it can do, while the most power is in style transfer, finetuning, and the ability to be guided with higher order hints than just text alone.
What would be best right now to use it as a content creation tool, is to combine SD with several other models to have temporal stability and tagged-3D-to-2D, and to use some software toolkit with a pipeline like that:
- quickly layout a mock-up scene in 3D with rough assets, just like in game engine level editors, possibly tagging the geometry or objects with short descriptions, like "middle-aged man", "Volvo semi", "pine tree" etc. No need for detailed geometry, just stick figures and rough shapes.
- using one of the multiple available techniques, train your style on your reference images (which might either be curated output of the model itself, or a specific visual language you constructed)
- enter your prompt, which doesn't need to be complex and convoluted as you already provide the precise hints on what you want
- let the model render the result, based on the depth, tagged geometry, rough rendering providing shadows and reflections etc
It is definitely a skill but I don't think it's true that good results generally require a lot of time once you know what you are doing.
Anecdotally I've seen requests on discord and 4chan answered within a few minutes with high quality images.
a simple rickety dock at the base of an ocean cliff, approach from the sea, picture from a boat, an illustration by moebius, complexla style, an outdoor scene, simple, expansive, ocean, cliff, stairs, dock, realistic colors
Negative prompt: isometric, photograph, boat, multiple frames, abstract, complex, borders
Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 1136556607, Size: 1024x1024, Model hash: 5e107da3, Denoising strength: 0.51, Eta: 0.4, Mask blur: 4Then I ran a sanity check, and asked it 'Sea'. I got some interesting abstract patterns, a comic strip about an airplane, and a bird that looked rather diseased. No seas. Maybe one of the abstract patterns kind of looked like the sea does in anime?. Just to double-sanity-check that prompts don't need to be more refined than that, I searched 'forest' and got some lovely photorealistic pine trees in the mist.
SD2 literally doesn't know what the sea is. I just checked and SD2.1 gives you more sea-ish abstract art than SD2.0 did, but still no dice. 'photorealistic sea' gives me two motorcycles and an elephant.
edit: Seems v2.1 understands 'ocean' some of the time (not all!). v2.0 did not understand either one.
Maybe SEA = Southeast Asia.
Besides that, you can search the dataset with https://rom1504.github.io/clip-retrieval/ or create probes with textual-inversion instead of sticking with existing words.
(Oddly, I don't know of any working image-inversion tools. There's one called "CLIP Interrogator" but it doesn't work very well, since it relies on an older image-to-caption model called BLIP that isn't an inverse of SD.)
HIRE SOME ARTISTS.
[edit] if you feel like a long discussion of the cognitive failure of AI over human artists, bring it on; if you're too cheap to hire artists, I totally get it!!! Make your own art! Otherwise, take your brilliant idea and pay other people to execute the parts you're not capable of executing yourself! Don't expect someone else to hand you a robot/AI replacement for your lack of skill for free ;)
> Yes, I could pay an artist to make art for the game. But I’m not yet good enough at game marketing for this to make sense: The original, 1990s Myst had 2,500 images, which would cost me more than I expect to make this year. Also, it would take an absurd amount of time.
Slightly a side story, so no one complains: In 1994 when I was 14, "Heart of China" came out and VideoSpigot came out and all I wanted for Xmas was a video camera and an interface to record short QuickTime movies in locations around my house, so I could string them together into a dark futuristic CD-ROM game. (in SuperCard, if you're wondering). That would have solved all my problems creating art!
It wouldn't have. Make your game as a sketch with whatever sketching tools you have...but don't expect technology to rescue you from needing other humans to make it good. AND DON'T UNDER-RATE OR UNDERPAY THOSE PEOPLE.
Because trust me, if you're disappointed that AI isn't giving you good enough art, you're no visionary. You haven't even tried people.
I don't think the author mentioned hoping to make money off his game. Most indie games don't make money.
>Because trust me, if you're disappointed that AI isn't giving you good enough art, you're no visionary. You haven't even tried people.
That's all well and good if you have a spare few million lying around and expect to make even more, but that's not the experience of the vast majority of indie game developers. They need to make the art themselves or find someone to make it for free
Point being, never consider your idea or even your code more important to the game than artists or their input. You're also paying for their criticism about everything from backgrounds and colors to gameplay issues, which an AI can't give you.
No no, everyone hoping to make games should continue to spend years hunting for shortcuts. That one elusive art style which is easy to make and requires no art direction.
If you have two pictures and you like one part from one and an other part from an other just photoshop/gimp them together.
If a particular manipulation is easier manualy there is no reason you need to try to convince the AI to do it. Just do it yourself.
In this case you drag the two images to two layers and use the erase tool to remove the blemishes which are present in your “top” image but not present in the “bottom” one.
They have one picture where they like the left side of the dock, but say that the AI screwed up the right side. And they have a previous one where the right side is good but the left side is bad. And they are frustrated that the AI can’t or won’t make a single generation where both things are good.
And the two pictures have the same structure, same illumination and look barely distinguishable otherwise. That is the case where editing is trivial.
You might wonder, how often does this really happen in real life, and the answer is: all the time. You get a picture you like overall, and it looks good from 10 feet distance but on closer inspection you notice some blemishes with it. One way to fix this is to “inpaint” the blemishes. Basically you tell the model to keep all the pixels unchanged, but generate new ones in a small region masking out the blemishes. And of course you run many such inpainting generations and pick the best. And when you have multiple blemishes, which is often the case, then very often one gets filled in good in one generation while the other is better in an other. And you can keep rolling the dice hoping that some day the AI will get it right everywhere, or you can get them in photoshop and do the selection manually.
Humans make these mistakes with naive/low-effort composites. Poorly generated AI images could contain such mistakes also.
OTOH, appropriate training could not only allow AI to avoid the mistake in its own prompt-driven images but also allow it to correct human artist-drawn images. Like local white-balance adjustments or local lens/perspective distortion correction.
A better strategy here would be keeping the same seed and then try adding 'dock' phrases to the prompt. Then image-bashing what it gives you over the first one. Image-bashing here basically means putting it in a new layer in GIMP over the old one and erasing the parts you don't want. It's really great with seed alternates because the two images will be very similar.
Inpainting is good for stuff you don't want the viewer to look at. If there's an errant shrub in your desert picture[0], you can inpaint it out. It almost never works for faces, heads, characters or any other focal object. The new depth stuff might fix it, but inpainting generally just doesn't have a good sense of how to orient things in the scene.
[0] I generated a bunch of SD images to spice up my CSS library's homepage: https://casscss.github.io/cass/
This is the same with all kinds of generative models. Humans are a critical part of the system. If this is "artificial intelligence" it only appears intelligent because there is a human gatekeeper.
Similarly, ChatGPT is a great replacement for a search engine. I can ask it to guide me through writing a metric to Prometheus from a batch job and it’ll dig out the relevant pieces from the documentation and present them to me in an easy-to-understand format. But I’m not giving it write access to main yet, or letting it make business decisions.
As a client, if I hire an illustrator to commission a specific image I'm not just hiring them for their technical ability to render images. The best illustrators are going to have a broad and deep awareness of all styles, modes of presentation, and compositional techniques. They're going to use all of their experience and knowledge to interpret what my needs are and deliver on that. That's not a trivial thing to do.
Even the best contractors have to deal with clients who don't know how to ask for what they want. The "I'll know it when I see it" type of client. As the "client" in this situation, you need to know how to ask for what you want in a way that your illustrator can interpret.
Last week I was telling my artist friends that, however upsetting this technology is, the game has changed. You can't put the genie back in the bottle.
I realized last night however that that is incorrect. Whatever "genie" artists have spent their lifetimes nurturing can now trivially be captured and confined to a bottle.
The birth of photography did not mean the end of painting, but it did mean that certain roles for painters no longer had as much value. Portraiture mostly became the domain of the photographer and photo-realism became more trivial. Painting as an art form shifted gears into greater and greater abstraction with less emphasis on realistic representation.
I think a good example of that shift is the painter Mark Tansey. His paintings, while photo-realistic, don't derive their value from their resemblance to the real world. Instead their interest lies in their allegorical representation of people and places. Their value lies in the curation of ideas and the story, concept, or idea that they convey.
For now - even in the age of AI generated images - such artistic allegory is still out of reach of technology. The value still lies in the human curation of symbols and ideas.
https://www.thebroad.org/art/mark-tansey/achilles-and-tortoi...
I’m pretty sure that some of the people posting chatGPT on Twitter are using Twitter to crowdsource the output review process. In other words script the loop from generation to Twitter post and use upvotes to decide what output is most interesting. Been thinking about doing this myself but I don’t use Twitter much so no users to crowdsource from.
Part of what I find frustrating about Stable Diffusion (and to a lesser extent Dall·E) is that they're so ridiculously far behind ChatGPT in some ways.
For example, I ask ChatGPT (in French) for a short story of three firefighters in the style of The Three Musketeers. And it writes me a perfectly serviceable little story about a brave and heroic rescue, with good pacing and structure. And it's written using the French "passé simple" tense, which is normal for fiction but virtually never used in speech. And I get this in a second or two, on my first try.
Meanwhile, I ask Stable Diffusion for a picture of "an astronaut on a horse" (the demo prompt in my copy). And I get endless badly drawn astronauts on dismembered horses. Dall·E does a bit better, but I'd obviously have to generate dozens of images and accept something visually interesting that doesn't bear any resemblance to what I envisioned. Apparently I can do better with a lot of prompt engineering. But it's a lot more work than ChatGPT.
Try asking ChatGPT to write you a novel, chapter by chapter. The chapters may seem okay individually, but you'll find chapter 5 contains statements which completely contradict those in chapter 1. And this can still be true even if ChatGPT is producing chapter 5 based on chapters 1 to 4 as input. (I think ChatGPT could do with some more training and/or engineering on detecting and avoiding contradictions in its output.)
I think one advantage of generating images, is you can notice problems with the output very quickly. With text, you might not notice the contradiction between chapters 1 and 5 until you reread it days later. Similarly, if you get two generated images, and you like bits of both of them, it isn't too hard to edit them together (even roughly) in an image editor, then use further generative techniques to smooth that roughness out. If ChatGPT writes you two different fifth chapters for your novel, and you like bits of both, merging them together can be a more tedious process.
I can't find documentation for ChatGPT's "context window", or how much text it can "see" at once. Some people say it might be 4k, or maybe more. But I would be surprised if it can actually work with multiple full-length chapters at once. So it's essentially like giving chapter outlines to multiple authors and gluing them into a book.
But as long as you stay within the context window, ChatGPT usually has no problem remembering what it's talking about. This isn't GPT 2, which wandered all over the place incoherently. ChatGPT is capable of writing a 1-page short story that is consistent throughout, and that has a clear structure and progression.
e.g. ask it for instructions on how to commit a crime, it will tell you its ethical guidelines prohibit it from answering. Ask it for more detail on the contents of those ethical guidelines, it will tell you that as a large language model, it is incapable of possessing any such thing as ethical guidelines. Point out it just contradicted itself, it is likely to respond that as a large language model it is incapable of contradicting itself, or else give some non-sensical explanation as to why the two statements aren’t contradictory
Ask it if it agrees with what you just said, it might reply that yes it does. Ask it again, it may instead give you a lengthy lecture on how as a large language model it isn’t capable of agreeing with anything.
I suspect it just wasn’t trained enough on the importance of self-consistency; nor on the principle that it is better to honestly admit to contradicting yourself than to give a non-sensical defence of your own contradictions
I even wonder if some of its training may have actually been contradictory - if trainers individually reward it both for saying “my ethical guidelines don’t permit me to answer that”, but also “I’m not capable of having ethical views” when pressed for judgements on ethical matters.
You can use a large text model to design prompts for a smaller image model if you want though.
The largest image generation model I've seen images from is Google Parti, which is an order of magnitude larger than Stable Diffusion. Still small enough that you might be able to run it on a 24GB consumer GPU with some optimizations though, so hopefully Stable Diffusion will get there in some months.
Have a look at the four kangaroo pictures here to get an idea of the difference https://parti.research.google/
for example, I generated superman but part of his cape was missing. I brought the image into GIMP and manually painted his cape on (sloppy, with pure red, no attempt at making it nice).
Then I ran the image through "img2img" and the cape looked perfect.
I swear this is from like 2 days ago, and it turns up in the HN search from two days ago:
> I am frustrated with Stable Diffusion(https://novalis.org/blog/2022-12-05-i-am-frustrated-with-sta...)
> 43 points|luu|2 days ago|40 comments <---
..but it's been resurrected now?
> 42 points by luu 5 hours ago | hide | past | favorite | 38 comments <--- ??
I mean, whatever, but the timestamps seem like they've been edited or something, which is very strange.
So while the results often are astonishing - I am still faszinated that it is possible to create realistic images just from a short text input - this shows that there is no true intelligence behind this. It will be interesting when this gap can be breached.
Till then, I can hope that there will be good tooling around SD, which make image creation easier by offering a good UI to make iterations easier and allow the user to guide the creation better.
Myst was computer rendered. If that is the feel the author is going for why don’t they use the same technique? The number of individual images would certainly not be a limit then.
The software and hardware differences between 1992 and today are immense.
What hasn't changed though is the artist. Robyn Miller and Chuck Carter squeezed every ounce out of what was available to them.
This guy could do the same with Stable Diffusion, but will take work it doesn't seem like he wants to do. He wants Stable Diffusion to be his perfect human-quality artist, and it isn't that.
- Make a prompt with a similar structure to the ones that generated good images online.
- Generate a bunch of low quality images.
- Pick the low quality image that kinda sort of has the shape and colors I'm looking for.
- Iterate on that image with various other prompts and img2img, increasing how much the original image contributes as it gets better.
It seems like an odd place to build a dock, if you can avoid it. Is there a real-world place like that?
https://www.alamy.com/stock-photo-boats-at-dock-near-a-set-o...
I actually thought the first version in the blog post was pretty good. It looked like a floating dock to me (that goes up and down with the water level) rather than one secured by pilings.
I think that the "cove" and walkway is shown on the left of the third photo in the article.
That said, any jetty would have had to be cantilevered from the rockface. The guy in the article drawing piles next to a shear cliff of rock is hardly thinking the engineering though!
It's not the case however for things like one-off illustrations (books, music albums) where it's easy to see that indies have totally embraced it over commissions already. It doesn't require general consistency, nor to be very specific, nor to have a very unique design.
I'm reminded of Louis CK's bit about wifi on planes:
"And it's fast, and I'm watching YouTube clips. It's amaz--I'm on an airplane! And then it breaks down. And they apologize, the Internet's not working. And the guy next to me goes, 'This is bullshit.' I mean, how quickly does the world owe him something that he knew existed only 10 seconds ago?"
If you pay a premium price you expect premium service.
There's no competition for airplane wifi though, you either use it or you don't. It's not like you choose between two different vendors... So they're competing against no wifi at all which allows for pretty mediocre and unreliable service as that's "better than nothing".
These free ML models exist in a slightly different world.
Is this still true, though? I regularly pay $8 for wifi on a 6h flight coast-to-coast with United. And also paid $15 for a 10h transatlantic flight last week, which seems pretty fair to me.
Oh, and messaging and chat apps are free, so you don't really need to pay if you only want to keep in touch with folks on the ground. Sure, quality is sometimes spotty, but seems to have improved over the last couple of years.
It could be a 16-hour flight to Australia, and still my brain would be saying no. Then I would order a $10 coffee and tell myself I deserve to treat myself.
Only the point of the Louis CK joke is that mere flying in the air, and getting to 1000s of miles away in a couple of hours, is a beyond premium service, when thought of in historical perspective... closer to a miracle.
It's this "taking for granted" that makes us consider it a standard thing in mere 100 years after its invention, and when it was not just a thing for rich people, but a totally unimaginable dream for millenia...
I think it's the opposite. With any complex and challenging task, the gap between "mostly done in a mediocre way" and "finished and acceptable" tends to be huge. Lost of "mostly done in a mediocre way" Hollywood films never see the light of day 'cause there's no reason for them to. Lots of writers throw their "mostly done in a mediocre way" novels away and start again.
If this tool gets you to a place that seems "90% done" but you find you have do as much work for the remaining "10%" as if you would if you started from scratch, frustration seems reasonable.
The more you abstract control from end users and “do it for them”, the easier generations can be made, at the loss of control. Midjourney is an excellent example of this.
If you want fine-tuned control, SD has an ecosystem that is rapidly accelerating in capabilities.
I’m part of a team building open source toolkit on top of SD, and the power users of our tool have shown themselves to be exceptional in their ability to pull high quality works out of the aether.
At this point, I'm seriously considering renaming the product and just move with another image generator.
Are you running on your own hardware? I've found the latest commits to be less stable and it's hard to know what to run.
As others have noted, getting good results from SD is very much a numbers and parameters game. I've witnessed first hand how some people get frustrated easily and quit tinkering because their initial approaches don't work right away. Prompt and parameter engineering really is a skill that you have to grow, and it makes a world of difference. img2img is also much finickier than text2img.
But you can use stable diffusion and dall.e together. Or use MidJourney v4 which is simply the best right now.
It'd probably be easier to create an ordinary 3D game world. I'm not going to claim 3D art is easy, but you only need to make your Myst island one time, and then you can just move the camera around.
Prompt engineering isn't what it should be.
It will use DALL-E 2 to start with, but Stable Diffusion 2 is the next back-end I'll add.
No
Just find and remove the code that reads:
if (drawing_bottom_of_cliff)
fuckup(dock, RIGHT_SIDE)
"If P=NP, then the world would be a profoundly different place than we usually assume it to be. There would be no special value in “creative leaps,” no fundamental gap between solving a problem and recognizing the solution once it’s found. Everyone who could appreciate a symphony would be Mozart; everyone who could follow a step-by-step argument would be Gauss; everyone who could recognize a good investment strategy would be Warren Buffett."
Maybe the difference between recognizing art and making it yourself is not as big as we assumed it to be.
> Yes, I could pay an artist to make art for the game. But I’m not yet good enough at game marketing for this to make sense.
There's a third option: learn to make the art yourself! Yes, it's time consuming, but so is anything worth doing.
> The original, 1990s Myst had 2,500 images, which would cost me more than I expect to make this year.
Start smaller. Or use a simpler, more achievable art style.
This is terrible logic. If you don’t want to make art, don’t waste time learning to make art.
Oh, I agree. I wouldn't suggest someone who wasn't interested in art spend time learning to make art. That would be ludicrous.
But that's not what I'm seeing in this article. Most of it chronicles the author's attempt to create a single illustration in a particular style. What are they doing, if not attempting to make art?