Hacker News Reader

Comment by orbital-decay | Hacker News Reader

Posts like that nearly always assume the text-to-image and "prompt engineering" being used, usually due to the lack of experience with those models. This is categorically not the way to do it outside of having fun. The way it's done for predictability and control looks much more like "draw the rest of the owl, in a manner similar to my other hand-drawn owl" combined with photobashing and manual fixing/compositing. It's a hybrid area similar to 3D CGI that requires both artistic and technical skills if you want to create something non-boring.

This has nothing to do with the model's poor understanding of natural language, and will not change until we have something that could reasonably pass for AGI, and likely not even then. Your text prompts simply don't have enough semantic capacity.

View on HN

You might be interested in the "Commercial illustrators will keep their jobs, but will mostly need to learn to use AI as a part of their workflow to maintain a higher pace of work" section of the article, which gets into this more.

You're right! I've stumbled upon the prompt engineering part and rolled my eyes, which was clearly too soon.

The more plausible evolution is that people drawing the base concept are not "commercial illustrators" nor have art training.

If a magazine editor with run of the mill drawing skill can feed the prompt a sketch with stick figures and object outlines, and get back a good enough rendition with an improved composition, the job of the illustrator will be a side job of that editor.

I'm partial to the argument that being able to fix the generated image in post is a valuable skill, but on that part we already have decades of progress and people are usually more comfortable with editing tools than drawing tools.

"This has nothing to do with the model's poor understanding of natural language, and will not change until we have something that could reasonably pass for AGI, and likely not even then. Your text prompts simply don't have enough semantic capacity."

I don't think it's going to take AGI to get to this point. It's 'just' going to take a top-tier model adding robust multi-modal input imho. A detailed prompt plus a bunch of examples of the style you're looking for seems like it would be enough.

That's not to say it isn't really hard, but it doesn't seem like it requires fundamental innovations to do this. The building blocks that are needed already exist.

The biggest problem I see with LLM-generated imagery is a near total inability to get details right, which makes perfect sense when one considers how they work.

LLMs pick out patterns in the data they're trained on and then regurgitates them. This works great for broad strokes, because those have relatively little variance between training pieces and have distinct visual signatures that act as anchors.

Details on the other hand differ dramatically between pieces and have no such consistent visual anchor. Take limbs for example, which are notoriously problematic for LLMs: there are so many different ways that arms, legs, and especially hands and fingers can look between their innumerable possible articulations, positions relative to the rest of the body, clothing, objects obscuring them, etc etc and the LLM, not actually understanding the subject matter, is predictably terrible at drawing the connections between all of these disparate states and struggles to draw them without human guidance.

You see this effect in other fine details, too. Jewelry, chain-link fences, fishing nets, chainmail, lace, etc are all near-guaranteed disasters for these things.

It's mostly a problem of resolution, model size, and dataset quality, which can be mitigated with compositing. Larger models don't have problems with hands, and if they do, it can be solved by higher-order guidance (e.g. controlnets) and doing multiple supersampled passes on regions to avoid to fit too much detail in one generation. Even SD 1.5 (a notoriously tiny model) issues with faces and hands can be solved with multiple passes, which is what everyone does.

There are two problems with this: a) natural language is inherently poor at giving artistic directions compared to higher-order ways like sketching and references, even if you got a human on the other end of the wire, and b) to create something conceptually appealing/novel, the model has to have much better conceptualizing ability than is currently possible with the best LLMs, and those already need some mighty hardware to run. Besides, tweaking the prompt will probably never be stable, partly due to the reasons outlined in the OP; although you could optimize for that, I guess.

That said, better understanding is always welcome. DeepFloyd IF tried to pair a full-fledged transformer with a diffusion part (albeit with only 11B parameters). It improved the understanding of complex prompts like "koi fish doing a handstand on a skateboard", but also pushed the hardware requirements way up, and haven't solved the fundamental issues above.

I think you're right about the current limitations, but imagine a trillion or ten trillion parameter model trained and RLHF'd for this specific use case. It may take a year or two, but I see no reason to think it isn't coming.

Yes, hardware requirements will be steep, but it will still be cheap compared to equivalent human illustrators. And compute costs will go down in the long run.

The post actually goes into a bit of detail on that process.

This was essentially my post

Its like you take your AI to school, or do a Matrix-style data upload into your AI so its up to speed on a new concept

Professionals will learn how to do that, the market will cater to people that want to do that

> Your text prompts simply don't have enough semantic capacity.

Mostly, current tools are abysmal at maxing the semantic capacity. Midjourney is great a generating things that look good, but terrible at piecing scenes together.

Recent example I tried: a robot playing magic the gathering seated across a human.

Even getting the human in the picture is a challenge, but then the model doesn't know enough about MTG (it correctly pattern matches to "board game" or "card game").

Some pictures generated are much better than other, and it would be great to take e.g. the table setup from one picture and the robot from another, but doing is not really possible atm ("blend" doesn't work for that).

I have no doubt this will improve, but I'm wondering if there's something underlying this that could be a more general limitation? Maybe simply not enough data (a google image search for magic the gathering is also pretty disapppointing).

Another example: a glowing blue <company logo> carved into a stone monolith. It sometimes got the logo to be carved (rarely), it was never glowing blue (usually the whole monolith, or a part of it).

Sorry I noticed your post too late, not sure you'll read it.

The reason it happens is that the models are far too small (parameter count-wise) and the prompt understanding part is simple, usually it's either CLIP or in the best case a small and dumb transformer. (But regardless of the current capabilities, text is just not a great tool to express artistic intent)

Generally, what you want can be done by giving the model higher-order hints like sketches and pose skeletons; see controlnets for Stable Diffusion for example. The overall idea here is to use a custom model created specifically to guide the diffusion model, based on the non-textual input. The problem is that MidJourney can't do this, you have to use SD.

Another thing is photobashing/compositing. Avoid fitting the entire composition into one generation, it will make the model lose track of your scene. Using multiple passes helps a lot. It's best to inpaint the objects or img2img them based on non-textual guidance to add objects and details in the specific spot.

Check my other comment for an example of a complex scene workflow. https://news.ycombinator.com/item?id=37140233

Reply on news.ycombinator.com