Hacker News Reader

Comment by jsheard | Hacker News Reader

If Rutkowski didn't already exist, would AI artists ever be able to arrive at his style from first principles? It seems that all novelty entering the AI art ecosystem comes from riding the coattails of non-AI artists, they're literally incapable of producing, for example, fan-art of a newly revealed character until non-AI artists produce enough examples to train a LoRA on.

Months on from the release of the new Zelda game I still can't find any on-model AI art of most of the distinctive characters. They just can't do it without a LoRA, and you can't make a LoRA without many existing examples, so they never could have put those characters to paper using AI if the design only existed in their head. Conventional artists have to do it first so that AI artists can follow.

View on HN

There are novel artists using Stable Diffusion or Firefly as a photoshop-like-tool. Not txt2img, but stuff like inpainting, controlnet, contextual modification, LORAs trained on their own portfolio and such.

They are obscure for the same reason most artists are obscure.

But most "AI artists" you are talking about are would-be consumers looking for something specific, not creators looking to pull something novel out of their head.

By the same token, you had no ability to conceive of any original character before it was shown to you; why would an AI art model be any different?

A fan artist must observe a character's official depiction before they can produce their own variants of it. Most humans do not produce art in a clean room, either - we all produce a synthesis of the existing art we've been exposed to. A LoRA or dreambooth model incorporating a novel concept is conceptually similar to a fan artist being shown a new character so they can produce new derivations of it.

You could produce "police sketches" of a previously unseen character based on shared description, but it would likely fail to capture the nuance of the individual just as police sketches would.

> By the same token, you had no ability to conceive of any original character before it was shown to you; why would an AI art model be any different?

I've yet to see a LoRA produce a meaningfully diverse and high quality range of interpretations based on just a single piece of character key art, or brief appearance in a trailer, the way the conventional fan-artists do. What actually happens is that those conventional artists produce a wide variety of interpretations, someone scrapes those hundreds or thousands of images, and then trains a model using that dataset instead. The model needs the variety and novelty that the human artists bring to the table, otherwise it would just over-fit to the official art.

When I was looking for examples of Zelda AI art I noticed there is a LoRA for Purah, and the first version was posted on CivetAI not when her TOTK design was revealed, nor when the game came out, but a month later when there was a wide enough corpus of existing fanart to scrape together a decent model with. It only works by piggybacking off the non-AI artists effort.

I get what you mean, but my point is that fundamentally, what's happening in the LoRA and in the human artist is conceptually similar - each learns through examples what it is about a character that makes that character distinctive, and then attempts to reproduce it within some set of constraints. Neither is capable of doing so without first being exposed to what it is that makes that character distinctive. Human brains are obviously far better at interpretation than current AIs are (and have the advantage of actual will and intent), but I don't know why one'd expect a model that's never seen a character to be able to reproduce the essential nature of that character.

I rather suspect that you can get more variation than you'd think from a homogenous dataset, though - I've trained a LoRA on myself that can produce photorealistic renderings of me, as well as anime, Disney, Pixar, and Dreamworks styles, all of which actually feel pretty "me"; I don't have any human-produced drawings of myself as a Disney character or renderings of me in a Pixar movie, but the model does a good job of projecting what I'd look like given what it knows about "Disney-style" drawings, plus what I've taught it about what I look like.

For fun, I took two shots of Sonia from the third TOTK trailer (who should be novel and not in the model), trained a LoRA for 400 steps on just them (based on the AbyssOrange anime model), then threw it into an entirely different model (Dreamshaper, which mixes anime and realistic models) with some prompting to get something other than the original.

https://imgur.com/a/iPNPojK

I'm not gonna say that it's perfect, but as an interpretation of the character based on exactly two nearly-identical screen grabs trained for a whole 400 steps and some amateur prompting? Not too shabby at all. It hit many of the distinctive points of the character - darker skin, blonde hair with ringlets, the forehead emblem, the tear tattoos under the eyes, even the laurels under the hair. It missed the bell earrings and the secret stone necklace, but given the limited source material, I think it's quite good. With a few more varied screenshots, I'll bet that it could do quite impressively.

Probably something similar? I don't see why not.

I don't look at that kind of artwork all that much, but Rutkowski's pictures seem to have a very D&D/Tolkien-ish quality to me, to put it in some words. I'm pretty sure other artists made works in similar styles.

I do recall purchasing a poster with a similar look around 20 years ago, when he'd have been 14-ish. He might be a prominent artist in the genre now, but he's far from the first or that unique.

Greg Rutkowski is most well known for his work on D&D and Magic the Gathering. You recognize his style as he was one of its pioneers.

I'm sure he's influential, but D&D and MTG were around before he was drawing, and he didn't develop his style in a vacuum. There's a lot of similar work out there.

Rutkowski was /never/ in Stable Diffusion - the article is incorrect.

His name worked in the text encoder released by OpenAI, but it was more like a synonym for "concept artist". The image encoder didn't see any of his images.

So if he's not in a newer text encoder, you could find him again with textual inversion.

Putting greg rutkowski in a prompt produces compositions that have a fantasy/sci fi edge, more muted earth tones, areas of both clean and rough brush strokes and areas of large contrast with a tendency towards a central focus. It also makes for better dragons/castles/etc. Nothing magic.

Artists using AI maybe could have. “AI artists” are largely a bunch of obnoxious trolls with a shallow understanding of art history, technique, etc, such that this seems unlikely. Almost all AI art I see is within a narrow range of stylized realism or anime/manga-derived work.

Reply on news.ycombinator.com