Illustrating Gutenberg library using Stable Diffusion | Hacker News Reader

Illustrating Gutenberg library using Stable Diffusion

storybooks.ai

149 pointsAedraie54 comments

We are illlustrating existing books using stable diffusion and other ML models.

We are currently on our quest to illustrate the Project Gutenberg library.

This Show HN is really early in our journey and we are happy to receive your feedback!

Is this really using stable diffusion? Has a really 'midjourney' type style to it.

Good eyes! We currently pipe text input to Stable Diffusion, Dall-E and Midjourney at the same time and continue with the most pleasing result. The (very small compared to what we want to achieve) content we currently have required hundreds of renderings and Midjourney came out as most pleasing nearly all of the times. From what's on the website only the Sherlock Holmes renderings have been produced using Stable Diffusion. Nothing published so far from Dall-E, which most of the times is showing us uncanny valley. In terms of preference: We are experimenting hard to achieve better results using Stable Diffusion.

With this effort it sounds it would be faster to actually draw the frames yourself. Get exactly what you want and so on…

One interesting feature of these services is that one can be surprised: suddenly you get an image that is different from what you envisioned when writing the prompt but that is still visually pleasing or interesting or inspiring

Of course, then you have to learn how to draw, which can take years and years for middling results (source: me)

Or in between the two: draw just some blobs to set the general concept and do img2img with diffusion models.

https://simonwillison.net/2022/Aug/29/stable-diffusion/

If the majority of the models were produced with Midjourney, why did you title it as SD as u did?

Probably easier to get on the front page since Stable Diffusion is trending.

The sad thing is that they could’ve also put Midjourney (and DALL-E) in the title and not lost any votes.

Why sad?

>>what we want to achieve

What is it, SPECIFICALLY, you want to achieve.

And, assuming you do achieve THING, what are the negative consequences of such achievement?

How will you be accountable for dark patterns?

What recourse would any biologically physical Human Being have against [THE BAD THING] that may ensue from said achievements...

SERIOUS QUESTION

You literally can't fucking answer this question?

I find stable diffusion requires a lot more prompt engineering than either midjourney or dalle, but has a higher ceiling of quality if you do put in that effort. I wouldn't expect just using exactly the same prompt for all three models would work very well, need at least a little tuning per-model.

Midjourney now used SD underneath

I feel like the next killer feature of these models will be when they can retain contextual visuals between different scenes being generated

As in, the same mouse in four different pictures in different situations, not four different mice styled in the same way over four pictures

Textual inversion for stable diffusion, which is related, was discussed a few days ago: https://news.ycombinator.com/item?id=32643564

This is nice but still has some huge limitations, e.g. you need 3-5 images of something to train it. It becomes a chicken and egg problem if I start by generating something, since I cannot create 5 different images of it to train the textual inversion. It is also intensive and slow to train on each object.

Also I think we're running into limitations of using a single prompt to describe things, as it's often hard to separate the 'what' is in the scene and the 'how' to draw the scene, with many prompts looking like a giant word salad trying to tap in the internal secret language of the model

Yeah, this seems to be the key challenge right now. Some how an image and its elements might be used as context to condition the subsequent generations.

Oh, looks like Google solved this last week with dream booth: https://arxiv.org/pdf/2208.12242.pdf

The same tool can also be used to evolve an image by generating variations and selecting a few to reproduce. You could breed and cross the results at will. You could use multiple preference models for the fitness test and evolve them afk.

An "Evolve" button could become a standard feature on content editors across media.

If I was a commercial artist doing illustration work for articles or the like, I'd be getting worried about now.

I’ve been using midjourney to add a bunch of images to a conference talk I’m doing. It’s great for “ambience” type pictures, but hopeless if you actually want to represent anything concrete.

You can use Stable Diffusion to make more specific compositions, but you usually need multiple steps and a little bit of manual editing.

It's definitely possible to get concrete representation out of DALL•E, although it takes some experimentation to achieve realism within the constraints of the tech today. Still, we're not far off at all from indistinguishable synth imagery.

Some generated work in a photographic style: — https://www.hearthe.art/ — https://objkt.com/profile/heartheart/created

i think people underestimate just how many commercial images effectively fall within the 'ambience' category. i'm really convinced that in the mid to longterm, the only viable future for human artists will be a matter of GANs being subjected to the same sort of social stigma that is attached to other forms of highly economically lucrative, but morally dubious practices like child labour. ultimately sustaining a living in art has always been a matter of convincing the layman that what we produce has value and this will be the same at scale. if society does not value human-made art, there wont be any.

I am a professional artist and I am worried.

It feels like self-driving cars and the 80/20 rule; it'll be impressively good but not ready for prime time for a decade.

Self-driving cars have already been not ready for prime time for 15 years and they have at least 5 more years in this state.

But self-driving cars have a serious good enough threshold that isn't remotely as life-or-death for illustration

Playin devil's avocado here. SD sucks at depth. It makes 3D images which should have well set perspective, look flat and uninspired. Tastes will change and people will want to move away from the "tiny rooms, flat images and only-faces" trend that's dominating art at the moment. SD will ride the wave.

Until tastes change and after the current youth generation driving the demand for this washes out, it'll probably suck to be an artist without a strong brand.

There is something of the meta-humanity going on here. These illustrations are drawn from hundreds of years of human effort, and are being used to enhance our enjoyment of these stories. But it is only possible because of the prior human work.

It's a little like copying the work of hundreds of other artists at once.

Index funds bother me in the same way as in at some point the price discovery mechanism ought to break as everyone just uses one corpus.

All in all, this is an amazing idea, enabled by the work of amazing people, and it is like looking at something significant - I just don't know what :-)

can you explain the index funds line

I think the notion is the index fund is simply buying a stock because it has a certain market cap. It is not evaluating the fundamentals of the company.

The algorithmic art is similar in the sense that it is an averaging of previous effort. People made art and it was fed into the algorithm. The algorithm is some weighting of the art it was fed. Analogously, people who thought about companies bought and sold the stock. It landed at some price. The index fund just followed that.

There has been criticism of index funds [1] and defense [2] as far as price discovery.

Other people have raised the possibility that an index fund holding large amounts of competitor companies is an antitrust issue. [3]

[1] https://www.ft.com/content/87a3b117-7085-40f4-8cca-66bb8353c... [2] https://www.blackrock.com/corporate/literature/whitepaper/po... [3] https://corpgov.law.harvard.edu/2018/12/04/why-common-owners...

Human learning itself is a little like copying the work of hundreds of other artists at once.

Index funds hold underlying assets under management... You are paying for a slice of the underlying assets... All slices of the pie are accounted for and it doesn't create anything new. Not sure how that relates to machines reproducing human creative efforts.

The gripe with index funds is that they wouldn't work if everyone used them.

With a strong rule of law, they probably would.

What does a strong rule of law have to do with it? The issue is that index funds depend on there being a price discovery mechanism on the underlying equities. If no one were buying and selling individual equities there would be no price discovery.

The DOW 30 has 30 successful companies in it. If one of these companies falters it will be removed from the index and replaced by an up-and-coming company. It would be there even if people invested or not.

How does a company faltering or succeeding end up being reflected in its membership of the Dow? If relative shifts in stock price are the main causal mechanism, then that's exactly the issue: universal passive index investing (roughly) means that there won't be relative shifts in price.

Good thing everyone doesn't do it then! Why would professional traders stop trading?

It's a common fallacy that things need to work for everyone or they're no good. We can do different things. Specialization is useful.

They don't mean that it's creating anything but just that index funds depend on "real" market participants to buy and sell on good and bad news, forecasts, etc. If you imagine them expanding to where there are no (or almost no) real buyers and sellers, then they don't really work, and also the market doesn't really work. Though I think you'd have to get to a very (maybe impossibly) high share of passive investors for this to really occur.

That's their analogy to ML/AI art: imagine it takes over and puts most artists out of business... are the models just stuck, taking their own output as input and not really improving? Similarly here it's maybe not possible for this to happen to such an extreme degree that the imagined problem would come to pass.

There is a deeper question here ;;

At what point will we doubt reality and truth in favor of the AI provisioned one?

We talk about singularity being an AI that surpasses human int... but its goig to happen much sooner and via a diff vector, such as AI images.

the fucking speech by biden the other day looked like a deep fak to me - his mouth flapping was way exaggerated, and the fact that he could hold a salient sentence were the tells...

singularity isnt about AI being able to fool us that hte __*AI*__ is a human - its that we FOOL OURSELVES...

So I think that the whole AI art thing is going to make reality so elastic in our minds that it will be likea dream, and we litereally cannot trust a single fucking thing online. period.

Along the same lines, I was thinking this morning that we might soon have revenge-as-a-service where you can generate fake, "incriminating" news stories at the click of a button. Did Joe leave a lot of hate behind when he quit his job? An unscrupulous boss can generate fake news stories on random news sites (just plug in the locality name, e.g. "cnnbostonarchives.org"). Future potential employers will be shocked to learn that Joe raped and massacred 300 puppies, with lots of realistic pictures of Joe in line with the story. Most people won't have stories like this generated about them, only the ones who have enough haters.

AI as a service is coming... if not hear - there were talks about AI voice manip such that they changed accents.

AI is going to kill the HUMANITY. (this is a phrase that one should all know... HUMANITY)

AI is killing it. and gues who is trying to own AI - fucking pedo-financial-hedge-funds

Yea, same thing happens after reading the subreddit where all posts and comments are generated by GPT-2: https://www.reddit.com/r/SubSimulatorGPT2/

If you read it too much, and go back to normal human forums you will wonder if they're also generated by AI or if we are all just AI generated bots on the internet :-O

I wonder how far you could go generating tons of images from human art "feedstock", having people select the best 0.1% of them, then using them as input for another round, and so on. Then replace the people with another AI whose job it is to grade aesthetics of an image somehow, for a fully closed loop.

Edit: In other words...

1. Train your stable diffusion model using a corpus of normal human images.

2. Use the model to generate thousands of images.

3. Use human analysts to train another AI how to find "good" images in that set.

4. Generate ten million stable diffusion images.

5. Use the other AI to find the "good" ones.

6. Use those to train another stable diffusion model.

7. Goto 4.

(You lose some information each iteration, though, so it would probably eventually converge to grey goo or millions of pictures of Agent Smith.)

The creators of stable diffusion and others are already doing that.

https://paperswithcode.com/dataset/simulacra-aesthetic-capti...

https://github.com/LAION-AI/laion-datasets/blob/main/laion-a...

I like it. Your methodology seems analogous to a GAN [1]

https://en.wikipedia.org/wiki/Generative_adversarial_network

>Index funds bother me in the same way as in at some point the price discovery mechanism ought to break as everyone just uses one corpus.

I'm not sure what you mean here. Are you saying that index funds are driving individual equities? If so, why do you think that? Because of broad market correlation?

The idea here is that if everyone is just holding index funds that follow the market, no one is responding to external signals to drive prices to any particular value. Index funds follow the market, but what does the market follow if all the equities are held by index funds?

It’s a great metaphor for what could happen if there’s no new art coming from outside the model; everything get stuck (though I’m not sure I agree, tbh, humans are just pattern-matching robots too and it seems to have worked ok)

Almost all modern thing is "from hundreds of years of human effort" and "only possible because of the prior human work", even humans themselves.

Excellent steps, kind of tragic to lose humanity bit by bit.

Edit: I recognize the images are curated. And that we're moving in a direction.

The reality of one premise of the book The Diamond Age is closing in ... https://en.m.wikipedia.org/wiki/The_Diamond_Age#Plot_summary the Young Lady's Primer was an AI that manifest in this book, to help the child that reads the book to become .. mentally strong, I don't fully recall. The book would converse with the child and tell appropriate stories.

So I suppose we're really close (10 years) to this, needing to try and see if AI can predict the "next step" in a child's psychological growth, and somehow observe their status.

We kind of do part of this profiling with Machine Learning and our advertising prediction capacities.

Midjourney isn't very good for this (eg with https://storybooks.ai/sb/the-lion-and-the-mouse/) because it only has one style it forces everything into. Anything else you make it just going to look like that.

Though it's not as bad if you turn down stylization to "light" or do your own postprocessing.

Reply on news.ycombinator.com