Stable Diffusion animation
replicate.com
replicate.com
The thing that really strikes me is that open source ML is starting to behave like open source software. I was able to take a pretrained text-to-image model and combine it with a pretrained video frame interpolation model and the two actually fit together! I didn't have to re-train or fine tune or map between incompatible embedding spaces, because these models can generalize to basically any image. I could treat these models as modular building blocks.
It just makes your creative mind spin. What if you generate some speech with https://replicate.com/afiaka87/tortoise-tts, generate an image of an alien with Stable Diffusion, and then feed those two into https://replicate.com/wyhsirius/lia. Talking alien! Machine learning is starting to become really fun, even if you don't know anything about partial derivatives.
This is simply very impressive... whether or not it was humbly stitched together, you were sort of the first to do it, so take pride.
The next real magic will be reading its net and figuring out how to get [vfx/film] effects from it... which if I were you would probably occupy 22 hours of my day now.
It's admirable that you're so modest regarding the antecedent work, but sometimes it's the "obvious in hindsight" compositional insights that really open up the possibility space. Top work!
Maybe that's what we were supposed to do all along. Not find or be found by aliens, but to invent them.
In the last 10 year the work of a concept artist changed dramatically we have gone from purely painted concept art to mostly "photobashed". Photobashed means basically that you rip apart other images and stitch them together to get the desired image. Some start with a rough sketch for the composition or make really rough grey shade 3d model and "overpaint" them. When it comes to "photobashing" the disregard for copyrights was always there and it's the worst in smaller studios and a bit better in the leading ones. Still most of the time everyone argues that if you only use really small parts of the images it is covered by fair use. There are some examples were studios got sued but mostly without bigger financial impact.
A few months ago I started working with "DiscoDiffusion" to generate the images I use to photobash. "DiscoDiffusion" can produce great "painterly" images but struggles with photorealism and is slower, not as coherent as "StableDiffusion". Still the adoption rate in the concept art community was insanely fast. This all got topped by "StableDiffusion" in the last week. Ofc there are still people that want to do it the "right" way and not use AI but we had the same discussion years ago when "photobashing" came into place and some artists still wanted to paint the whole image. As concept artist you are mostly paid for your design thinking that means it is less about the process and more about the finished product. The turnaround time for styleframes got reduced from 3-4 hours while painting to 45 min - 1 hour when photobashing with stable diffusion me and my peers in the studio are now at 20-45 min per styleframe. When "photobashing" most people constrain themselves on their image library and ressources like Photobashing Kits. Not only does "StableDiffusion" cut the time in half it also gives greater freedom in composition and design especially if you are using img2img.
So where does this leave us? For the work in fast paced art environments like VFX, games, conept art or advertising "StableDiffusion" is a welcome gamechanger. Tradionalists and Artists outside of the industry might feel threatened but for us in these industries it's a god send.
So yeah, StableDif is great for commercially constrained environments, plus, it frees some of your time so you can enjoy creating art that matters to you :p
For me, that's the big insight.
Not just SD, but any synthetic media. My take? The "upcoming revolution" is not one of a boss firing their artists and making prompts themselves. It's a shift from addictive art" to subtractive art* in commercial environments.
Most of the discussion, has been on the technical details of these very interesting advances.
Do you think it will be concern for the production process, the origin of the source data used to train these models?
Do you have resources about getting into this style of art as a novice?
Also what StableDiffusion and other AI tools do you recommend?
On the other hand artists are amazingly adept at leveraging new technology, mediums and techniques to create art so it's probably less surprising if they jump at Stable Diffusion in a lot of corners of the industry.
And FWIW, I think these tools are still a little too limited to replace even a single person. They might make someone slightly more productive, but the modern stack for both programmers and digital artists is many levels deep. Automating any one of those levels is not sufficient to replace a human. You would need to automate every slice of the stack, plus the work required to stitch those slices together.
0. Parent, every input you put in that box is an input you soon wont have to make (and that will feel great at first) until you have no input to give anymore.
0.5 By art, I mean entertainment mainly. Not constrained to fine art.
1. Of course this looks like a tool now. And tools are great. We gota make sure they stay tools though.
2. Whatever work you do now with them, the AI will learn to do it by itself without you. You are training the AI.
3. The copyright issue is not unlike music and games in the early 2000'. You would pay if you were proposed the right delivery method.
3.5 the AI isnt creative, this is sophisticated plagiarism.
4. Moore's law. Moore's law. Moore's law. There are for-profit companies behind these AI that have no limit in the amount of value they wanna grab. Their goal is that we can click on a button and voila, you have a movie.
5. Please automate my job you may think. Absolutely, I wish mine (software developer) was. But art / entertainment is different. Now I cant explain it entirely, Im not philosopher (there is something about we still gota control what we do and do not) but if you automate art, our brain becomes useless. Our soul dies. And nature tend to recycle whats useless. Turns it into food.
6. "We ll just do some new art / entertainment". Which the AI will mimic the next day. So you cant even hope to work on entertainment anymore. It's a death I cant qualify.
The core of whats wrong here is ultimately, in a perfect world, an artist could choose if he wants the AI to scan its work (i.e. integrate its work in the tool) and be payed for it. That way we would choose what we like to do and automate everything else. If we cant do this, the AI will do everything even what we should be doing. So we should ban those models (or improve their practicality).
My first thoughts when I saw DALL-E were, "Wow, I can't wait to matte paint with this!"
A texture here, something robot like here, some technical or magical doodads there. Smash them all together and even use img2img to blend things, then do the finishing touches and lighting by hand. It's so nice for speed "painting".
As a programmer, it honestly reminds me of the situation with Copilot.
I would be pretty afraid and angry if i was artist too.
You have many professions who work with visuals and never would call themselves artist - designers, illustrators.
If you photobash all day to generate ideas its much closer to design than to what people would call artist.
Maybe thats a cultural difference but i always thought that distinction is clear. Anyone working in anything pop culture, commercial, “industry” is those other professions. People independly doing visuals (anything really) for visuals sake are called arists.
So when i mention artist i imagine completely different group of people than you.
The author claims to have made this with Stable Diffusion, Disco, and Wiggle: https://www.youtube.com/watch?v=Nz_n0qxqoPg
I believe Wiggle is used to automate the tilt/zoom between frames.
https://www.youtube.com/watch?v=CgDbbg802-8
Its incredible what 6 months only can do
Every day there’s a new application of Stable Diffusion. It’s incredible to watch unfold
I feel left behind...
If you are interested in my project (I doubt, you are too busy playing like me) I'm posting a lot of things on https://unshush.com and on the Instagram account: https://www.instagram.com/unshushproject/ (Sorry for posting my stuff but I'm not very social so no one will ever see them)
If you want to generate videos I can share some links I bookmarked of software/code to make them more smooth.
[1] Deforum's Colab (based on Stable Diffusion): https://colab.research.google.com/github/deforum/stable-diff...
You have generated some pretty cool designs.
If you make the generations with some similarities and use the right interpolation, you don't need 1000 images like my video and can obtain a smooth movement.
First, generate images with some kind of visual anchor (background, an object). You can use frames generated using the previous frame as reference image, or the same seed but different prompt/parameters, or you can go wild using img2img/inpainting (btw I struggle to find an inpainting tool for Stable Diffusion: they seem to be just img2img with a mask, without contest).
Then pass the generated images to one of the most recent interpolation algorithms, like this one https://github.com/megvii-research/ECCV2022-RIFE or the one used in the replicate we are commenting on (someone posted this reference: https://github.com/google-research/frame-interpolation )
The first link reports some free and paid implementation and a Colab, so depending on how deep you want to go, you have a lot of choices.
In the end, I'd use some good app to stabilize the image if needed, to get a more "calm" look. I use Luma Fusion, but it's a paid app (cheap, one-time payment, for iOS). I'm sure there are a ton of open-source implementations.
It's an approach similar to the animation on replicate, but it allows a lot of fine-tuning and you can add new animation ideas/tools to the process.
Nothing revolutionary, but I hope it helps!
> You have generated some pretty cool designs.
Thanks! I put in a lot of work in the last weeks. The project has a mission, I wrote something, but it's not ready yet. I believe it will be with the launch of Dall-E 8 :-/
I've always loved animation, but I'll admit part of that comes from the hubris involved. It's pure insanity that people ever drew, by hand, mountains of individual drawings each slightly changed and assembled them into compelling illusions to tell stories. The amount of work that goes into animation is just staggering and anyone sensible would have rejected the entire concept as absurd. I wonder if animation will start losing part of its magic for me when it's done primarily by AI.
On the other hand though, another thing I've always loved about animation as a storytelling medium is that it isn't as limited by practical concerns like physics or reality. If something can be imagined, it can be drawn and animated if somebody has the skill and the resources to fund the massive amounts of work. It's time/money that forces animators to take shortcuts and make compromises. Creative decisions are made and rejected all the time due to those constraints. If AI driven animation gets more advanced to the point where that's no longer such a barrier it could create output more in line with the vision of creators and that's exciting too!
I hope that traditional hand drawn animation never dies, but I look forward to seeing how AI continues to change the industry and the output.
These comments on every single post are getting really boring.
And so on.
Even the direct costs of using OpenAI APIs are not zero.
While I can't expect it to interest everyone, I don't personally mind discussions of specific industries when it looks like their time is coming up though. Each industry is going to have to deal with the change in their own way and we'll all have to adapt in different ways. The more interested in the industry I am, the more interesting I'll find it's decline/collapse. Brace yourself, because when AI comes for the coders that topic is going to dominate this site for some time (at least until the AIs themselves start commenting)
This water-morph effect is undesirable for inbetweens.
The artwork is inside a scene, SD does not understand that scene. The artwork has spatial and human readable emotional relationships. SD does not understand those relationships.
SD can maybe create morphs between frames, as a lateral move between two pieces of generated information, but it will never know how to connect up those images in a manner that satisfies the human requirement of creating a good image.
We already have mathematical tools for interpolating between frames. They are wholly unsatisfying for creating novel artworks. Adding SD to that stack doesn't magically solve that problem.
Your dream idea of killing the inbetween with mathematics would require automating what an artist does by hand to construct and bend space-time upon a blank piece of paper. Describing what an artist does takes time. CTRL+Paint does a good job. With mapping out every possible emotional/visual interpretation of those shapes, between the two frames and allowing the user to pick the resulting outcome.
That is the "tea, earl grey, hot" star trek replicator for art inbetweens. SD is just another tool for filling in gaps with random spam. The real value in this, is that there's a hoarde of young people who want SD and it's outcome. The real art will continue unphased, using SD as a tool, where it fits.
A few years ago I heard people saying the same thing about going from a piece of text to a picture that "satisfies the human requirement of creating a good image"
Inbetweening is not going to be the obstacle that these AI approaches are finally going to be unable to manage.
Yeah, never say never. If AI can replace artists, it can replace animators, it can replace programmers, and ultimately it can replace humanity altogether.
I think we all knew this in the abstract but the pace is a bit faster than anybody expects.
Traditional animation by itself is nothing short of insanity, convincingly blending live action and traditional animation takes it a step further, and then there's the "Bumping the Lamp"[0] scene in Who Framed Roger Rabbit.
The film as a whole refuses to keep the camera static to make things easy for the animators, which was unusual enough by itself, but then they went above and beyond — they casually bumped a pendant lamp and let it flail about. Every time it slows down, it gets bumped again. And they shaded and cast shadows for the damned rabbit for every single frame of that sequence. Madness.
Very impressive for the time.
From what I can see, they later revamped PhotoSynth to include actual 3d mesh reconstruction in 2014.
In morphing you end up with a blury mess for the between frames. This technique tracks individual features (eg eyes) and keeps them coherent.
I've had a hard time finding ongoing work on A. and B, perhaps it isn't much of a priority for research groups.
Imagine opening blender and typing
> A medium sized classroom, well lit, with two blackboards and many geography posters
And the AI just generates all the 3D meshes and places them appropriately.
Repeat that for other props or characters that you need. After that you can manually tweak the scene as you currently would (moving things, etc).
Then you select a character, and to animate you tell it
> The character calmly walks to the door and proceeds to open it
You could literally do a 100+ hour job in 5 minutes.
same mine arrives tomorrow, can't wait to try textual-inversion
- Mandela effect for famous art pieces: "Monalisa was AI generated" "No it wasn't" "Yes it was".
- Art critics will get the last laugh, as people start giving them truckloads of money to ask whether a piece of art is human or AI generated.
- Each image or painting should have a reference to the organization maintaining it
- The organization produces hashes for its artworks
- Users compare hashes
- Problem solved
https://twitter.com/dreamwieber/status/1565008078466326528?s...
It's a pretty magical time with this tech. Things are moving very rapidly and I feel excited the way I did when I first rendered 3d animation on my 286 from RadioShack.
Is there something like an index of cool new AI projects that is easy to follow? HN works for this to an extent but I’d love to track more closely.
What is getting very clear though and this link proves it out is that 'prompt engineering' is really a thing, I tried this out and it took a while to get something I would consider half decent.
I feel like there is a space here for tools / technologies to 'suggest' prompts based on understanding user intentions. If anyone is actually working on this then reach out to me. Email on profile.