Photorealistic Video Generation with Diffusion Models
walt-video-diffusion.github.io
walt-video-diffusion.github.io
Everyone of these generative AI demo pages feels exactly like that. Nothing new other than we've done the same thing as someone else type of releases.
Now, I'm willing to say I'm jaded, so if this represents something we haven't already seen umpteengillion times already, where did they bury the lede? It's all still uncanny valley. The wheels on do not look real. The wake from the viking's vacuum does not look real. I'm not sure what the corgi is walking on, but it doesn't look like the ground.
Except you don't have to hire anyone and you get results in two minutes, not two days later.
These folks wanted to make their own thing so they made their own thing. Eventually someone will make a thing that's far better than anything we have today.
I also get the sense that in the early days where data curation matters, research teams need to hire curators that can siphon good from bad examples. E.g. don't train on shit stock footage, pick up a book on fine art photographers, train on that shit + good films and you'll get a pretty good model.
In the future I suspect curation will matter less and less because in the limit you should train on literally the entire internet.
I'm definitely excited for where it might go though. If it ever develops to a sophisticated point, which it most certainly will, we'll be able to do stuff with it that is just impossible by typical production means.
Agreed, this is no more impressive than Stable Video or any of the other things like this we've seen.
This is the film to digital transition. It's going to be rough for a bit, then take over everything all at once.
The upside for these editors is that they'll be able to be single-person studios and pursue their own vision without meddling from studios or massive capital, equipment, and personnel needs.
You say out of business. I say, mediocrity all the way down from here.
Feels "Old man yells at cloud." There will be lots of garbage, for sure, but more people than ever before will have access to creation.
Becoming a director means getting access to a very limited playing field. Very few domestic productions get greenlit per year due to the intense financial, logistical, and personnel requirements. Very few people are able to make the opportunity cost sacrifice to learn the tools and people network.
For all the people that write fan fiction, web comics, or have a DeviantArt, Itch.io, or Newgrounds, this is going to give them access to their own personal Miyazaki/Scorsese tooling.
It's going to be awesome seeing passionate people finally have access.
It much more like, old man has sat through many a product demo and seen the brochures, but what's in the box rarely matches. If this is what's being demoed as the examples they've culled together, the tech is lacking.
Anyone could go and make their own clerks or reservoir dogs and some people are going out and trying. Where are all the incredible movies?
Even forgetting the tools where are all the incredible scripts since everyone has access to books on script writing and a computer?
You were wrong last time you debated me on AI porn. It's here [1, 2].
Movies are next.
> Anyone with a few hundred dollars can buy mics, steady cams, action cams and more off of amazon that would give them quality above the black and white or 16mm film that some directors started out on.
I've been down this path, and frankly it sucks.
The task: tell a visual story.
The current process: script, edit, break down with AD, location scout, secure filming rights to a location, go to a rental house and get equipment, get insurance, find camera crew that can handle it, rent lights, get gels, go to a prop house, pick props, rent props, transport props to set, set dec, run electrical, light set, do blocking, rehearse, film, retake, film, reset, capture photons, smear pixels around with an editing package for three months.
This ceremony freaking sucks. It's so far divorced from the objective at hand, it's more primitive than punch cards, and nothing is reusable. I absolutely hate it. And I've always hated it. It needs to die. These steps have nothing to do with storytelling. It's vestigial and is an artifact of our primitive technologies.
The task is to convey visual stories. Not to play dress up, logistics, and awkward gridded photon capture.
Anyway, movies are next. I'll eat my hat if they're not. The current filmmaking process is as dumb as bloodletting. And it feels about the same.
The current process:
This is not the current process. Anyone can buy cameras, steady cam rigs and wireless mics from amazon then edit on a pc.
Where are you getting this idea that anyone can't buy stuff on amazon and edit on their computer?
Why aren't you answering the questions of why there isn't a glut of high quality scripts, sitcoms, tv shows and independent movies? At some point you will have to accept that these things are difficult and it isn't just about how easy it is to make images. Anyone can write a book or start a youtube channel, but that doesn't mean that anyone can do it well. The best people are still the best.
Just wait. Generating > hosting.
> Where are you getting this idea that anyone can't buy stuff on amazon and edit on their computer?
It's fucking difficult.
> Why aren't you answering the questions of why there isn't a glut of high quality scripts, sitcoms, tv shows and independent movies?
Because the US market can only bear a few thousand theatrical and television productions a year. There are more than a thousand directors that would love a shot to be greenlit, but the capital, personnel, and logistics inputs are too high. Compare to the number of software engineers, fan fiction authors, web comic authors, etc.
> At some point you will have to accept that these things are difficult and it isn't just about how easy it is to make images.
Momentary technological limitation. At some point in the future, our daydreams will become a live canvas we share with others in real time. We're getting there.
That might be a long way off, but some of the next steps along the way include easy to use generative film tech.
> The best people are still the best.
You'll never know who could have been the best. The best director might have been bound by opportunity cost to earn money for their family working two jobs.
You said your predictions came true in your last comment, now you're saying "just wait". If "generating > hosting" (whatever that means) why are all the companies making the most money the ones that host?
It's fucking difficult.
Anyone can buy stuff off of amazon and set it up. We can see that by how many people have youtube channels. Anyone could make their own movie and put it on youtube but we don't see much of that at all.
You'll never know who could have been the best.
You haven't confronted why we don't see the same stuff in scripts, novels, plays or other forms that don't take the same amount of effort.
The truth is that it takes practice. It isn't about some innate ability as if there is a good will hunting cinematic genius sitting doing nothing because he doesn't have generative AI.
Didn't you say "hollywood" would collapse in a year? (forgetting for a second that hollywood is a city and paramount is the only movie studio actually in hollywood)
Other than that, these demos will eventually develop into tools. You get to decide what to make with the tools, they won't do it for you.
Which seems to have been endorsed by one of the best retouchers I know of (https://www.youtube.com/@pratiknaikedu)
A line of sight, however disappointing is still better than no line of sight.
Maybe it will turn the open source efforts onto these things that are announced.
2. ???
3. Profit
Here is an example I made yesterday, probably around 5 minutes of computing on my 3080ti: https://imgur.com/a/4dPv1hC
It's pretty much the standard comfyui svd workflow with extra upscaling.
this would be good enough for promos of those!
https://github.com/thecooltechguy/ComfyUI-Stable-Video-Diffu...
Right now everyone is coming out with video models, and they're all very similar.
Stability, Meta, Google, Nvidia, tencent, Alibaba all have released either demos or code for Stable Diffusion based video generation. They're all fairly similar but slightly different architecture.
Any helpful tips?
Also can't say no to a cat video (even if it doesn't really remind of Van Gogh).
The fact that we can synthesize stuff like this from text makes me believe the real world obeys some laws of physics we still haven't figured out.
Specifically, the fact that when we interpolate in embedding space, the interpolated point still represents something almost always physically plausible (not always though, witness the surfing cat that doesn't get wet underwater).
I am playing around with SD right now - pulling sleepless nights because it's so fascinating, but I do get to see the patterns.
Generating unreal, magical stuff is actually a lot of fun, but if you want it to generate something that makes sense - good luck. It is really hard.
I can get Elizabeth Olsen into a Halo suit in a war zone on an alien world, but getting her to sit in a car like a normal human being has been impossible. It's always some twisted horror show. The predictive model does not know what to do with it and how it all fits. That's after you clean up all the anatomical nightmare fuel like the fingers situation.
This is why most of the SD artwork you see are magical elf girls in the foreground or some unearthly monsters.
A well-defined subject in the foreground (a human-ish one), it will do VERY well. If you really want to get your imagination going with every little detail being near-perfect, you will be busy.
Oh, and the text - it obviously cannot do text. It's in the right place, but it looks funny.
This is amazing tech, but it's not about to go sentient and kill me at night.
Like you said, as well, it's impossible to do some of the stuff that seems super simple but based in reality where completely dream-like images are generally pretty automatic. It's cool and, when I have another project that I need it for, I'll go nuts with it again but I kind of lost interest in just generating images to experiment because the experimentation stopped netting interesting results. It was just always the same stuff.
> obeys some laws of physics we still haven't figured out.
That statement might have been over the top, but I think they're right with respect to the information. Spatial logic, optics, signal processing, reasoning. It's beginning to feel like there's an interesting representation that cross-cuts these spaces in a powerful, generalizable way.
We can't synthesize stuff like that from text. It's synthesized from a(n obscenely large) combination of annotated still images and video clips. The text guides the synthesis (or perhaps better to call it reconstitution) process.