Hacker News Reader

Comment by Der_Einzige | Hacker News Reader

Controlnet is adapted to video today, the issues are that it's very slow. Haven't you seen the insane quality of videos on civitai?

View on HN

I have seen them, the workflows to create those videos are extremely labor intensive. Control net lets you maintain poses between frames, it doesn’t solve the temporal consistency of small details.

People use animatediff’s motion module (or other models that have cross frame attention layers). Consistency is close to being solved.

Hopefully this new model will be a step beyond what you can do with animatediff

Temporal consistency is improving, but “close to being solved” is very optimistic.

No I think we’re actually close. My source is I’m working on this problem and the incredible progress of our tiny 3 person team at drip.art (http://api.drip.art) - we can generate a lot of frames that are consistent, and with interpolation between them, smoothly restyle even long videos. Cross-frame attention works for most cases, it just needs to be scaled up.

And that’s just for diffusion focused approaches like ours. There are probably other techniques from the token flow or nerf family of approaches close to breakout levels of quality, tons of talented researchers working on that too.

The demo clips on the site are cool, but when you call it a "solved problem," I'd expect to see panning, rotating, and zooming within a cohesive scene with multiple subjects.

Thanks for checking it out! We’re certainly not done yet, but much of what you ask is possible or will be soon on the modeling side and we need tools to expose that to a sane workflow in traditional video editors.

Once a video can show a person twisting round, and their belt buckle is the same at the end as it was at the start of the turn, it's solved. VFX pipelines need consistency. TC is a long, long way from being solved, except by hitching it to 3DMMs and SMPL models (and even then, the results are not fabulous yet).

> Haven't you seen the insane quality of videos on civitai?

I have not, so I went to https://civitai.com/ which I guess is what you're talking about? But I cannot find a single video there, just images and models.

https://www.youtube.com/shorts/ZN-NbdFwfNQ

https://www.youtube.com/watch?v=3WWy98ylLT4

https://www.youtube.com/shorts/1vqOjYWEF84

https://www.youtube.com/shorts/jOIb9QbrhZ8

https://www.youtube.com/shorts/C3F_YI84TXA

https://www.youtube.com/shorts/4IqJHozY4F0

https://www.youtube.com/shorts/h3OmBLlm5-g

https://www.youtube.com/shorts/ZT7tuIgSDRk

https://www.youtube.com/shorts/WnUYbsOMyvs

https://www.youtube.com/shorts/BKKqX2aMlSg

The inconsistencies are what's most interesting in these videos in fact

Not sure I'd call that "insane quality", more like neat prototypes. I'm excited where things will be in the future, but clearly it has a long way to go.

A small percentage of the images are animations. This id (for obvious reasons) particularly common for images used on the catalog pages for animation-related tools and models, but also its not uncommon for (AnimateDiff-based, mostly) animations to be used to demo the output of other models.

https://civitai.com/images

Go there, in the top right of the content area it has two drop-downs: Most Reactions | Filters

Under filters, change the media setting to video.

Civitai has a notoriously poor layout for finding/browsing things unfortunately.

Reply on news.ycombinator.com