Comment by Demiurge | Hacker News Reader

Comment by Demiurge | Hacker News Reader

Parent Full thread

That sounds like an interesting challenge. Have you seriously considered solving it? Because in about 10 seconds I came up with a process that should work, provided enough compute power. Simply model the traditional film making process by starting with a script, character stories. Design your world, then design the storyboard, and all the scenes. Create a list of all the visual elements that need to be replicated between all the scenes. Then you have to built prompts and reference art of the objects, faces, people. Make sure to do multiple takes of each scene, and have the vLLM critique and analyze the performances and technicalities. Should work?

I think, also, like in the traditional film makers career, this process should be built iteratively, start with a fast food commercial, then do a music video, then you can probably do a short film. Continue to improve the process, and one day I’m sure the LLM film studio can make you any movie you want, provided you have enough tokens.

I'm getting a Poe's Law feeling. I'm genuinely unsure about whether this post is a stone cold parody or not. I think I need to turn off the internet and go to bed.

EDIT: Your username doesn't help, either.

Nobody wants to watch such films, they want to muck with the filmmaker, the generation apparatus. That's the product, if there is one here, and absolutely not the 3 hour epic that's spit out with 4 variations to choose between. That's work. We'll have other LLMs pointlessly tell us which should be watched, we'll view a summary, and vote the Oscar on that.

> Have you seriously considered solving it?

Sort of, but I want it to be relatively low on human effort. I feel burned by spending lots of time in 2023 learning image generation pipelines (using control net etc) only for that to be rendered trivial by the next generation of LLMs.

This movie would be only for personal consumption and I’m okay with waiting for model improvements.

Control nets are still useful in Krea and H3. For images, Krea can usually get close enough to a reference that it's not a big deal, but for H3 conditioning makes a big difference over prompting for complex actions.

the problem is you have to make the AI watch the whole thing to make sure it works.

I've done this sort of with comfyui/same agent factory stuff, but the verification loop only works for models like fable as planner/writer, with gemini as verifier for like a very short movie. Sub 3-5 mins. After that you burn through million tokens.

Can't go too low fidelity audio/video or it craps out. Too long video and it loses consistency. Look at only snippets, it lacks global consistency, etc.

On that topic, check higgsfield cinema studio 4; they already provide amazing tech for the cinematic experience, somewhat similar to what you are describing.

Reply on news.ycombinator.com