Hacker News Reader

Comment by daveguy | Hacker News Reader

> How about we get GPT to turn good movies into great books? Like a large inverse prompt engineering problem.

GPT has trouble making great sentences, much less great books from a data type completely different from its training. GPT, at the core, is a "likely next word" generator. No great literature came as a process of "likely next word" in a vacuum. Likely next word from complex experiences of a decade and plot designed from those complex experiences, sure. But not the algorithm that GPT is.

> Could we then even feed that text as input to make a better, though arbitrarily different movie?

Nothing about a feedback loop of mediocrity describes a practical way to improve quality. This concept is as sound as the alien encoding stick and the sloot encoding scheme itself.

View on HN

What should be doable:

Deconstruct a movie into a set of 3D models, textures, voices, parameters for those 3D models' movement, procedural generation of background objects like trees, grass, sea, sand, weather conditions, etc, etc. Like the data that forms the content of modern, near-photorealistic games.

A game engine-like rendering system would then render a scene, tweak parameters until rendered scene / frames match the original movie closely, and work through the rest of the movie the same way.

When done, one could produce derivations of such movie by changing actors' voices, have them move differently or speak in another language, swap out buildings or other objects, reduce polygon count or resolution of textures, have a cornfield show a little taller stalks, put the sun in a different angle, etc, etc.

Of course this is way beyond current compute capabilities. Not to mention software frameworks to do this job. But in theory this should work. For a 2min trailer it would probably be pointless. But for a 2..3h movie, maybe not.

And yes, of course this would be lossy 'compression'. Just more high-level than current video compression schemes.

For an audio analogy: compare mp3 compression with MIDI + quality sound banks for every instrument under the sun + parameters like how hard a piano key was struck, etc. Vary such high-level parameters until rendered output matches the original music.

What is the limiting factor though? I could make a bad sketch and get a description of that used to render a better picture right? We currently don't optimize for this as a goal so could get better, no?

And how would you train a fitness function to recognize book goodness? It's inherently subjective.

GPT can make great sentences, it's just not the default operating mode. LLMs are capable of amazing things if you can get them into the right section of the latent space.

Reply on news.ycombinator.com