Generative Image Dynamics

Impressed and sad with how toxic HN can be sometimes

Just sharing some art here, and getting downvoted without even a comment about it. And even getting flagged

@dang care to check what’s going on here?

These just look like vibrating images on my phone

You'll see those things! We already have DLSS, for instance. But unfortunately we can't simply glue expensive black boxes onto games and ship them. Wrangling performant, richly interactive media is difficult enough without these models. This modern ML + gaming fusion space is barely in its infancy. We need to explore what's practical, and discover patterns to do it.

Even without further breakthroughs, the next 5 to 10 years will be incredible. I'm so excited.

Wouldn’t say tech is there yet. It still needs a lot of human input and direction so slapping it into a video game would just be immersion breaking when it generates something out of character randomly.

There’s less impactful ways to implement it like generating art paintings in a museum dynamically but that is in the “a little gimmicky” territory.

I've been playing Red Dead Redemption 2 recently. When it comes to 3D nature scenes, this game is state of the art, and incredible in its attention to detail.

Still, one of the things that stands out to me is how fake the small movements of leaves, grass, and flowers in the wind still feels. The actual physics of tree branches as you push past them isn't bad, but you can't use the physics engine for every leaf. The wind in trees and grass is still just a randomized oscillation of some kind that doesn't quite feel real.

So I was immediately struck that this technique would be an improvement, perhaps using animated billboards. But it would come with its own issues - can't be done in real time any more than physics, for example, so it would be hard to use it for real time weather systems. And anyway, in that case since you're using a transparent video, there's probably easier ways to make it.