We used it to solve a very peculiar challenge: a video game level that needs to convey narrative beats by placing videos at certain points, that also had to look like a video game with realistic physics.
The challenge is that making a video game is a radically different process from creating motion graphics. In Cinema 4D, you create and tweak keyframes, and the first step to making something look good is locking the motion. That’s what lets you choose where things will appear onscreen and customize the frame. Whereas in a video game, you throw rules on top of a physics engine and then the player plays.
So Claude pulled real telemetry from games, built a physics model, and created a framework that tracked all of my notes and solved the camera, adjusting weights for different constraints to create different versions of the videos. It's got a player model that responds to threats with movements, and I can put my thumb on the scale with different beats that all get integrated into the solver.
For each round of notes, it puts one folder for each lane we're tracking, with deliverables and A/B's and a markdown file. I give notes, it tracks them, and we rinse and repeat.
Two weeks in and we are almost at a finished level, which we'll import back into Cinema 4D for a last style pass.
I've been pretty mind-blown by Fable's ability to track so many moving pieces and build a cohesive system around them.
This is the kind of work that LLMs have been truly bad at: controlling software that doesn't want to be controlled, tweaking hundreds of values that relate to vision and spatial reasoning. So we did a lot of rounds of adversarial testing. A recent report was "this looks like a handsome screensaver with a health bar" which came with 40 recommendations.
I asked Claude to make a 12-minute video going into detail about how it executed this and go into all the nerdy details. Video is in the Substack post above...Claude really likes byte determinism.