Hacker News Reader

Comment by crazygringo | Hacker News Reader

Just wow. This is the first time I've seen AI generate convincing human movement. (And the folds of fabric in dresses seem to move realistically too!)

Of course, the actual movement skeleton is coming from presumably real motion-capture, but still.

I'm curious what the current state is of generating the movement skeletons, which obviously has a ton of relevance to video games. Where's the progress in models where you can type "a burly man stands with erect posture, then crouches down and sneaks five steps forward, then freezes in fear" and output an appropriate movement skeleton?

View on HN

The input poses appear to be generated from OpenPose [0], which uses regular images as input. With the creation of stable diffusion video, you could theoretically prompt it to make a video of what you wanted, then run it through OpenPose.

But I think the more realistic approach is to take a video of yourself doing the motions you want the AI to generate, and run that through OpenPose instead.

Using just words leaves a lot to the model’s interpretation. I feel like you might wind up spending a lot of time manually fixing little things, similar to how you might infill the “wrong” parts of an AI generated image. It might be easier to just take a 15 second video to get the exact skeleton animation you want.

[0] https://github.com/CMU-Perceptual-Computing-Lab/openpose

This is already highly, highly relevant for 2d animation.

Many complex moves (especially dancing) are filmed in video first, then the movement is traced over by hand. This is called Rotoscoping.

This is basically auto-rotoscoping, and I expect it to see commercial usage within popular high budget projects within 2 years. Previously, even the highest budget anime couldn't really afford 2d dance scenes due to the insane drawings required.

As I recall, Ralph Bakshi used this to mixed effect back in the 70s with The Lord of The Rings animated films.

Rotoscoping is directly tracing from film frame by frame. Most of the time animators instead use what are called reference key poses. For extremely stylized characters it can be difficult if not impossible to use rotoscoping for complex sequences because the proportions make the motions impossible. Instead specific frames where there's resting motion in the sequence are pulled, and the animators translate the positions of the joints to the unusual silhouette of the character for the key frames. The Castlevania and Masters Of The Universe: Revelation series from Netflix use this quite a bit for realistic characters -- it was heavily used during the Trevor and Death fight and Skelegod versus Savage He-Man fight -- and Rise Of The Teenage Mutant Ninja Turtles and High Guardian Spice used it for their stylized characters for perspective and fight scenes.

If this technology can instead be used to not only find the key poses but also roughly determine the inbetweens for a heavily stylized character like Rocko Wallaby or Blossom, Bubbles, and Buttercup then it is extremely useful.

> Many complex moves (especially dancing) are filmed in video first, then the movement is traced over by hand.

If the "animator" is a hack.

It still doesn't look ready for primetime to me. When the characters' hands move, they either gain a billion extra fingers or morph into fleshy shapeless clubs. It's just hard to tell because the size of the video samples are small and poor quality. At 1080p, it would look extremely bad.

The individual example videos are nearly 1080p (1024 height) and they look pretty good to me. There is some glitchiness on the hands but it's very minor and could definitely be fixed easily in post-production or through further refinements. In fact, if you play the high resolution videos and pause at random points, it's hard to even find a frame where there are extra fingers, though there are a few.

I don't think that text to movement will be desired. There is a lot of room for interpretation there, and if a creator already has a vision for movement, they would probably choose to have the model mimic a video of them acting rather than trying to get lucky and correctly describe a set of movements.

To the contrary, I think text-to-movement is going to be huge for videogames especially.

I don't see any other way to smoothly link 1,000 possible movements to and from each other, including when there are various fixed distances between you and a ladder or ledge etc.

I think models will learn "movement personalities" the same way they learn a particular celebrity's voice -- everybody moves with different rhythms. So your big burly action-hero character will move with a totally different rhythm from your waif-thin ethereal elf.

But there will still be a textual vocabluary that generates the motion -- "stealthily creeps to the door 2.2 meters away, taking 6.3 seconds, and then suddenly and dramatically opens it with a flourish".

Inverse kinematics have been solving most of the animation blending you're describing here for years. Not to dismiss completely the potential of this, but there already pretty good, reliable ways to solve these issues. I believe the person you answered to to be correct in most cases, there is a lot of nuance lost if done by text.

Text to movement is highly desirable. There is a long way to go before tools like this are used in AAA games, and blockbuster films, but as much money as is tied up in those areas, even more is tied up in advertising and lower tier content production pipelines.

Look at tools like Adobe Animate and Character Animator, these tools are absolutely capable of doing interpolation between frames while giving the person operating the tools a fine degree of control. When you can use text to movement to quickly create tens or hundreds of sample scenes quickly, then manually iterate on and edit them, instead of hand drawing and compositing, the value proposition is pretty crazy for time and budget constrained production where tradeoffs on quality and fidelity are acceptable.

> Of course, the actual movement skeleton is coming from presumably real motion-capture, but still.

I recall seeing from-scratch learned realistic motion somewhere. I think was meant for character animation that would more realistically interact with the terrain and cross obstacles in video games. So that should be synthesizable too.

Reply on news.ycombinator.com