Using NeuralNets to make smooth character animations | Hacker News Reader

Using NeuralNets to make smooth character animations

techcrunch.com

154 pointsadikus21 comments

What I'm really curious about is the latency of it. Having smooth and accurate animations makes for a very rich experience, but in most games having your character react instantly to your commands is more valuable. Even if it's a 100ms delay in order to accomplish this, it might still be too much especially for fast, action packed games. I know in the Witcher 3 they reduced some of the complexity in animations to make them react quicker. What this does sound amazing for though is NPCs.

You just need to keep your models in GPU memory... which the game is going to want to use most of already

but maybe in the future we'll have fully NN autoencoded games!

I think you're misunderstanding his point - it's not a question of latency due to the performance of the algorithm, it's a fairly fundamental tradeoff between smoothly blended animations and low latency response to user input.

In many games the user wants to see close to immediate response to their input, e.g. moving the stick to turn or pressing a button to jump but that inherently conflicts with the desire to smoothly transition between animations.

In reality, your brain starts planning an upcoming turn or jump and moving your body accordingly some time before reaching the physical obstacle you are avoiding. Control schemes for games generally don't allow for this kind of anticipatory input but rather you press the jump button at the moment you want to jump. This leaves no time to smoothly blend the animation - by the time the game receives the input it should already have started the blend some time ago.

Exactly. In about 2002 I worked on a PS2 game where you played a dog (Yes you could do a turd. No you couldn't lick your balls).

I spent ages tuning the code that determined how high to place the dog's centroid above the ground. I gained a lot of respect for the amount of geometry calculations a real dog has to do in order to keep its face out of the floor when running. The code needed to start reacting to a probable landscape collision hundreds of milliseconds before it happened in order to prevent the need for unnaturally large forces.

Fascinating.

Obviously the thing to do is to predict user input. Capacitive sensors within buttons to try to detect when your finger is above or resting on a key? Or just hall effect analog buttons, so you can tell when the button starts to go down before it reaches the bottom of the depress action.

The paper says "Runtime" is 1-2 ms. I'm under the impression that this is before any compression, as well.

I don't think the GP was referring only to the speed of the algorithm. If the system adds more frames to the animation in order to smooth it out then the animation slows down by definition, unless the frame rate is also increased.

If you play a game like Street Fighter 2 you'll note that the animations for many of the basic attacks are extremely brief, something like 11 frames in duration. This may further be reduced by animation cancelling, which allows a new animation to override the currently playing one, speeding things up.

I'm generally skeptical of the ability to speed up the frame rate in order to smooth things out anyway. Players tend to use certain key frames as cues to help them properly time their attacks. By running at a higher frame rate, you reduce the temporal distinctiveness of these key frames. At any rate, I'd like to see and judge the result for myself.

Exactly this. The character control in some games (eg GTA 4 and 5) feels laggy and imprecise because stopping, turning, reversing takes a lot of time. Yes, it is "natural", "realistic" and it looks nice if you're watching someone play, but playing yourself it feels like steering an oil tanker :)

I, on the contrary, like some inertia in movement because, as you've mentioned, it is more realistic. Sure, it get some time to get used to (especially if you had played Quake style shooters for a while), but I like inertial movement more than weightless ragdolls.

Yes, but it is very game dependent. Maybe this "engine" does not suited for fast, action packed games, but I think what players hate is annoying and logically/physically wrong animations, not smooth and fair. Also you can train the NN on faster more reactive training data.

This is amazing. It should even be possible to have characters traverse obstacles differently depending on their in-game skill level.

Imagine this system used to implement rock climbing. The player is simply pressing the "W" key to go up, but depending on their character's skill level the speed and manner in which traversal occurs could be very different.

I really hope Bethesda is paying attention to this, because open world games could use it more than anything. In Skyrim you climb mountains by mashing the jump button. Far from ideal, and tends to break immersion.

The tech could even be used to power melee animations some day, perhaps entire fighting styles.

Fully agreed. While Skyrim Horse Physics™ was entertaining in its own way, something like the OP would be a huge leap forward for immersion.

It would likely have secondary effects of easing the burden on animators, allowing them to concentrate on more important aspects of the game, like facial interactions.

(Looking at you, BioWare.)

That's a good point. I suspect neural nets would be suitable for near-perfect lip syncing animations that aren't too expensive.

We can already synthesize the speech itself to the point it sounds real, although it's very computationally intensive at present. Infusing the speech with emotion remains very difficult. Some day voice actors will probably just license their likeness along with a sample of emotive recordings.

After that, one of the few remaining holy grails is generative dialog for NPCs that's both believable, dynamic in relation to the game world, and don't sound like glorified ELIZA bot output.

Neural nets can probably be used for level design today. One use case might be creating an entire urban environment complete with residential interiors, where the artists don't have to slave over each individual apartment for it to be believable. Imagine playing a war game where the levels look like there's actually people living there.

Speaking of emotive, synthesized speech: https://lyrebird.ai/demo

Clearly early days, but still, the promise is there.

Ah yes with the breaking of spring the annual SIGGRAPH paper flood begins

A glorious time indeed

The results are amazing. I am do not have a trained eye around animation, but it looks no different than motion capture.

So far, correct me if I am wrong, the state of the art prior to this would be something like Unity's Mecanim, where you define a state machine for animation transitions, which would interpolate animations to save you work.

You can do both 1D and 2D blendspaces in Unreal, so you end up with something like this: https://80.lv/wp-content/uploads/2016/05/walktorund2.gif

Very handy for character locomotion, but only gets you so far. I'm not sure how applicable offline processing is to this though, since if you have a large number of different types of obstacles to be traversed, you'd have to bake >= that number of animations, and your animation state machine would be enormous. The end goal might be to do that in real time, but I don't think anyone is going to seriously suggest running a neural net to calculate animation positions while the game is running. Maybe if you had some smarts in the level designer you could determine the potential pool of animations needed based on placed geometry, and build the animations at packaging time along with the state machine.

Well, the expensive part is to train the network. After that it should not be as expensive, depending on the size.

If that is still not enough, they can also try to look at ways of producing a "good enough" effect with a smaller network.

Then, it can also be limited to only some characters, the ones that you are more likely to pay attention to.

Now, the MS Kinect does some intense processing behind the scenes (random forests) to capture your motion in real time. Yet you can still run games with decent performance. I think a neural network to adjust animations is not too dissimilar in terms of performance cost.

Source paper is here:

http://theorangeduck.com/media/uploads/other_stuff/phasefunc...

One of the references used throughout the paper is this:

http://www.gameanim.com/2016/05/03/motion-matching-ubisofts-...

I would always trust actual concrete implementation rather than a research project though... unless they have source and demo available.

Reply on news.ycombinator.com