Neural Actor: Neural Free-View Synthesis of Human Actors with Pose Control

This has been prophesised before (last in the early 2000s) and didn't come to pass.

I wouldn't expect that to change given this technology either. There's a market that would benefit greatly from this, though: marketing and training videos as well as stock-footage.

Some companies (Samsung comes to mind) already use synthetic avatars to represent their products. Artificial influencers and the like will also take the place of real humans.

Film actors, though, not so much. There's more to a successful actor than just appearing in film (who'd sit at meet & greets, autograph signings or attend shows?) so there's little reason to replace living actors. Deceased ones, though...

I dunno, when I first saw a Vocaloid on stage, my jaw dropped as I realized that in 10 years highly paid pop stars will be a thing of the past. Yet, it doesn't seem like music industry is switching to synthetic avatars.

The move to virtual characters is already happening on Twitch / YouTube, check out CodeMiko:

https://www.youtube.com/watch?v=CsQjxEd-gsw

When people talk about replacing artists with virtual artists, I think an important missing piece is that people are into performance because a human with a story is doing the performing, not just the aesthetics of the end result. For the most part, great art isn't hermetic. It's situated within the context of the artist's personal story and career arc, an the relationship of those to the audience / fanbase.

We probably will see greater uptake of virtual art and artists, but it hasn't been and won't be a revolution that sweeps away human performance.

My mind went towards stuntmen. Now, studios can record natural actions for a large database of actions, and pick and choose. Instead of the Wilhelm Scream, you could see "Flip #564" and "Crash #827". If it doesn't even matter what shape the stuntman is, it seems to me like the industry might be looking to fill that occupation less and less.

If anything, it's the low-paid actors that would go instead.

Highly-paid actors are celebrities. They are engaging in a way that is incredibly hard to describe but nonetheless seems to be widely recognized. They aren't necessarily "good actors" (a term that is itself hard to define), but people enjoy looking at them. People go to a film just because some well-known actor is in it, and that's what makes them so valuable.

Every film set is chock full of talented people. Plenty of people could have done great work as, say, Tony Stark. Hollywood is full of beautiful and talented women who just never manage to get that "it factor". To displace highly-paid actors they'll have to replicate that, and that'll open up a whole new set of questions.

The ones who go first will be the low-paid actors who surround them. They are the ones doing sitcom pilots that get shelved, or commercials for dog food and allergy medicine, or being saved from having a bus dropped on them. Such actors are actually reasonably well paid on the days the work (over $1,000 a day for big-budget films or TV; $9,064 for a week as a guest star), though most of them work only a tiny fraction of the time. There are more people making a living at that than being a star, though the odds still suck.

I think this technology is in its infancy and we might be looking at far more than 10 years to even get the functionality to make this possible regardless of whether it could actually replace highly paid actors. Occlusion and multi-agent are typically hard problems in image spaces, so once you add objects and other agents to a scene it becomes a great deal more challenging to make the motions life-like and realistic. Admittedly if you're mapping onto an actor in the scene with other objects and just changing their appearance it might be doable but you might run into substantial challenges if your target is 5ft8 and your source is 6ft2 (or Vice Versa). If you're not using a source-target mapping at all you have the challenge of generating realistic movements given all the obstacles and interactions, which is much harder than the current one-agent zero-object space.

That's especially visible when comparing with the "driving person", who is floating in a loose open trench-coat that flits about as they dance

Does Poser work with photos and videos as inputs?

If not, there's your answer.

You don't need to create or capture a mesh. This is synthesizing one from a few 2d images, with impressively accurate results.

Poser requires an existing rigged, textured mesh.

Poser requires that you manually animate the skeleton.

This approach is more like a mixture of mocap/photometry; given an input video (a) and (b), synthesise a fully textured model from (a) and apply the motion from (b) on it.

Of course, the output of this is horribly bad compared to state of the art 3d models; the point is that the inputs are trivial and it shows that NeRF are increasingly plausible as a solution for real use cases.

Of course... this is all a bit 'proof of concept stuff'.

> To further evaluate our method on a wider variety of body poses and more challenging textured clothing, we captured a new multi-view human performance corpus with 79 − 86 cameras at a resolution of 1285 × 940. It contains four sequences, 𝑁1-𝑁4, and each has 12,000−16,000 frames for training and around 8, 000 frames for testing

These captured images are also, notably, green screen, although not with the traditional mocap markers.

So, no, you're not going to get a 3d animation from a video off your phone anytime even remotely soon.

...but, it's still pretty darn awesome, and they're going to release the datasets for others to play with.

Not for me (v88.0.1, OSX)

No, not for me. I think you'll need to review file association actions in Firefox prefs. I have to do this regularly for PDF.

I'm on Chrome 90.0.4430.212 on Catalina and I had multiple videos with narration all start on page-open. Had to mute the page because I couldn't figure out which one was talking.