A sneak peek at MetaHuman Creator
unrealengine.com
unrealengine.com
With human-like agents that have human-like voices, appearances, and mannerisms, they'll be able to communicate with millions of average Joes, and all in a way particular to that specific Joe. AI being able to do infinite amounts of "ambient work" and expose us to the results via human-like entities makes for a very different world of possibilities.
And right now the big AIs that most people directly interface with are clumsy and abstract, but they're also tied in the backend to large pools of closed data owned by mostly well-meaning entities. But how will things change when and if the AI Joe interacts with isn't directed and contained by wealthy US-based corporations?
Their AI-driven avatars are already in use in a handful of places, and fairly well received.
"The future is already here – it's just not evenly distributed."
— Appearance-wise, I think MetaHuman's stuff looks better than Soul Machines' — but SM's avatars are backed with a conversational AI, and are not available as a UE-plugin. So it's not an apples-for-apples comparison of course.
[0] https://www.soulmachines.com
[1] https://www.auckland.ac.nz/en/engineering/community-engageme...
[2] https://www.ibm.com/watson/advantage-reports/future-of-artif...
I don't feel like this is an accident. But I don't know what to think of it, I'm still a bit confused about what we're really trying to get out of these facsimiles.
What are we going to get out of it?
An ad inside your game that is a male/female/whatever of your preferred attributes selling you things with a seductive voice. Or a company selling that avatar to you as a in-game companion/skin. Or propagating services through that avatar.
Looking here, it fits into TenCent masterplan. Sell an avatar/avatars to users as companions. Add it to snapchat - see it in AR!. Let it read a spotify audibook to you! Buy it clothes and skins, play with it in Fortnite or Roblox! Imagine all the possibilities of upselling, crosselling and overselling virtually generated items (your only cost is data and electricity - potential unlimited) for virtual characters that people will start to feel emotional ties to since they'll grow up surrounded with them or to fill an emotional void left by lack of socialisation.
This was the source of a lot of complaints about Cyberpunk 2077. They made a lavishly detailed world, but it was all a millimeter of paint over nothing. Food stalls that don't sell food, cars that disappear when you're not looking at them. As it turns out, graphics are a lot easier than behavior! We should have guessed this, but while you can see all the complicated visual detail that should be hard to replicate, behavior is invisible.
Shamus Young had a good post drilling allllll the way down into the nitty gritty details in one small AI subsystem of a game, Thief 3, and what it would mean to upgrade it: https://www.shamusyoung.com/twentysidedtale/?p=270
>In the game, you can sneak up on anyone (servants, nobility, or guards) and whack them on the back of the head to knock them out. (You can also stab someone in the back, but that’s noisy and bloody, and why kill them when you can just knock them out?) Once they are knocked out, you usually need to hide them. If you leave them laying in the middle of the room or hallway, someone else is likely to come along and discover your work. When this happens, they always assume the victim is dead. Then they start with the running and the screaming and everyone searching for you.
>This can be amusing. I was in a large manor, working my way through a sleeping area for servants. One of them was already asleep in the bunk beds, but a few others were still wandering around. I zonked one of them and placed the sleeping victim into one of the beds. I thought I was being clever. Another servant came in, saw their compatriot in the bed and exclaimed, “Dead!? But who could have killed him? I’ll go tell the guards!” Then he ran off.
>Blast it all.
>I should have known better. The AI was just looking for knocked-out people. It didn’t care where the body was. I was so into the game I stopped the metagame thinking about AI and started thinking about what I’d do in the given situation. In that situation, placing a zonked person in a bed made a lot more sense than dumping them in a corner. However, to the AI it was just a poorly hidden body. Sigh.
>But fixing this problem would be tricky, and would involve a lot of extra work. The level designers would have to designate certain areas or objects as “beds”, and the programmers would need to make it so that bodies laying on beds rouse less suspicion than bodies found elsewhere. Then they would need to add some new dialog and behavior: If an NPC sees someone “sleeping” on a bed (most likely not in their own bed) while fully clothed and while they should be working, he shouldn’t ignore them, but he also shouldn’t run away screaming about murders and dead bodies. You need some new behavior along the lines of “try to wake someone up and then discover they have been knocked out”.
>But even with that extra effort, you can still have some amusing failure modes. Placing a servant girl on a bed in the priest’s quarters or the barracks should raise some eyebrows. Likewise, stacking two or more people in the same bed should tip off guards and servants that something is amiss. It wouldn’t make sense for them to just assume they all decided to take a nap together.
>It would be annoying to code in such a way that it works right. The programmer would probably need to take the victim’s position into account as well. If I just toss somebody on the bed so that their upper body hangs over the side and their head is resting on the floor, it’s going to look pretty stupid if someone comes along and assumes they’re asleep.
>Also, it seems like the length of time since the NPC’s last saw each other should be taken into account as well. If you greet your fellow housekeeper, walk out of the room, and come back a few seconds later to find them motionless in someone else’s bed, you are not going to think they are sleeping.
And that's just one tiny detail, resulting in a vast increase of work. Our old friend, the combinatorial explosion.
A goomba in Super Mario Brothers just walks back and forth. A couple lines of code. Doing better than that is not ten times harder, it's millions of times harder! Who would pay tens of thousands of AI programmers for a decade to make a version of Cyperpunk 2077 where each noodle vendor had a name, a home, a routine, a simulated economy... If you could do it, which you can't, it would an incredible shining gem of technical achievement... but would it make the game more interesting to play? Spend a thousand times as much to make the game 1.4 times more interesting?
But there's other facets besides this description of what Thief might do. A lot of a game is aesthetics, visual details that don't affect gameplay at all. There are behavioral aesthetics as well, which can be more incremental. They don't have to interact in clever ways with gameplay. They don't have to be resistant to the player hacking the system.
Small cases might be the way bystanders react. Or conversations you might overhear. The backstories of Dwarf Fortress are like this.
But there is a danger you are creating a pretty but ultimately annoying facade on a very fixed system. Games that put fancy conversation in front of a store interaction aren't fooling anyone. Still, if you can invoke an emotional background to a game I think it would be meaningful even if it doesn't significantly affect the optimal gameplay.
It won't be a machine in all cases. You have recorded stuff from mo-cap stages and maybe somehow you consider that a machine you don't care about but would care about a movie (not sure why the distinction?).
But aside from that, VR headsets are also getting mouth cameras and eye tracking cameras that can track facial expressions. You can also facetrack from things like an iphone with face id while playing a game in 2D.
Notice that they're realtime rendered, so they don't look completely realistic, but that is the easy part that has been solved already. It's possible to render those characters with offline renderers and they will look photorealistic.
As far as scans go, it's much cheaper than you would imagine to get high quality face scans. Rigging them for animation is the real challenge, and I'd be really impressed if they were using GANs for rigging.
For state of the art in this space.. check this out: https://twitter.com/ak92501/status/1311832183078879232
I can imagine they might be using scans that they fit their model to to generate some blendshapes, but it's just as likely that they don't.
Just look at the rig controls, it's exactly a better, and more polished version of rigging controls that have been used in animation studios for years. I think they are just doing what everyone else is doing, but with just a ton more money.
Edit: May be it is the Real Time Rendering.
But the real innovation is achieving this in hours and not weeks or months.
[_] Can this be used for porn?
I think we have a winner here.
(Other examples: 8mm film, VHS, the Internet, newsgroups, JPEG, streaming video, etc.)
Announce trailer: https://www.youtube.com/watch?v=S3F1vZYpH8c
Sample characters: https://www.unrealengine.com/marketplace/en-US/learn/metahum...
Free alternatives I've explored are at least an order of magnitude worse than this, and often difficult to integrate.
No piracy, no software ownership and all IP stays in Epic’s datacenter.
"You’ll also get the source data in the form of a Maya file, including meshes, skeleton, facial rig, animation controls, and materials."
Of course, that's not really the point. The point is that the metahuman creator tool apparently makes it very fast-and-easy-ish to create "metahuman" characters that look a notch or two better than the Final Fantasy movie from 20 years ago.
The older faces that they made in the video that demoed the tool itself, rather than the result, were pretty impressive.
And then you get to teeth, which are like wet rocks. You're building this thing that is going to bomb if it can't do soft, meaty, hairy things convincingly and then there's this itty bitty section you're only going to see sometimes that is a dark membraneous cave full of wet rocks.
My guess is that teeth haven't been that interesting to date and that the modellers don't have access to hundreds of reference pictures / geometries of teeth for their modelling. As opposed to reference images of skin textures, etc. Tongues are probably the same. In fact I don't think I have ever created a render of a figure with their tongue out.
Once they take the time, the teeth should be easy to improve.
Some form of subsurface scattering[1] is a must, and not easily done in games/apps because you can't really "bake" it like you do for most other textures to run performantly.
[1] https://en.wikipedia.org/wiki/Subsurface_scattering#:~:text=....
Subsurface scattering is the correct answer. It's a very small area and a very expensive calculation.
Teeth are translucent and the coloured lighting on them changes rapidly.
Technically speaking an animator might know. A lighter would have a better idea. And finally a shader writer could tell you exactly.
If you were interested in the more technical breakdown of responsibilities in the industry: (of course it'll change from studio to studio, but this is pretty good)
https://discover.therookies.co/2019/05/22/job-titles-in-3d-a...