Hyper-realistic digital humans in Unity
unity.com
unity.com
It seems to me that they are trying to prove that they are a serious 'AAA' engine, but these demos aren't that convincing to me - AAA is a lot more than putting fancy shaders on high-poly models, it's about handling huge amounts of objects in a dynamics situation, displaying large worlds via streaming, having a workflow that accommodates every creative professional, and offering great performance and visuals even on very complex scenes.
I've hear that even these highly impressive demos are fake - they built a ton of custom code for each one they rebuild core Unity features, meaning if you wanted to replicate this for yourself, you'd be in for a ton of development.
Comparatively UE5's Nanite demo showed off a tech that's ready to go for production.
In that context it would make sense to make demos for the capabilities that people don't know that you have.
They don't need to convince their already huge and already convinced core users, so there's not much point in building big tech demos that apply in that space. The aspirational AAA stuff that Unity puts out does 3 things I think:
- convince current indies with big dreams that their investment in unity has long legs if they grow larger and more ambitious.
- open up to non-gaming gfx tech sectors: broadcast, movies, simulation, etc.
- push their engine to its limits so they know where it hurts the most in places that actual users are not pushing. (it has many known rough edges and areas, but Unity users already report these)
I'm just starting to play with Unity, but I am playing with it at all because I can essentially learn along the way with them since they are still releasing things that I just don't quite need yet. This may not make sense to some people, but no offense to them... their opinions don't matter. We are the ones using this, not them.
But make no mistake: Unity is the reason Unreal suddenly got cheaper and open source a few years back. Both companies read The Innovator's Dilemma and both know WTF they are doing.
BTW Will Epic's ludocrisly, mission-distortingly successful Fortnite, like Wrigleys, cause them to become The Fortnite Company? (similarly will Rockstar become the GTAV Company?)
You'll find it on Steam and Epic and other gaming platforms.
Many years ago I was playing Runescape and a friend quipped that MMOs are basically Myspace with a game attached to it. And I think he was right. MMOs live and die by the network effect. Runescape had 150 million user accounts something like a decade ago, if not even before that. I'm sure World of Warcraft, Lineage 2, and League of Legends had at least comparable numbers.
I think if this were the next step of social networking, then it would have already happened. Perhaps it already did? We just call them gaming communities. If that's the case, then video games have created competing social networks that exist at the same time and aren't just being eaten by the biggest one.
Easy choice as to which is more authentic.
I guess it depends where do you live, though. Where I’m from Messenger is still the most popular messaging platform so most people (unless they have very strong feelings about FB) have an account.
There are very few close relationships that I have on Facebook. I have it because I needed an account for a while, and haven't deleted it so that it can function as a placeholder should anyone try to impersonate me.
Meanwhile, almost all of my friends have a presence on Steam - including some who I rarely if ever do or have seen in person.
parity: Equality, as in amount, status, or value.
parrot: One who imitates the words or actions of another, especially without understanding them.
The difference is Unreal is eating their own dog food.
With Unity, the price per seat means only people who leave will send a signal, which will be too late. They are not incentivized to make day-to-day work efficient, which is why there are a million small bugs.
[1] - https://www.unrealengine.com/en-US/faq - "This license is free to use and incurs 5% royalties when you monetize your game or other interactive off-the-shelf product and your lifetime gross revenues from that product exceed $1,000,000 USD."
I’m not sure how the overall quality compares, but I’d expect Unity’s users to be way more critical even if it had more bugs just because of all the indie/small developers who’d fall below the 1mil threshold for Unreal but have to pay for Unity.
This is a big one, and I guess will be true in the near future if not already. If their tech is easy / common enough, some sites will benefit greatly from this, such as virtual museum / zoo, better google street view interface, virtual space exploration and physics demonstration for educational purpose, etc.
Also, Unreal has enough money to burn (from Fortnite) to venture off into new spaces. Unity is taking far more of a risk here, since these indie gamedevs are still the main revenue flow (especially considering that Unity sells their engine in a monthly seat-based system rather than a revenue sharing model like Unreal). If these core users move away and investors become skeptical that Unity's going to make money (they're still burning tons of investor money every year without significant revenue), what we're going to see is a total disaster.
I'm a full time Unity dev, but have many years of Unreal experience, and we occasionally look at what Unreal is doing now. To be honest, I just don't want to code in Blueprints or go back to C++.
If I were building a multiplayer game, or a big open world game, then I might be convinced to give up C#, and go back to Unreal.
With regard to management of the engine, I think there have been some less than optimal decisions, (URP vs HDRP, Multplayer, DOTS) but its really a huge project I hate to criticize because I'm not sure I would have done anything differently.
It's all terrible right now but they will get there.
oof
GDK_SCALE=2 GDK_DPI_SCALE=0.5 ~/Unity/Hub/Editor/2020.3.25f1/Editor/Unity
The revenue from mobile apps is quickly drying up now that we are getting stronger anti-gambling laws. Because let's be honest, mobile games make money by hooking rich gambling addicts. That's why the industry calls them "Chinese Whales" and the "99.9%-ers" who are allowed to play for free so that the actual customers have enemies to crush with their pay-to-win items.
That said, I have a very low expectation of Unity-the-company to pull this off successfully.
Care to elaborate? As in, what sort of legislation is coming up widespread across countries that would harm the revenues of, say, Candy Crush?
https://www.bbc.com/news/newsbeat-49674333
"Loot boxes 'link to problem gambling'"
https://www.bbc.co.uk/news/av/uk-england-46295317
"Number of child gamblers quadruples in just two years"
https://www.bbc.co.uk/news/business-46286945
"Could Candy Crush Saga be a gateway to gambling?"
https://www.smh.com.au/technology/could-candy-crush-saga-be-...
"How ‘Near-misses’ in Candy Crush Increase Frustration, and the Urge to Continue Gameplay"
[..] "some players can, and do spend more than they can afford on these games (Lloyd 2016)."
[..] "Tellingly, Candy Crush players often specifically liken Candy Crush to slot machine play to convey its appeal"
I actually disagree with this on a much broader level: I think that every company should continue to convince their core users.
1. Their core users are always considering alternatives
2. The people using their competitor’s products will see how well you treat your core users and may look at their current company and see the lack of it. It’s like when a company chases new customers by giving discount after discount but never once offering a discount to great long-term customers. It leaves a bad taste in the mouth of the people who’ve carried your company.
It has been a while since Unity is not just a mobile engine.
https://arstechnica.com/gaming/2020/02/the-mandalorian-was-s...
I think if I were to start a high fidelity project with a big team, I'd start seriously considering the art pipeline benefits you're getting from Unreal. It looks like they're trying pretty hard to take some of the tedious work out of lighting and setting up LODs, which could be a big time savings.
People who are just getting started will see this and see that Unity must surely be capable of everything they need, and it may even be trivial there, if it can also handle this.
It's also a piece of aspirational marketing. It doesn't make sense to show the reality of the mobile game space as that won't get anyone excited. It's what you could do, not what you will do.
An anecdote from my studio: My studio is work for hire with mostly Unity devs. The "big ticket" contracts are in Unreal, so the studio is starting to push for devs to learn Unreal, so we can start landing those contracts. In this case, the publisher perception of Unity/Unreal is creating a business need in our company to move away from Unity and into Unreal.
source: work in marketing
> The short movie demos aren't impressive. It's embarassing when half your keynote is a group of artists explaining how they spent 6 months re-writing half the engine to render something that they could made in Maya in a week.
Really wish Unity stopped being recommended as a great first choice for hobbyists with a bit of programming experience in general. In my experience I've found it a lot harder to get going with it, and requires dealing with a lot of systems which are in various stages of deprecation or early access compared to UE.
> Show me a 100 player PUBG type game running at solid 60fps on a current gen console or mobile. We already know you can render a GTA3 cutscene.
So, as I understand, he complains that the demo team spends more time trying to make demos that looks like movies instead of making demos that are more focus on the game parts.
Unity is definitely a very stable engine with a lot of features and very nice and simple to work with. I know several developers who see Unity as a better alternative than Unreal for any scale because the development process is so smooth. As a graphics programmer I feel liberated by how easy it is to set things up.
I've also looked into performance quite a lot and it's not bad. There are issues, especially on certain platforms, and a lot of gotchas. But their analysis tools are really mature and help solve a lot of issues without having to load up external tools. It's definitely possible to make beautiful and performant Unity games.
This is the view they want to promote. What they don't want you to know is that the engine keeps churning out half baked features which never get fully fleshed out or documented properly. A lot of indie developers also complain about support but at least with their enterprise support you often get swift and helpful answers.
I fully agree. One of the top voted comments on the YouTube demo video even captures this pretty good in a humorous way:
> Can't wait to use this to make another low poly game
It's sadly even worse. To replicate most of this, you need access to the C++ part of the engine source code. If you're small (e.g. less than $10 mio annual revenue), they won't even discuss pricing with you.
games produced and distributed by a mid-sized or major publisher, which typically have higher development and marketing budgets than other tiers of games.
However, you do see plenty of exceptions like the original Dark Souls which took serious risks. Which was then followed by two lower risk sequels.
Can't afford to just keep narrowing down on existing customer base. They need to keep expanding and competing, for e.g. with Unreal https://youtu.be/bErPsq5kPzE
(Though whether that's just a couple years of software technology progress, or a decade+ for hardware progress, who can say?)
what does that mean? To me, they might have just as well said middle out compression.
There is one out there from 5 years ago or so that is similar to Google's Seurat but for animated stuff, I think pre-baking triangle culling for different views within a limited volume. I can't remember the name of it, from the details I remember (there was a realistic orangutan or something like that rendered with fur) I should be able to find it on Google, but Google search has become degraded recently.
> Tension tech for blood flow simulation and wrinkle maps, eliminating the need for a facial rig for fine details
Which sounds like it is not 100% prebaked animation?
chuckles
One baby step at a time.
With this, you just have the character do exactly what you want, when you want, without needing to talk to an agent.
Of course most games don't care about 5 years later, but it still looks crap.
eighteen million things happen around the character to distract the viewer's attention from the human character throughout most of the video
Not the most confident tech demo. Looks like a reasonable degree of evolution, but as they start to get up toward turn-of-the-century movie CGI in games, now you run into the same issues that you see in mocap for movies--including that a lot of details of an actor's expressions and movements actually aren't captured, so you have to have animators go back and laboriously add all those lost details back in to have something that looks convincing.
(Mind, I say turn-of-the-century, but man, the fundamental techniques for rendering skin convincingly have come a long way, even in current game engines.)
Every thing draws from the same well so to speak, a human that looked a little better but had to exist alone in a white room would have far less utility for real things.
Yes. Why did they do all those crappy CG effects while showcasing photorealistic characters? Two people seated across a chessboard in a realistic room would have been more effective.
It's getting to the point that everybody is doing this. Unreal Engine has Metahuman Creator.[1] Even Second Life has reasonably good heads now.[2] And has facial tracking on their roadmap.
It also has the benefit of necessarily head tracking as well, so you can also do TrackIR style head tracking which again adds a lot to the experience to be able to move your view around independant of the mouse and in ways a mouse can't move.
Interesting to see that hair apparently is still a real problem. It looks and moves very unnatural.
The skin and hair look great which are both really challenging for their own reasons, but one particular detail that was surprising was at 1:36. If you watch her lips closely you can see where the top and bottom lip stick together as they open. If they can simulate that level of skin detail when the motion is backed by mocap data this could be a huge quality jump for character-driven and dialogue heavy games.
There are definitely still moments where the lips, teeth, and tongue look "3D" but I don't think I would notice if I weren't hyper focusing on the mouth and just enjoying the story.
Consider how art exploded when people realized they could paint something other than, you know, reality.
Reality is boring. Get your realism off my games. Give me fantastical alien worlds to explore rendered in stylized and weird looks.
No one cares if you automate the painting of a black square. They care about the intention of the mind that thought it was important enough to do. That's why it's art.
Step 2: Don't tell anyone that you automated.
Step 3: Publish a personal account on what "experience" this particular piece represents.
That being said, I think there’s room for both and luckily, the rendering learnings from one often improve the other.
Nothing wrong with exploring both venues.
The idea that the ultimate endgame for video games is to have them look like a playable movie, or be a real-world simulation just seems silly to me.
Photography gave painting (and visual arts in general) a massive boost by the virtue of stripping the aspiration of going for "ultra-realism" in paintings. After all, that's what photography can accomplish much better and much easier. This made painters look for other directions to push their art into, which freed up artists' bandwidth for a lot of other, previously overlooked and not as popular genres, such as surrealism.
Not saying that photography is responsible for creating those genres, they existed before. But (relatively) cheap and accessible photography definitely diverted attention from ultra-realistic paintings to other painting genres that enjoyed greater growth due to that.
After all, what do you think will happen once it becomes very easy and cheap for anyone to have ultra-realistic 3D models and animation in videogames? It will become much less valued, as it is something that anyone can do very easily. So people doing 3D art will need to look in other directions to distinguish their work. I have already been observing it happen over the decades in gaming.
For present day examples: Nintendo manages to nail the style of visuals in a lot of their first party titles without even attempting photo-realism. Guilty Gear -STRIVE- (made by ArcSystem Works) is killing it with their over-stylized visuals. Atlus and their Persona series keeps pushing more abstract and surreal depictions with each game, and they resonate well with the audience. And I've only listed AAA-tier examples. Once you dive into indies, you can go on forever with lots of well-executed products that pretty much intentionally went against photo-realism.
What is truly astonishing to me is we are close, not quite there, but close. I remember when The Spirits Within came out in 2001 and we were so blown away. I look at it now and it feels so amateurish in comparison.
What if I told you you can have fantastical alien worlds using stylized and weird rendering that are also physically correct?
There are a ton of people who value photorealism, and that's why resources are being put into it. And it's certainly not seen as "wastefulness" by those people. Personally, I would absolutely love to be able to put on a pair of VR goggles and be unable to tell if I'm viewing a real-time render, or a remote stereoscopic camera feed of a real scene. Yet, I don't think that fantastical physically-unrealistic rendering is a "wasteful" use of resources.
I feel like these types of games really benefit of the work that is put in to making humans and environments look realistic.
Heck, even if you're making some weird alien world you're still going benefit from realistic physics engines for example, because that's just how our universe works.
Pixar and Disney have been among the most influential and revolutionary companies in terms of rendering techniques. And most of their products are stylized.
Stylized looks come from the assets, not from the rendering engine. The rendering engine gives you physically correct light interactions (which you almost always want, and even then you can turn knobs to make it non physically correct or outright do whatever you want using shaders), things like fur/hair rendering, fluid simulations, volumetrics etc. All these things can be tweaked to non realistic values by the artist.
If you want a movie/game where characters are made of wax, or fire, or smoke, or something even crazier with shaders, you want as a basis a renderer that is physically based and uses mathematical models based on the real world, and then tweak from there.
Here's someone developing an engine capable of rendering infinite, fractal worlds:
https://www.youtube.com/watch?v=V1MjZbGbi4I
Here's someone developing an engine with hyperbolic geometry (though dressed in mundane setting):
https://www.youtube.com/watch?v=VYfWfrk5P7w
Mirror Drop is an abstract art puzzle game that's kind of difficult to define:
https://www.youtube.com/watch?v=e1zHgYUE9HY
Here's a game where the speed of light is simulated and variable:
https://www.youtube.com/watch?v=uu7jA8EHi_0
The primary point of my argument is that we don't have things in mind that current rendering engines can't handle, precisely because our current engines are so hell-bent on replicating reality. Why dream of what your tools can't make?
You can't capture psychedelic and or mystic experiences, which are very subjective anyway and yes, rooted in _reality_ or require a reference to _reality_ as they are the so far away from that.
Do you believe that there were no genres of music born in the 20th century, or that we conveniently discovered all the genres just a few decades ago? Neither of those seems reasonable to me.
If you prefer not to accept cookies.
An effing stupid move to force cookies on a promotional site like this.
Foot, shot.
It's in that context that this post feels like another step towards achieving some promised vision. If this is realtime, it is truly fantastic.
Now to hope we can deal with climate change and despots and poverty so it's not all for nought.
As an aside, one early wow moment was the first time I saw a mini camcorder, then another for the first genuinely mobile phone; both Sony I think. Also, though a bit later, I remember seeing an in-car GPS and deciding it was basically a perfect interface for it's task.
Is Discord dystopian because people alone in their rooms have a headphone fitted to their head "just to be 'together with other people'"?
Is it that it's more immersive than just audio and therefore...somehow bad?
It’s literally senseless in comparison, lossy digitalization of basic analog humanity ie dystopian
VR is working to try and bridge all of these by essentially creating a transporter/teleporter to a shared physical space. Imagine if VRChat was a 1-to-1 replication of you. I feel like this is one of the end goals of VR.
Its easier to connect and find similar people through internet so i get why it is like this, but i think its also the factor thats making us less mentally balanced
There's a lot of room for improvement in the animation department, though. For me the worst offender is the mouth, it moves in a completely unrealistic manner. The second worst offender is the head/neck movement, which moves with robot-like precision. Finally, the eyes, which are (granted) not as "dead" as in other models, stare too much and communicate too little.
Moreover, we are extremely good at spotting something off with hands movement.
No, I can't imagine any benifits for modern tech from the 80s ;-)
Yes the image quality of a 4K camera is technically less than that of a film video camera, but show a 1980's movie producer a digital workflow and their mind would be blown. You shoot the picture and can then immediately review and even edit.
So 4K video beats home VHS on quality by leaps and bounds, and for anyone who would have been producing video on film the workflow for digital is amazing.
Anyone who doesn't experience a genuine "Holy shit, I'm actually living in the future" moment when they first encounter a Mavic Mini or similar modern consumer drone is probably in need of pharmaceutical help or talk therapy.
…
“It works sometimes, I swear.”
You don't exactly need time travel to get an opinion from someone who lived the 1980s... try asking your parents?
They'll have watched The Jetsons, Blade Runner, 2001, 2010: The Year We Make Contact, and a host of other science fiction that all made the year 2000 look like the year 2100. In 1981, we sure didn't think the Berlin Wall would fall and the USSR would cease to exist a decade later (unless that came about by an apocalypse).
I am way more disappointed on our societal progress than I am on our technological progress.
But to be completely honest, I was expecting at least a moonbase by now.
Instead, they got Tang
> The problem with Unity cinematics is that they are much less representative of their tools and engine out-of-the-box capabilities than Unreal's demos.
> They use heavily customized assets specifically for a video and even do a lot of custom changes to the actual engine held by hot glue and duct tape to make it work. Then you try to recreate only fragments of what was show and nothing works. Even their released demos are usually not very portable and useful for devs, while in Unreal you can pretty much copy useful modules and most of it just works. This is why devs struggle to make things that look like demos Unity released 5 years ago...
> Unreal is less smoke and mirror. They focus more on things that are actually available inside the editor (eg. the main character in Matrix Awakens was taken straight from the metahuman editor instead of spending hundreds of thousands of dollars on custom scanned model rigged by a huge team of world experts).
> Unity does much better job with their demos to impress general public, but look at comments on their own forums and you will see over and over again how many devs are frustrated with their marketing tricks that don't turn into something real.
> Some devs say that this difference comes from the fact that Unreal guys have to eat their own food that they make, because they also make games (like Fortnite), while Unity is more detached from actual needs of game devs and recently they became obsessed with film industry, even acquired Weta Digital that made LOTR and Avatar. It should be exciting and impressive, but the comments from their licensees (waiting for years to fix core issues) were quite negative.
[0] http://www.reddit.com/r/pcgaming/comments/tjasoe/enemies_rea...
I put my ganer hat on looked at their old demos. Demos always look great. I'll believe it when I see it in gameplay footage.
IME they tend to show off how tech will look 4-5 years down the line, about the time a modern AAA game needs to from production to release.
Unity's tech demo "Adam" is now more than 5 years old and I still don't see any Unity games coming even remotely close to that. Unity's tech demos are just hoaxes: Carefully crafted demos with custom engine parts and basically hand-written stuff like shaders and controllers all over. The irony is that these demos are primarily aimed at developers, yet you can't take a single piece of those things and just make it work on your own setup. In contrast when it comes to Unreal it's usually just a Ctrl+C and Ctrl+V away from success ...
I don't think this is because you can't just copy-paste an Adam asset into the Unity engine, however. AAA devs are going to leverage their own assets regardless.
I think that's more of an image problem than a "technologically possible" problem, however. Many of Unity's huge successes come from the indie side, the Ori's, the Hollow Knights, the Cupheads. And it has a very big grasp on the mobile market. Even Pokemon Go decided to use Unity despite the IP being backed by a publisher known for their in-house engines. so a AAA studio isn't thinking of Unity for their next Call of Duty, but maybe for their Call of Duty mobile title.
They seem to be trying to combat that sentiment with the likes of DOTS,HDRP, and whatever Ziva is trying to do, but those are still TBD.
sucks for me because I am weird and I find it hard not to pay close attention to hair in real life, sometimes. this makes flaws in rendered hair extremely obvious to me. :(
I also thought the lips were great. It was the interplay between lips and teeth that really threw me off.
That's a sheer compute power problem. If you have a big enough render farm, you can run physics on every strand of hair.
Even those parts?
The Italian senate approves this comment.
I don't see how a game engine is much different for the foreseeable future. It is so far away from under cutting humans in this domain in terms of time and expense.
Perhaps this is similar to how some people aren't bothered by bad kerning (to which I'm fairly tolerant).
The tech maybe is good, but I wouldn't know it from this video. The animation and lighting are just awful.
Title claims something the blog doesn't, and creates overexpectation
Go to 1:34 in the video, slow it down to 0.25x on Youtube, and press play.
Granted, the gliding across an unwalkable moving floor gear contraption didn't help avoid the perception that I was looking at an automaton.
Does this mean this was rendered in real-time?
by rob cupisz (Tech and Rendering Lead at Unity Demo Team) https://twitter.com/robcupisz/status/1505875759243612172
But live? On a single (hard to get) consumer GPU? That's seriously impressive. It makes me wonder how much of this is hand-tuned rigging and how much is physics based; if you tried to shake hands with this digital human using a game controller or VR rig, how would that look?
But at some point, we definitely passed it. The room is nifty, but mostly been done. But that is a pretty good person. Lip sync is a bit off somehow... I think perhaps just too overexaggerated in the motions. But I couldn't tell you from a still frame that wasn't a real person, in real clothes.
I also continue to find it amusing that we can build a person like that and sell it as commercial tech, but we still have to record people talking. (Though TTS has taken an interesting turn lately, after years of not much.)
I suspect part of the problem is that dialog in games is still largely "static"; if the writing is pre-canned then it does not make that much sense to try to develop advanced tts to act it out. The situation will become interesting if we manage to produce sufficiently dynamic dialog system where it is not feasible to use pre-recorded voice acting anymore.
It will probably become as routine as automatic dialog replacement.
https://80.lv/articles/enemies-the-working-process-behind-un...
All of this is just trickery to distract you from the uncanny valley. The real test of this rendering is "perfect" faces like Charlize Theron or George Clooney. Those are real people too, by the way.
I also notice that the latest "hyper-realistic" models are almost always female too. I wonder why that is.
'Uncanny valley' issues related to face-based (as well as body motion-based) realism in simulation for both games development and video production, whilst there have been recent very significant advances in addressing them are still a (pun unavoidable) rapidly moving target, a target where this demo highlights both substantial advances and still as-yet unresolved serious shortcomings.
The advances mostly relate to the introduction of machine learning and 15TB of training data into the content production workflow to reduce a rendering job from 6 hours per 50 frames to 3 milliseconds per frame as detailed in this article:
https://www.globenewswire.com/news-release/2021/12/10/234977...
Returning for just a moment to the UE vs Unreal thing, those results were discussed in an article about a demo created using Ziva Dynamics's Real Time Trainer and Unreal Engine 4.26, but Ziva was subsequently acquired by Unity.
The discussion I'd be interested to see here (or somewhere else!) is about how ZRT compares with (or perhaps even fits into?) Unreal Engine's Metahuman workflow and resulting content creation and realism results.
Perhaps more importantly, how might the ML-driven capabilities of ZRT (will they still continue to be available under Unreal as well as Unity?) and ML-based impact upon realism be expected to manifest themselves
Here's a link to the ZRT 'preparation workflow', which a Z-brush user who was trialling the trainer setup says took them about an hour to produce a workable result:
https://drive.google.com/uc?export=download&id=15J6vX-6vU5EU...
https://schedule.gdconf.com/session/inside-unitys-new-flagsh...
Actually they should have plugged in stockfish and let us play against their character.
Don't most games use realtime cutscenes these days? Nothing shown here is a gigantic leap to what you see in Death Stranding for example. I'm not looking at her and seeing an actress it still just looks like a very good 3D model.
But face animation - no way. It is freakish and screams fake.