Nvidia tool generates full 3D models from a single still image

It seems like an April Fool's joke to me. The dramatic narration overlaid on lumpy terrible-looking cars made me laugh out loud several times. I mean, I'm sure what they're doing is technically impressive but the results are downright awful. All the cars look like they're beaten to hell and back and I honestly wouldn't be able to pick "KITT" out of a lineup of any other black car if it weren't for the cylon light on the front.

I was thinking that the results looked pretty bad, until the video mentioned storyboarding and it clicked to me. Even as low-quality as the models are, they still can work great as placeholders. So this would allow very quick iteration on designing a overall scene, pulling items from reference images from google searches etc, and when you are happy with the result you can send it to an army of 3d modelers (or designers or whatev) to be built.

So the comparison point should probably be like grey boxes or whatev people have been using as placeholders/throwaways so far, and compared to that these generated models actually look pretty great

Definitely not great... I bet that in a few years I bet it will be shockingly good. Experts have probably been working on it for years, and now that they've chosen a sufficient model structure with effective convolutions and whatnot, it's just a matter of tweaking the parameters and feeding it more and more data (3D scanning tools are making this easier as time goes on too).

That’s just the “texture” view. There’s also the parts and materials view that helpful segmented the various pieces of the car.

I’m not sure why they chose to show off the “texture” view without even explaining what it is.

Also, while it may look iffy right now, it seems they’ve done the heavy lifting. A round of polish and in a couple years nobody will be laughing at this. (I’m already impressed, personally).

The horse is hilariously bad:

https://youtu.be/gz5E9wszZSI?t=53

>This gives creators of any skill level the ability to easily generate models for diverse uses - storyboarding, pre-vis, game content, architectural models, and more.

It sounds like they expect you to use it when you just need a vaguely correct shape anyway, at least as a starting point for further refinement.

Kitt itself looks like a Nissan with a rounded butt :-)

That’s because nvidia hired like every graphics researcher.

The paper: "Image GANs Meet Differentiable Rendering For Inverse Graphics And Interpretable 3D Neural Rendering"

https://arxiv.org/pdf/2010.09125.pdf

I like to take a line from Two Minute Papers[1] - If this is what one "paper" was able to produce, imagine the improvements will be two "papers" down the line.

[1] https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg

Same. These results are far better than I (with little 3D modeling experience) could generate.

A pro could do better, but a) do you have a pro available? and b) can they afford the time to model all of the item you want to throw into your mockup?

Are you referring to procedurally generated content?

https://en.wikipedia.org/wiki/Procedural_generation

.kkrieger got a lot of attention for using this method and only being 96KB.

https://www.youtube.com/watch?v=2NBG-sKFaB0

Well, there's Minecraft. Its worlds are huge and procedurally generated.

As to faces, Skyrim does that. There aren't unique face meshes for every NPC, there's some base meshes and various parameters to adjust texture and shape.

I mean is file size really a concern? I would rather have a 300 GB game with something like quixel megascanned assets than procedurally generated crap.

The latter sounds nice but in practice game execs just use that to cut corners and you end up with an unrewarding grirnd fest of a game.

It's not just HD graphics, but uncompressed or less-compressed game assets that are provided as such to minimize load times

It can still be the same in every instance of the game -- generate from a known seed. This is basically good data compression, customized for faces.

Spoken dialog is a killer. Studios have to contract voice actors and then it's very difficult to make tweaks later. Why not fancy text to speech with variables for intonation, smoker/nonsmoker, age, gender, accent, etc.

Going even further, things like GPT can generate text given an input. It's beyond me but couldn't something similar be used to describe environments on the fly to be generated, or generate NPC stories on the fly given inputs?

I think a lot of this tech is coming together and games are about to get a whole lot larger, more immersive, and cheaper to produce. Even just replacing voice actors would be a huge leap.

You might be overthinking the problem. There are games that would lose half their weight replacing uncompressed audio with mp3.

Present games just completely ignore file size because compared to GPU 1tb SSD are cheap and multi TB HD cheaper.

There's a trade off between processing time and storage space in pretty much everything to do with games. For example the lighting in a scene is often precomputed offline and then baked in to a texture which ships with the game. This increases the speed in which a frame can be rendered, but also means that your game just gained another few mbs in weight. Similar thing with animations of clothing, these are often physically based simulations done offline which are then baked in (again we often use textures to store the information, mapping RGB to XYZ).

If you were to generate faces "on the fly" as a level streamed off disk then that would likely take up too much of your frame budget and your game would grind to a stuttering halt. The other way of doing it (and this is generally how games which use procgen work) would be to either have a precompute step when you first install or launch a game. This might give you a long install or launch time and would still mean that you have a huge install size, but you would avoid taking up a chunk of your frame budget. Or you could do some clever scheduling and level design where you force the player through "tunnels" between areas that gives you enough time to generate a few new faces and load in assets before they enter an area. This is a pretty common technique where you need to load in a bunch of assets or do some heavy calculatuins. The Witcher 2 had an interesting and slightly buggy version of this where whenever Gerald opened a door to go from outside to inside the camera would swing around so you couldn't see inside the building, a second long animation would play and then when the camera turned back so you could see into the now fully loaded interior if the building. It didn't work properly but it was a good idea.

You could do what ms flight does and have a locally installed low resolution world and a high resolution streamed world. Adapting this you could stream in procgen'd faces and other items generated offline. I think this would work rather nicely.

Lastly, I know it was just an example but still, faces are probably the last thing you want to procgen with out passing the results by a human filter as humans are incredibly sensitive to them looking "off". If all your characters end up looking grotesque and off putting you might be shooting your self in the foot. See the furore about eFootball's terrifying crowd models.

I think most open world AAA games have done this for over a decade. They either vary a base face model randomly, or mix from hundreds of variants, hairstyles, clothing options, accessories etc. Literally no one is making 400 unique models for a crowd of 400.

Yes! The tech mentioned in the article solves this problem: no more "we have to remove/compress all of these models/textures because our game can't be 300GB". Just use 10 high-res pictures of the object from all angles instead of a model. This could save orders of magnitude of disk space

It also solves the problem of "we have to choose what objects to model because our 3D artists don't have infinite time". Now the artists review GAN output and tweak what the algorithm gets wrong, creating tons of content without needing to create every vertex from scratch. Very exciting

I remember as a kid messing around with early desktop-computer photogrammetry software: the kind where you have to print-out some black+white marker circles and place them around some trinket you want to capture in 3D.

..it worked, but the output *.3ds files were low-resolution and the textures only looked good from one direction - and the material's lighting was way off.

So the concept is nothing new, but it's never as simple as taking even a bunch of photos, let alone 1 photo - there's so much information necessary for future renders that cannot be captured from photos alone.

Maybe art is worth its market rate.

I don't know of one yet. Lot of people are working on it, using various strategies. The grail being able to provide a virtual fitting-room experience. Or monitoring fitness of individuals.

A 2021 paper with code (I haven't tried it): https://github.com/akashsengupta1997/hierarchicalprobabilist... to generate a 3d morph-able human-shape in the correct pose from a single picture.

A picture always has scale uncertainty (a 2-meter human viewed from 1 meter away look the same that a 1-meter human viewed from 0.5 meter away), so that is an additional problem that must be taken care of.

But now recent phone have 3d sensor that provide information that could be useful.

To generate 3d human models there is also makehuman . In the old days there was a soft called facegen, that could generate 3d face models and could automatically fit their parameters to two pictures using an iterating refinement procedure.

Deep-learning usually estimate everything jointly in a single step so they are faster, but often less accurate. But there exist models that learn to refine a previously generated model, so you can apply them repeatedly and get improved quality (Denoising Diffusion Probabilistic Models is one generic class of models that does this).

totally different

This is photogrammetry

You might be thinking of Looking Glass.

usually its NERFs derivatives

Yeah, it was nothing like this.

Please share link to video.

Hard to believe that Mr. Daniels is still alive all these years later.

"Intelligence" seems kinda hype-y to me. Machine learning is mainly just statistics. We still have just a ways to go until real "intelligence" afaik