StyleNeRF: A Style-Based 3D-Aware Generator for High-Resolution Image Synthesis
jiataogu.me
jiataogu.me
1. They generate a low-resolution position map using NeRF => Good camera and 3D control
2. They then upsample that with StyleGAN2 => Good features in the final 2D image.
So in effect, this is a StyleGAN2 network where the latent space has the 3D-related components split off into a different generator network.
it's not a nerf based solution but since the goal is high resolution image syntesis..
That said, the results are actually very impressive.
But I'm concerned about what solving this is useful for, especially with the focus on generating human faces. Obligatory Jurassic Park quote: "Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.".
I can imagine this technology to be used in the entertainment industry (movies, video games) as a way to save money on art. But it's quite obvious that this technology can be weaponized and used in the propaganda industry (if it isn't already).
So, this could save money in fun applications on one end, and provide means for mass opinion manipulation weapon on the other end. I am certainly biased but I'm not sure the recreative use case is worth the risk of the weaponized one (a bit like using tactical nukes for a firework show).
What am I missing then? Are there any other use cases that would make this technology actually desirable for the greater good or humanity, or is it just a weapon with some potential recreational use cases?
Rather, I think of it as "if we create an AGI, it will have required the ability to imagine things. Also, we have huge datasets of faces that all sorta point the right way and have a good medium of feature complexity, and we're extremely specialized in spotting visual errors in faces."
Consider the similar if slightly smaller research in creating photos of living rooms. (Or OpenAI's famous "comfy chair in the shape of an avocado.") Same reasoning. It's not about faces, faces just have beneficial properties.
>I'm concerned about what solving this is useful for
As for all research, it is pretty hard to predict what and what for something will be useful (despite researchers being pushed to say stuff like "my research is going to cure cancer" to get funding). But about neural networks for image processing, they certainly start to be useful for radiology.
There are quite a lot of harder and potentially even more useful if solved problems that receive significantly less attention and funding in this domain. Vast majority of GAN papers nowadays are just different applications of them, without significant contributions to underlying theoretical foundations.