HyperNeRF
hypernerf.github.io
hypernerf.github.io
https://arxiv.org/abs/2112.05131 did such work and found you can get NeRF quality without any neural network at all and as a result 100x faster.
The reasons for using deep learning function approximators are manifold, for instance in RL, state and state-action spaces quickly become too large for tabular methods. Using grids or tables also basically closes off the opportunities for exploiting meta-learning and analysis-by-synthesis.
Plenoxels also rely on explicit specification of a known grid structure, where the HyperNeRF method can learn latent parametric manifolds and handle dynamic objects with changing topologies.
Extending plenoxels to support dynamic objects would be great future work.
Vanilla NeRF has just under half a million weights (if we ignore the coarse network), although more recent work has shown comparable results with only four layers. To compare apples-to-apples, we should probably also apply the (hand-tuned) TVD regularization used in plenoxels.
Plenoxels store 28 parameters per grid point, so at a 256^3 grid size this would be just under half a billion parameters. You don’t know which voxels you can prune at the outset (btw, DNNs can be pruned as well), but for real scenes they show sparsity reaching ~5% after 12.8k iterations. For a real 360 deg scene, because the voxelized volume is quite bounded, you also need to consider the 64 layer multi-sphere image required to capture the background. Basically, you need a fairly large GPU card to train plenoxels on any kind of real scene.
The option to make this space-compute tradeoff is still useful.
This is a great point.
Does anybody know of any papers that explore non-neural text to speech / voice synthesis that achieve better than parametric quality?
By the way, in the plenoxel video they say "the key component is the differentiable volumetric rendering, not the neural network".
After watching the fractal community struggle to come up with good distance estimators for years, this makes so much sense. In the end, automatic differentiation came out to be one of the most solid methods for coming up with distance estimators for an arbitrary fractal formula.
> Original one: "Nerf, Representing scenes as neural radiance fields for view synthesis"
> https://scholar.google.com/scholar?cites=9378169911033868166...
What I find most exciting about it is that a NeRF represents images as neural nets, one neural net for each image (in the OP paper generalised to image + deformations). By evaluating the net at various pixel coordinates it gives the color.
Up until now learning to replicate the input exactly was called overfitting and considered a bug, not a feature, but they showed a completely new way to wield neural nets.
An interesting detail is that they depend on Fourier encoding for the input coordinates. A variant called SIREN uses `sin` as activation function throughout the net.
Maybe neural nets will become the data compressors of tomorrow? Shoot a picture, send a neural net around. Game assets could be NeRFs.
Long a topic of research, with many interesting ramifications: https://ai.googleblog.com/2016/09/image-compression-with-neu... In analogy to standard compression techniques, you can think of the neural net as a very large "dictionary".
For text compression, there is of course the famous Hutter Prize, launched in 2006: https://en.wikipedia.org/wiki/Hutter_Prize ("Prediction is the golden key that opens all locks". Compressing each byte of wikipedia text is equivalent to predicting it-- to compactly represent its knowledge is to understand it.)
To avoid these sort of "joke" decompressors, they evaluate compressors on (size of compressed data) + (size of decompressor) in the compression competitions last time I checked. That means we won't get a winner based on GPT3 anytime soon. 350+ GB of weights is a lot to overcome :)
Though of course, given enough data to compress, it might well be that full-on neural language models are still worth it.
You're right, and in-there lays the crucial difference between compression and language modelling. Models are concerned with having a good representation for both past and future distributions, while compressors only care about the past. Models support many tasks while compressors are just for input replication.
A good illustration of this is that you can quite easily use a general purpose compression algorithm for classification, and get surprisingly good performance.
Well, for that particular ting there was a predecessor of sort in Deep Image Priors from 2017.
https://arxiv.org/abs/1711.10925
That was all about overfitting a neural net on a single image, which they used to get impressive inpainting, noise removal and superresolution results without any training at all (though of course it did not beat state of the art training-based approaches, even then.)
I had a lot of fun playing around with it when it came. The idea is dead simple, within reach to implement yourself with no complex mathematical understanding.
I actually had a similar idea and used it to write a lossy image compressor for my bachelor thesis in 2018: https://theses.liacs.nl/pdf/2018-2019-PetersO.pdf
It's awe-inspiring and even frightening at first, in the usual ways, but IMO it has a lot of long-term promise in other ways.
Spitballing: I like that this kind of result, which clearly calls into question the role or perception of physical identity, may eventually inform (or even necessitate?) the deconstruction of the physical "I" as a permission broker, and further open a many-to-many interface between the dimensions that underlay what we now think of as "self" and the true depth and variety within what we now think of as "individual humans who are not me". That opening process alone ought to be a huge jump for human development.
Right now we're each held, and holding ourselves, way too responsible for maintaining a singular subjective identity, looking at the aggregate. Not only does this compromise our outlook on others based on our subjective perception of the identity match, but it also compromises our ability to reliably consume and metabolize identity-construct-breaking information and experiences. And many of those things, when consumed without so many identity borders--so to speak--will end up being incredibly useful for individuals and group both.
Thanks for sharing op.
what uses do you anticipate for a more [fluid? porous? plural?] self-regard?
There is no reason now that Tom Cruise even needs to exist as a real person, or needs to ever act in a movie ever again. Tom Cruise can just become an abstract concept, no longer a living object. Perhaps it is Tom Cruise himself in these videos. Perhaps the real Tom Cruise no longer exists. Perhaps the whole thing is an elaborate art project. Our certainty of its falsity is tied solely to whether we believe the story of those who claim to have created the videos. Is it easier to create fake videos of Tom Cruise or to create real videos of Tom Cruise and a fake story?
What is a Tom Cruise?
So with that said, some ideas could be started around topics like 1) massive identity theft causing a re-thinking of identity 2) creativity and constraints around the moderation of physical identity and 3) technical-presentational dynamics surrounding physical presence and the moderation of identity presentation in a physical presence context.
Any one of these is a great setup for the question: How do we interpret personal identity?
And this--again just IMO--would be an amazing point at which to say, "look, if the only word-tool we can use is 'identity' to describe this crisis/opportunity, then maybe all we have is a metaphorical hammer and we have all these endless annoying nails--in the form of identity questions--to hammer down. But if we had maybe some other word-tools to use instead of 'identity', maybe this really would look more like an opportunity to move humanity one more step up the evolutionary ladder."
We already moderate our identity every day, either consciously or unconsciously. It's been studied for thousands of years. It's in books you've read, movies you've watched. It's been done for fun, for comic relief, and also it's been done to solve mind-shattering problems. But now we start to really unwind this question of physical identity, the one concrete thing we thought was so much more certain...! and things get _really_ interesting. This is a different level, where there's maybe not such a need to hide or hide from this departure from "this one idea of who I am" which is really just a mess of a complex of ideas.
> what uses do you anticipate for a more [fluid? porous? plural?] self-regard?
For one: More, and healthier, exposure to alternatives. Your identity is almost synonymous with your subjective past. To that degree, you're screwed in a lot of ways. To give a personal example, I was born into a cult. I was screwed from birth, in that way.
One of the best tools I had in removing myself from that environment was the concept of an "online identity" which could be moderated, intentionally, into whatever it needed to be to help me explore alternative perceptions of what it was I was involved in. I could even try on a non-cult identity, and write, online, from the perspective of someone who had freed themselves. And then I could consider how that felt, and reflect on what I learned. Did it kill me? No. Am I in hell now? Nope. etc.
Consider the millions of various points of identity just like that. Not just cults, no way! Am I Coke or Pepsi? eh, boring. Am I...which race am I? Is that a tricky question in the future? And from the outside, will I get better treatment from medical professionals if I can moderate my physical presentation at will? Wow so many random questions that can be asked for learning's sake.
But again, to emphasize--I love and respect the unknown. I don't have answers, only openness where I don't want to have certainty anymore, because that c-word makes it a little too hard to solve big problems, or a little too easy to avoid them. Don't leave the cult man, you'll lose all your certainty.
Hope that helps, a little, or examples, a little.
It would enable some really cool ideation and modeling, maybe even some of which could be used for psychology work, or sports psychology in the case of the NFL (I'm reminded of those "imagine yourself winning" tricks)
Could someone help me outline what's most interesting here? Maybe applications?
But this only works with rigid scenes: e.g. if you apply NeRF to images of a person, they cannot move between the pictures.
This is what HyperNeRF is trying to solve. If there are pictures of a person, and in one of them they are smiling but on another not, 1. this method will not fail, and 2. looks like it will give reasonable new views/images.
The channel is excellent and I recommend subscribing to it if you like this kind of stuff.
And for this paper in particular, I fail to see how they improve over other nerf approaches with deformation terms like Nerfies or D-Nerf
I presume something along these lines will make it's way to the Pixel 6 camera software, given the origin of the research and the onboard edgeTPU block.
Or am I understanding this incorrectly? Sorry, this feels very much beyond me.
- as a storage format for 3d models - it could become a method to package parametrised game assets, so you can easily customise them.
- as a representation for 3d inputs in spatial reasoning tasks, like protein folding and self driving