Designing deep networks to process other deep networks | Hacker News Reader

Designing deep networks to process other deep networks

developer.nvidia.com

yo dagw: how to design neural networks that can process the weights of other neural networks that can process the weights of other neural networks that can process the weights of other neural networks...

Seriously though, Nvidia seems to be getting better and better not only on the HW side, but on the SW side too. Some of these more recent projects from them are bordeline magic (e.g. ray reconstruction in dlss 3.5)

Seriously, when you own the hardware and your hardware and software teams collaborate, this is what you can do. To the point about Nvidia basically tearing down barriers, it’s astonishing not just what they can do with the chips, but at the efficiency they can do it. The next 10 years are going to be wild.

Since there is no code in this post, it explains the lack of any technical questions or responses to this research here despite the extreme hype in AI.

It only takes several experiments to re-implement the findings in this paper and find a special use case for them to get an edge over the current state of today's inefficiencies with deep neural networks.

But obviously, no-code means little to no interest here or reading this post.

Interesting, from my perspective, they gave you everything you needed to reimplement this method with that description of the W_M x W_M matrix![0]

You know what related idea from programming that corresponds to? Dynamic dispatch and type overloading.

Let’s say you had a datetime type that measured time in eg Wednesdays since July 1st 1983. But then you wanted to re-parameterize that in terms of unix time eg seconds since Dec 31 1969. So you add a routine to convert the old format to the new format such that it represents the same thing.

Turns out many ml primitives have associated parameters and parameterizations and can be converted into equivalent things cast in a new parameterization.

I think the real contribution here is probably the completeness of the library of implemented a->b transformations, and the method for optimizing the overall distortion when things have to be lossily approximated. For related (but not really overlapping) works you might want to look into Git Re-basin and ZipIt! Merge, as well as the numerous quantization methods.

[0] that said, much like almost all ml projects, the juicy bit will probably have a very concise description, but in reality require thousands of lines of code to make concrete..

This is how Neuromancer meets Wintermute.

As long as humans maintain control of the chip fabs, and self-assembling chip tech is yet to be discovered, we are okay.

If you loosen the time constraints, there is a compelling case for self-manufacturing vacuum tube systems. They're more resilient to space environments and easier to manufacture than their silicon counterparts. Who says Von Neumann probes need to be fast?

Nvidia management: How can we not just double the need for AI chips, but make the demand exponentially greater? Hmmm ... here's an idea ...

Nvidia have played Universal Paperclip.

Reply on news.ycombinator.com