Hacker News Reader

Hacker News

Comment by sillysaurusx | Hacker News Reader

Parent Full thread

sillysaurusx

View on HN

This comment reads like it was written by a GPT

GPT-2 no less.

You may not be the intended audience.

This made no sense at all

A truly marvelous proposition, which these comments are too narrow to contain.

Maybe you're thinking of differentiable rendering?

https://blog.qarnot.com/an-overview-of-differentiable-render...

I have some experience with ML and I have no idea what you're talking about. It kinda sounds like neural architecture search and sparse models created using weight pruning. Lots of people are working on both of those things, but IMO the latter (if that's what you mean by "Can you add weights non uniformly?") is a dead end for most use cases where you have some sort of accelerator or deep learning instructions available like on every modern desktop, laptop, phone, and server. Models that use their weights "efficiently" in terms of FLOPs tend to perform like crap on real hardware.

I’m unfortunately the sickest I’ve been in years, so this will have to wait. Maybe it’s part of why my comment sounded strange.

There is an idea here, and it’s a mistake to dismiss it out of hand. Adding weights non uniformly during training (not after) is the key to smaller models that outperform present day GPT3.

A sketch of the algorithm is to start with a 2x2 block of weights, sum the gradients across 10 training steps, then subdivide the quadrant with the highest delta.

Doing this recursively is prohibitive, which is where megatexture comes in.

Many advantages. At runtime you don’t need weight compression because you can simply switch to a lower miplevel if running on a phone. Different accelerators during training can focus on different areas of the network. Weight dropout is an automatic feature. Etc.

If you’ll excuse me, it’s back to hugging the porcelain bowl.

You mention this technique has been shown to outperform GPT3, do you have a citation for that? Would love to read more details about this interesting concept.

they didn't say it was shown, he stated that those to be developed models would do that.

"Adding weights non uniformly during training (not after) is the key to smaller models that outperform present day GPT3." seems to greatly imply a certainty of result that has already been discovered. Though this commenter is familiar to me, and I know he has made silly claims in other threads throughout the years here.

I would hope so. It’s in my name!

I ended up calling an ambulance so I’ll postpone this until later. Feeling a little better but a full explanation will have to wait.

The answer is that of course it’s not proved yet, since no one has implemented it (or at least efficiently). It’s fine to be skeptical.

Current techniques are blocked by the technical challenge of getting 10GB+ to fit on a pod. Very few people have those skills. If there’s even a chance that this will work, it’s worth exploring, so I will be.

Sounds kinda like progressive growing except you're not doubling the resolution uniformly. See ProGAN and its successors. You'd still need to add a large block of weights at a time for performance reasons.

Edit: Ah I checked your profile and you already know all this. You probably should have mentioned that lol

Network compression is already a thing that is studied and forms of it are already used in production neural net models where latency / cost is important.

The way pruning works is not like how a "megatexture" works.

I think he's talking about lossy compression with some sort of high-importance areas where losses are minimized.

Reply on news.ycombinator.com