Hacker News Reader

Comment by embedding-shape | Hacker News Reader

> We found that the model's modulation weights (~40% of the total parameters) could be pruned and replaced with a functionally equivalent lookup table, dramatically shrinking the memory footprint with no loss in output quality.

Is this a common approach to reducing weights with "no loss in output quality", assuming this is true? Seems almost too simple to work. If this is doable, would this be applicable to LLMs as well?

Neat with native frame-to-frame generation, but wonder how easy it is to "link" together clips at the intersection, typically the models kind of lose the "momentum" across these stiches, being able to merge things with frame-to-frame between clips might help with this it feels like.

View on HN

It is a well-known trick, given that the timestep is between 0 to 1, you can slicing them at any resolution (1000, or 10000, give or take), and then keep a look-up table for modulation scale / bias etc for each. It is quite different from quantization and it is indeed lossless.

It is also only applicable to diffusion models as only these operates at per-timestep.

So... why didn't the model ship this way to begin with? They just wanted to waste VRAM for fun?

They ship a complete checkpoint for easily management (inference & training) in their own infrastructure. Moving to a LUT would make training on these layers impossible. BTW, these are not useful for lightweight fine-tuning, but might still be useful if you do serious post-training work.

Of course, these are also not an issue for things like FLUX.2 which adopts DiT-Air arch, that doesn't have this wasted space issue.

It may or may not be true. The people who made this modification and the other commenters didn't do anything rigorous to verify what they did. They just eyeball it. They could very well make some other error - this has happened frequently - that developing on prod, not knowing what they are doing, has and hopefully will again solve.

"Modulation weights" here refers to weights used to adjust layer normalization depending on the task (adaLN). General-purpose LLMs generally don't have those in the first place.

One thing similar would be projecting both the head.weight and the final LLM activations into a smaller vector space, since that is basically just cosine similarity ranking step (so that would reduce the head.weight size). But again, it must be tried many times and just not working as well. LLM space is pretty saturated with tricks.

Also begs the question whether this is applicable for high-throughput applications on FPGAs, which are to my novice mind basically LUTs, right?

I remember a paper which was posted on HN a few weeks ago where somebody implemented KAN networks in FPGAs, since those can readily be approximated as LUTs.

Whoah, could this mean we can treat layers like a jpg, where we come up with a formula that estimates the weight values of a layer instead of storing all of the weights?

This is more or less what those lovely 1.5 bit quantisations are already doing

This isn't the same with lookup tables, but in Explaining Attention with Program Synthesis they were able to replace some attention heads with Python programs:

https://arxiv.org/abs/2606.19317

Reply on news.ycombinator.com