That still looks like a “converge faster” paper.
https://arxiv.org/abs/2006.10732
The above provides a nuanced theoretical view. GD inductive bias is probably better unless your model is misspecified
424 karma · joined May 16, 2017
https://arxiv.org/abs/2006.10732
The above provides a nuanced theoretical view. GD inductive bias is probably better unless your model is misspecified
Results may vary :)
> Again, and I keep repeating this but it seems to be ignored every time: this is experimental work and it's still in progress. This story of small group-sizes on large models should not be an issue.
Apologies if something I said (or I guess did not say...) offended you! It's a hypothetical, and one IME is not so easy to achieve, but maybe you have different results. So I didn't want to comment on this, maybe it's possible (but LLMs don't scale up as easily in terms of quantization than other networks like image classifiers, in my experience).
> The extreme quant buys you potentially 70x more efficient matmul via binary/ternary operations.
To be clear, such hardware does not yet exist, and it's unclear if you really can have more efficient binary/ternary matmul if you need high-precision accumulators and more frequent broadcasting shiftss. It's again a complicated hardware question to answer if the sum total latency of doing many high-precision accumulations and many scales/shifts will be smaller (or, chip-area-wise, even feasible to implement), compared to a 4-bit baseline.
You could just do a layer-by-layer fetching scheme with 4 bit weights.
For training too, just fetch each layer twice per step as needed for fwd/bwd.
And all for hbm cost equal to one layer’s worth
Why not do the same optimization for layer weights themselves?
1 Could you post the full memory use of the methods? E.g. you include quip metadata in its GB but not hqq metadata in its GB.
2 If you have to go to cpu to shift and scale, how did you get latency lower than pure on device? Was this bsz1? No speculative decoding?
3 how can lora absorb shifts with only increasing rank by 1 if you have a shift per group?
1 - is it fair to use ram in two places and report only one of them without any asterisk? (If you think this is fair-oh boy wait till you hear about my 0GB hbm use inference algorithm)
2 - i know how subchannel quantization works. Are they hitting those reported latency numbers with per layer cpu pingpong to rescale?
> We compared the performance of the Llama2-7B model in three configurations: FP16 (full precision), HQQ (without fine-tuning), and HQQ+ (with adapter layers) using a group-size of 8.
Interesting, what is "group-size of 8"?
From their HQQ post (https://mobiusml.github.io/hqq_blog/), it's the block size at which they add scales (presumably 16-bit) and shifts (in that post, it's 8-bit).
So for every 8 binary weights we have a 16-bit scale and 8-bit shift?
> Fine-tuning with Low-Rank Adapters
They say they inline the shift into the LoRA but how can you do this, block-wise, without increasing your LoRA rank by num-blocks (they claim to only use 1 additional rank)?
Then, the reported 7B sizes, in GB:
> 13.5 (fp16) 1.76 (HQQ 1-bit) 1.85 (HQQ+ 1-bit) 2.72 (quip# 2-bit)
those numbers would make sense if it was _actually_ 1 bit. But if you include the overhead of 16-bit scales (and why is the shift inlineable into lora? still unexplained) it'd be more like 3-bit.
From their HF page:
> This version offloads the meta-data to the CPU, so only the binary weights and the low-rank adapters are stored in the GPU memory.
Interesting, so we have to go back to CPU to rescale? Is this how they counted GB? This should have been clearly caveated in the table. I also am amazed they got latency lower than quip if they pingpong to CPU.
In fact, in Haskell, operating on an option conditionally has the exact same functor as a list: `map`.
So what am I to do, with an iterator? It's conflicting advice! An if is a for for an option.
The formula can be "oblivious" to the final size of the matrix too, which is helpful if you're doing some sparse ML training on edges (e.g., GNNs).
Say a paper maker is about to cut down a forest in the US. It buys up the rights to do so for $1M. Then, instead, it sells $1M worth of carbon credits to Amazon (maybe for a premium) for its data center greenwashing and doesn't end up cutting the US forest.
But then, secretly, or through some shell company machinery, it buys a plot of land in the Amazon and cuts trees down anyway to match its demand. Even though US law makers evaluate the counterfactual of the paper maker cutting trees down in the US, the net amount of trees in the world goes down.
It'd be really valuable to have "fuzzed" versions of these benchmarks, where you replace quantities in the questions with randomly-sampled values, so that this wasn't a concern. Of course, then the score would itself be a random variable, but you could just return an interval.
however i've found that the mixes introduce me to recommendations that I like at roughly the same rate as Discover Weekly.
What are you listening to for such good recommendations? Is the the automatic "Mix" playlists it creates?
But if you have some capacity constraint (e.g., memory, I guess?) then you can imagine dynamic rank allocation helping in the case where the maximum rank across all layers isn't within budget.
It's a bit of a stretch though, I agree