Sadly it didn't. It talked about "perplexities" and showed some floating point numbers.
I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
The dropdown at the top selects which comparison: Falcon compares GGML, Vicuna compares bits and bytes. I have some more comparisons planned, feel free to open an issue if you'd like to see something specific: https://github.com/the-crypt-keeper/can-ai-code
I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
The closer we get to assessing human like performance, the tougher it is, because it becomes more subjective and less deterministic by the nature of the task. I don't know the answer, but I know that for the metrics we have it's not so easy to translate them into any idea about the kind of performance on some specific thing you might want to do with the model.
Subjective ratings are different, but for compression things are quite well defined.
I think this right here is the answer to measuring and comparing model performance.
Instead of trying to compare models holistically, we should be comparing them for specific problem sets and use cases... the same as we compare humans against one another.
Using people as an example, a hiring manager doesn't compare 2 people holistically, they compare 2 people based on how well they're expected to perform a certain task or set of tasks.
We should be measuring and comparing models discriminately rather than holistically.
Would setting the temperature to zero degrade the quality of response?
For an llm, a trivial change in wording could produce a big change in answer, same as running it again with a new random seed. "Prompt engineering" is basically overfitting if not approached methodically. For example, it would be interesting to try deliberate permutations of an input that don't change the meaning and see how the answer changes as part of an evaluation.
Look at this minimal implementation (Karpathy's) of LLaMA, the only randomness is in the "sample" function that comes in at non-zero temperature, otherwise its easy to see everything is deterministic: https://github.com/karpathy/llama2.c/blob/master/run.c
Otoh, with MoE like GPT-4 has, it can still vary at zero temperature.
That seems simplistic, but its really simple as that. Naive 3 bit quantization will turn llama 7B into blubbering nonsense.
But llama.cpp quantization is good! I recommend checking out the graphs ikawrakow made for their K-quants implementation:
https://github.com/ggerganov/llama.cpp/pull/1684
Basically, the more you quantize with K-quant, the dumber the model gets. 2 bit llama 13B quant, for instance, is about as dumb as 7B F16, but the dropoff is not nearly as severe from 3-6 bits.
e.g. take a look at these two rows:
llama-65b.ggmlv3.q4_K_M.bin 4.90639 llama.cpp
llama-65b.ggmlv3.q3_K_M.bin 5.01299 llama.cpp
If you want more fine-grained explanations of how generation of certain types of texts get clobbered, you would probably need to prepare datasets comprised of that type of string, and measure the perplexity delta on that subset. i.e.
dperplexity/dquantization(typed_inputs).
dtype/dquantization(all_outputs).