Very interesting, as I understand it was fine tuned on GSM8k, so we know we can remove 60% of the model and still answer GSM8k questions
Bigger question is, does the sparse model maintain any general knowledge?
Bigger question is, does the sparse model maintain any general knowledge?
The quantization part was interesting - they also quantized activations and have some adaptive quantization that accommodates outliers.