> The number of model parameters is slightly higher in the BitLinear setting, as we both have 1.58-bit weights as well as the 16-bit shadow weights. However, this fact does not change the number of trainable/optimized parameters in practice.
https://arxiv.org/abs/1603.05279
I personally have a pretty negative opinion of the bitnet paper.
>we introduce BitLinear as a drop-in replacement of the the nn.Linear layer in order to train 1-bit weights from scratch
>While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work [ LSL+21 ], we maintain a latent weight in a high-precision format for the learnable parameters to accumulate the parameter updates. The latent weights are binarized on the fly during the forward pass and never used for the inference process.
https://arxiv.org/pdf/2310.11453
The other paper had a much nicer and clearer introduction to bitlinear than the original Microsoft paper, which is why I used it. Uncharitably you might say that they aren't hiding the lead 10 paragraphs in.