More generally than "unlearning", I wonder if taking any fp16 model and running it in fp32 or fp64 does anything positive to it? e.g. exposes knowledge that isn't accessible at the lower precision
Correct me if I'm wrong, but isn't there no effect on a floating point operation if you make the numbers more precise?
I don't think that's always correct when you're talking about operators in neural nets. E.g. the sin and cos in rope embeddings would get more precise, large sums like softmax would become more precise, potentially attention too due to dot products
If the model wasn't optimized with a parameter = 3.08 (that is, the optimization figured 3.08 increased the objective function), why would it be better than parameter = 3.1 ?