Please exclude my ignorance, but what would happen if network weights were modelled with arbitrary-precision numerics, such as a rational type, instead of IEEE floats?
(Other than drastically slowing things down, of course)
(Other than drastically slowing things down, of course)
- rounding errors in arithmetic between small and large numbers, large and large numbers, or small and small numbers
- unrepresentable numbers (eg 0.1)
- non-associativity of arithmetic making order of evaluation relevant
- residual connections can introduce, quite easily, arithmetic errors (heard this from research I can’t find atm): x + f(x) happens a lot in transformers, and the terms might be prone to cancellation or inadvertent rounding
- layernorm blocks introducing some error
- gradients might get added with error if too small or too big
On the bright side, some people have suggested that ieee float problems are usually not so terrible that you can see them instead as introducing slightly beneficial regularization