Are there flaws in some ARM64 instructions?
eclecticlight.co
eclecticlight.co
The basic operation here is
x = (((a * b) + c) - c) / b
Let's denote by fp(y) the floating-point number closest to y. If you don't use FMA, then the middle of the formula is essentially fp(fp(M + c) - c) for some M. When you pick the natural choice of initial a, b and c having the same order of magnitude, this gives exactly M in 99.99% of cases. Then you're just measuring the difference between a and (a * b) / b. Instead, FMA gives you x = fp(fp(fp(a * b + c) - c) / b), which is admittedly more complex to study.
But as you pick larger and larger c values, you can observe that FMA becomes more accurate compared to the other version. When c reaches about 0.5e+6 times larger than a and b, FMA starts winning. Note that a is incremented 1M times as the proposed algorithm iterates, so this corresponds to (a * b) and c having the same orders of magnitude on average, as the loop progresses.
You can probably go much more in-depth in studying this, and explain it better. But essentially this is a non-issue, just floating-point being floating-point.
If you want to play with the code, you can find it here:
Depending on the case, the accuracy gains can be tiny (so much so that in this example the FMA version does indeed return larger errors in most runs), and the performance gains can be small too.
On the other hand, the price to pay for letting the compiler freely decide whether/when to fuse * and + is huge: reproducibility. Since x87 was deprecated, having floating-point results be bit-for-bit consistent across platforms, compilers and optimization flags has been amazing for debugging. Losing that is a step back and a recipe for headaches.
I was surprised that gcc enables -fexcess-precision=fast by default (in GNU mode), hence enabling transparent FMA and potential headaches. Specifying the standard (e.g. -stdc=c17) fixes that.
It should be noted that 1e+6 sticks out as right on the rounding limit of single precision floating point numbers, which is about 1e+7. Thus it suggests that the residue amplification from b ceases to be a problem once b simply falls out of the picture as it gets close to a roundoff error.
Therefore, this test isn't evaluating FMA as much as it is showcasing how floating point division amplifies numerical residues.
The number line for floats is non-linear. It has holes in it.
To do this correctly requires:
* performing the calculation with a higher degree of accuracy (and precision).
* comparing the two approaches against the reference results.
To do this you should look into either writing a simulation of ieee with a larger mantissa or using an arbitrary precision library.
Which result is more precise depends on the input data (I randomized it), but it seems that most commonly, there was no actual difference using FMA and not using it, and when there was a difference, it was even odds which one was the more precise result.
There seems to have been a similar issue in Chromium a few years ago.
https://bugs.chromium.org/p/chromium/issues/detail?id=401117
Where the conclusion was:
"Having said that, fmadd is doing exactly what it should do; it isn't broken. The model that gave you a result of 0x4 _is_ broken, but the hardware isn't."
Next up though is whether fused multiply add is part of the IEEE floating point standards, and whether the arm implementation meets these. Chances are it does (I see references to it from 2008). If the x64 FMADD produces difference results, it's likely that either the standard is wooly in some respects and allows wiggle room (so they both meet the standard), or, the Intel/AMD implementation pre-dates the standard and isn't compliant.
I'm sure someone who knows these standards and processors better than me will be along in a minute to comment.
I'm just guessing here tho.
> The operation fusedMultiplyAdd(x, y, z) computes (x × y) + z as if with unbounded range and precision, rounding only once to the destination format. [...] fusedMultiplyAdd differs from a multiplication operation followed by an addition operation.
[elided exception discussion]
I expect that the ARM processor precisely implements this standard, as does x86-64. Any "bugs" here are the user misunderstanding how the precision is supposed to work.
The result of (IEEE 754) operations -- including PLUS, MINUS and FMA -- is specified bit-for-bit, at least for `isnormal()` numbers. It is not a matter of vaguely "more precise" or "less precise".
The test performed brings no information towards answering the question, because the loss of precision precision in one particular complex algorithm can perfectly well happen despite one individual operation having better worst-case precision.
To check the unlikely event of a flaw in M1's FMA, one would need to compare the output of invidual FMAs with MPFR's output on the same operands.
> ARM64 v Intel: I haven’t attempted to look at Intel processor fused instructions
Instead, this test could have been a clue. FP64 results follow the spec and hence must be bit-for-bit equal across platforms (as long as one avoids legacy x87 on Intel).
Without such a smoking gun, I would guess the explanation for this is that “subtract c, then divide by b” is a better inverse for “multiply by b, then add c” than for the fused multiply-add.
GLSL and the like are different.
Doesn't seem o be the case: MSVC avoids FMA, while GCC prefers it: https://godbolt.org/z/e9EcfjdeY
gcc and icc will use fma under `-O3`, but Clang won't.
"If you use different tools and want to ensure best results from floating point arithmetic on ARM64, you may wish to check that code generation doesn’t use fused instructions, particularly on large loops which could accumulate significant errors. It’s worth bearing in mind that authoritative texts on floating-point arithmetic are also extremely cautious about the use of such fused instructions."
I do hope that the original poster updates the post to reflect some definitive and validated conclusions. (And also not rely on Betteridge's Law for the headline!)
The article makes no mention if the fused instruction is producing incorrect results or just different results. That is the critical question.
Is the error the same in non-Apple targets? Does replacing FMADD with FNMADD (negative mult) or FNSUB change the magnitude of error in this case? Also, would the relative error be the same if the numbers were random integers or random floats of a given range (-1 to 1; -0.01 to 0.01; -100 to 100; 0 to 1) rather than integers in sequence?
-----
In most robotics and AI applications, calculations are unlikely to suffer greatly from inaccuracy but benefit from speed. What are the applications where someone would require floating point math AND both extreme speed and extreme accuracy?
Scientific computation.
I'm not even sure, it is a trade-off between (computational) speed and accuracy.
In my experience, error-analysis was a mandatory course for physics, but not in robotics. The error analysis was manual and time-consuming.
In robotics, no one cared why it occasionally produced NaNs, etc.. people just added a small number here and there.
That's faster to program.
To put it more positively, I think, the view here is more having an application, which is robust in the face of errors / edge cases. What counts is the output (action).
While in science the correct model is part of the output. Edge cases are an important part of it.
FMA instructions are more accurate and faster compared to a sequence of separate multiply and add/sub instructions, not less. The article literally talks about it in the beginning. The issue is reproducibility, which with floats can be quite bad even without FMA.
Also both robotics and neural networks (I dislike the tendency to call it "AI") can be quite susceptible to errors. Gradient-based optimization algorithms can suffer noticeably from calculation errors. This is why you usually want to train ANNs using 64-bit floats.
Do you find this genuinely important? If not, have you had any success encouraging e.g. validating against golden data in a non-bitwise comparison?
Sometimes this is understood by the banks and the solution is to implement a tolerance in the comparisons. In other cases other solutions may have to be used, such as forcing certain reports to be computed on certain software/hardware.
You remind me of the alleged UK Post Office scandal:
A difference in the 15'th decimal has no impact on the risk number.
The truth is, I have faith in Murphy's law.