EDIT: Also, try converting JPEGs to uncompressed PNG using Python with Pillow or OpenCV, then try on C using libjpeg and the stb_image header library. All four create slightly different output. I discovered this evaluating the accuracy of a MobileNet implementation. The accuracy of the model changed based on what language/library converted the JPEGs.
Note that even between two versions of the same library, you can get different results.
Optimizations in FMACs can lead to LSB differences. IEEE doesn't specify how an FMAC is to be implemented, only the format of the number.
(Also: I'm 99% sure I'm right... still 1% could be wrong as I understand it.)
Quoth the spec:
> Each of the computational operations that return a numeric result specified by this standard shall be performed as if it first produced an intermediate result correct to infinite precision and with unbounded range, and then rounded that intermediate result, if necessary, to fit in the destination’s format (see 4 and 7).
Different standard libraries for different processors could have different precision for transcendental functions though. There are those that are written to always produce reproducible results though; and in the case of CRlibm, proven correctly rounded.
Older compilers for x86 used the x87 FPU which used a 80-bit type internally but all modern compilers for x86 and x86-64 use SSE/AVX instead which supports IEEE types.
What compiler settings? With -O3, GCC will flush-to-zero your denormals, because Intel CPUs have _SEVERE_ penalties on subnormal floats.
You don't even get the same results with -O vs -O3 on the same programming language on the same system... let alone different CPUs.
https://carlh.net/plugins/denormals.php
Denormals are part of IEEE 754, but in practice are more complicated for hardware to figure out. So as a common efficiency trick, you disable the processing of denormals entirely (IE: all denormals are seen as "0" to the hardware, speeding up processing severely).
On Intel processors, a configurable register can set up how your floating-point units act when they come across a denormal. So an inline-assembly statement can change your results, even if it was no where "close" to your code.
On the contrary. One of the most common multiplayer bugs is the multiplayer-desync due to different settings of floating-point arithmetic.
https://stackoverflow.com/questions/1948929/keeping-sync-in-...
This stuff happens in pretty much every video game. You "solve" it by...
1. In FPS games, most _clients_ serve as the source of truth. This means that in most FPS games, you allow the client to "hack" the location data for clipping, speed-hacks, and the like. (Server calculated you at point X. Client claims to be at point X+.05. Well, client wins the argument).
2. In games like Factorio, an extraordinary amount of effort is placed upon finding all sources of desync (https://wiki.factorio.com/Desynchronization) and squashing them. Protocols communicate with integers as much as possible, avoiding any floating point issues.
> but do they start becoming problems in weather forecasting or particle simulations? What kind of applications are impacted?
I'm no supercomputer expert. But for reliability, a lot of supercomputer simulations step through twice to "verify" the results. That means both simulations must be setup with the same settings, so that your floats all line up.
In my undergraduate studies, my professor for numeric computing stressed the importance of sorting numbers: not only to avoid floating-point cancellation error, but also to have a precisely defined ordering for supercomputer applications.
You want to do things like add / subtract numbers from smallest to biggest, and all the positive numbers add together, then all the negative numbers add together and only do one subtraction. Cancellation error is serious business!!
In such cases, the result of one simulation is not enough, and we do several ones with slightly different initial conditions, just to make sure that we don’t spend time analysing an outlier. Exact reproducibility is important to validate the codes running test suites, but for the purpose of using simulations to make predictions, it does not really matter as long as each simulation is physically accurate enough.
To enable that, you need to use -Ofast or -ffast-math, there's a reason they're not default enabled after all.
For a project relying on multiple runs on different computers for verification of correctness, this was a serious issue. IIRC before they solved it properly, they had to increase the number of validations per piece of work, as well as trying to segregate the work so a work unit was only issued to either Intel or AMD but not both. This only reduced the impact somewhat though.
From their[1] page:
Besides helping build the LHC, LHC@home has proved to be an invaluable, even essential, tool for investigating and solving fundamental problems with floating-point arithmetic. Different processors, even of the same brand, all conforming more or less to the IEE754 standard, produce different results, a problem further compounded by the different libraries provided for Linux and Windows. In particular, a major problem was the evaluation of the elementary functions, such as exponent and logarithm, which are not covered by the standard at all (apart from the square root function). This problem was solved by the use of the correctly rounded elementary function library (crlibm) developed by a group at the Ecole Nationale Supérieure in Lyon. Due to the often chaotic motion of the particles even a one unit of difference in the last digit of a binary number produces completely different results, as the difference grows exponentially as the particle circulates for up to one million turns round the LHC. Thus it is extremely difficult to find bugs or to verify new software and hardware platforms. This methodology has allowed us to detect failing hardware, otherwise unnoticed, and to find compiler bugs. With your help, it is being extended to ensure identical results on any IEE754 compliant processor with a standard compliant compiler (C++ or FORTRAN) which will allow us to work on any such system and even on Graphical Processing Units (GPUs).
[1]: https://lhcathome.web.cern.ch/projects/sixtrack/sixtrack-and... (