How to defeat the purpose of IEEE floating point (2008) | Hacker News Reader

How to defeat the purpose of IEEE floating point (2008)

yosefk.com

35 pointsluu29 comments

I have often found myself closer to the opposite end of the spectrum. Many machine learning algorithms and applications (certainly not all) can tolerate a fair degree of inaccuracy. This is often justified by the fact that the training data is typically noisy to begin with and is a random sample from the space it lives in. If your algorithm can be stable under those conditions it ought to be stable under floating point jitter.

This usually means you need regularization, or smoothing, or low pass filter, or averaging in some guise or the other (in some abstract way they are the same thing). There are some frequently used operations in ML that are very unstable under inaccuracies, numeric differentiation is one, just dont do it without smoothening. Inverting matrices is another, but you better have a real solid reason to ever invert a matrix, yes there are reasons when you do really want the inverse.

In any case the final point is that I have had to baby talk the compilers into, "yes, please please dont worry about it, just do these commutative (for real numbers) operations in any order that you want". This exposes more parallelism and other compiler optimizations. Often I would gladly take the hit over some precision in higher order decimal places to gain speed. But this should not be done blindly, and a smattering of numerical analysis helps gauging when it would be safe to do so. I doubt whether numerical analysis figures prominently in the trajectory of a graduating machine learner, although I think it ought to.

One of the advantages of languages with N-ary operators.

* (a,b,c) can take advantage of those tricks, whereas the programmer can still be explicit and go * (a, * (b, c)), for example.

Indeed. Do you have some specific examples in mind, I would love knowing more. The usual suspects, for example fold_left, fold_right would typically adhere to fixed order for floating point operations. A reason Java is a FLOPs killer is that its standard is too strictly specified, it does not leave much room for the runtime to take advantage of these features. Even order of argument evaluation for functions are specified to be left to right.

If you're a game developer, you should look at this Floating Point Determinism article, which shows some additional techniques for implementing deterministic physics with floating point.

http://gafferongames.com/networking-for-game-programmers/flo...

The quest for consistency seems misguided.

If your code changes behavior based on the timing of the various sections, you have a bug, a race condition. The solution isn't to lock down your code and build with only one configuration. The solution is to fix the bug.

In the same way, if the behavior of your code is dependent on the order of operations in floating point, it is very likely that you are doing something wrong. The solution is not careful ordering of floating point operations, but using algorithms appropriate to floating-point (and avoiding floating point when it isn't applicable.)

Code that has been ported to many platforms tends to be more stable, because the heterogeneous environments have exposed bugs and flawed assumptions in the code's construction.

Simple use case: a multiplayer game containing physics. I'd like the physics to be reasonably accurate, but more than anything it needs to be consistent, because I want to get the same results on both players' machines, and then the only thing I have to send over the slow internet is a command stream.

But if infinitesimal disturbances to the results makes a practical difference to the game, then your physics engine is not stable.

No physics engine is stable in that sense. Neither is the real world. You need full determinism to do what lmm is saying. Source: Worked on this exact thing for commercial multiplayer games.

Many interesting physical phenomena are deterministic but chaotic, in the sense that arbitrarily tiny changes in initial conditions can rapidly lead to arbitrarily large changes in behavior.

Would you say that anyone who tries to make a game that has the same property is "doing something wrong"?

You're probably imagining a short-lived game. Think of a game that goes on for months at a time. There are ways around this, but none of them as elegant as a floating point routine which produces the same results on any platform across any ordering of calculations.

Even for short-lived games, if you want to implement a replay system, you need your physics to be deterministic.

> But if infinitesimal disturbances to the results makes a practical difference to the game

They don't! The system can tolerate answers which are slightly wrong... as long as all machines calculate the SAME slightly-wrong answer.

The same logic applies to synchronized PRNGs. It doesn't (directly) matter what the "random" dice-roll result is, as long as all clients get the same answer.

Ah, thank you for this response. It "clicked" for me: Game developers are using the low bits (below significance) as pseudorandom numbers! They are overloading a new function on top of the existing use of floating-point numbers.

That's not what I'm saying. I'm just trying to give you an example where a "correct result" is far less-important than a "deterministic result".

It's a feature of physics, not physics engines. Many process in the real world are exponential in nature, so even a very small difference gets magnified very quickly.

I read the article so you don't have to! The author dreams of consistent behavior so he can get:

- Reproducible results across platforms

- Reproducible results with different build settings (such as debug builds or optimization levels)

- Useful test cases for regression testing

- Test cases that help him find misbehavior in the optimizer

- An improved chance of finding more subtle bugs, absent 'noise' from small changes in floating point operations

In particular, he gives examples of how changes to nearby code can change the output of floating point operations, due to hairiness in how registers are allocated. You can see how this would make testing quite painful.

These all seem like worthy goals that deserve a better response than "you're doing it wrong", along with some non-sequitur about race conditions.

Regarding race conditions, I was referring to this:

> It's not just floating point that's inconsistent across modes – it's code snippets with behavior undefined by the language, buggy dependence on timing, optimizer bugs, conditional compilation, etc.

There's quite a few algorithms that are generally vulnerable to precision issues because they can affect comparison order, such as k-means clustering. "Don't do that then" is not a useful answer.

Lots of problems dictate code that depend on the ordering of floating point operations. It's not something that you can design away.

Getting consistently worse results? :) He didn't even mention when matlab 5 started using his 64b only recommendation, the results were getting in-precise and matlab was getting unusable. http://www.cs.berkeley.edu/~wkahan/MxMulEps.pdf

I cannot take a floating point rant seriously without knowing Kahan's arguments. And a crazy requirement to compare it to slow and false debugging code does not really help.

This kind of thing makes me happy I work on the JVM. strictfp may be "cowardly" but it's relatively easy to use.

I understand that avoids some platform-capability differences, but does it also protect from issues with execution-order or ghostly-coupling between threads?

I have never heard of those problems, what are they? Hardware bugs?

In C terms strictfp just sets FLT_EVAL_METHOD to 0, which in today's SSE and 64-bit world most C compilers do anyway.

Maybe I am wrong, but is the author just looking for a decimal type? Those exist in most major languages, right?

The performance of decimal arithmetic on hardware not recently made by IBM means it cannot be used in most places.

The original post said he'd be happy to trade performance for consistency.

He could also use fdlibm ( http://netlib.org/fdlibm/index.html ), at least for anything other than primitive operations. But, again, it trades performance for consistency.

This week I learned that C99 and C11 have an FP_CONTRACT pragma, that can be set "off" for this reason ( http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf , section 6.5, paragraph 8).

Decimal types suffer from many of the same issues.

Sure, 0.1 can be represented exactly. But that's about the only issue with floating-point that that solves.

Reply on news.ycombinator.com