An Interview with the Old Man of Floating-Point (1998) | Hacker News Reader

An Interview with the Old Man of Floating-Point (1998)

eecs.berkeley.edu

86 pointsHachiro21 comments

Open articleView on HN

For those looking for more details visit dr Kahan home page at http://www.cs.berkeley.edu/~wkahan/

For example: http://www.cs.berkeley.edu/~wkahan/ARITH_17U.pdf explains what gradual underflow is.

And for Excel haters: http://www.cs.berkeley.edu/~wkahan/Mind1ess.pdf :-)

Don't forget the floating-point walkthrough from the ever-lovable Ridiculous Fish:

http://ridiculousfish.com/blog/posts/float.html

This is always a great read. I think most programmers fail to realize what a hellish nightmare floating-point arithmetic was and still could be if not for the amazing work of Kahan and his colleagues and a lot of luck that allowed this standardization process to be one of the most successful ones ever.

I have heard programmers complain about the hellish nightmare of IEEE754 floating-point arithmetic. I'd heard it was a compromise between reliability and implementability, but I hadn't know until know how far it pushed towards reliability, or how much more of a hellish nightmare pre-754 floating point must have been.

754 arithmetic is not really hellish; it's quite easy to use correctly if one can be bothered to learn how it works. The trouble that most programmers run into is that they can't be bothered, and assume that it works just like arithmetic over the reals (which they typically don't really understand either).

This part is interesting:

> "If better precision, directed roundings, and exception handling capabilities latent in hardware but linguistically inaccessible remain unused, their mathematically provable necessity will not save them from atrophy. The new C9X proposal before ANSI X3J11 is a fragile attempt to insinuate their support into the C and C++ language standards. It deserves the informed consideration of the programmers it tries to serve, not indifference."

And it's now 2013 and directed roundings and exception handling largely remain linguistically inaccessible. C99 did add them, but they remain fragile. And most other languages haven't added them. It seems that the programming community has largely settled on indifference.

Directed rounding as a thread-local state (or global state), which is what the first 25 years of IEEE-754 processors supported, turns out to be not a very useful model (except for debugging tricks like estimating stability, which is dear to Kahan); that's a big part of why directed rounding isn't widely used or supported.

Several newer hardware designs have added per-instruction stateless directed rounding (the rounding direction is part of the opcode); this is a much more useful model (and its presence makes dynamic rounding modes more usable as well!). This feature was added to ieee-754 in the 2008 revision, too late for language bindings to be added to c[++]11, but there's already language support in OpenCL and other compute languages, and bindings have been proposed to the C committee.

So, progress is being made, slowly but surely.

For what it's worth, Julia supports setting rounding modes in a way that works across both native floating-point types (Float32 and Float64) and BigFloats:

http://docs.julialang.org/en/latest/stdlib/base/#Base.get_ro...

This stuff is incredibly important for numerical work and it's really a shame that mainstream languages have effectively rejected this key piece of functionality – that the hardware supports.

There's an open GitHub umbrella issue tracking support for all of the pieces of functionality that Kahan advocates:

https://github.com/JuliaLang/julia/issues/2976

I'm not sure about signaling NaNs though. There have been rumblings that future hardware might not support them.

This is sadly true. If you want to do reliable floating-point computations beyond +, * and / with round-to-nearest (and even that isn't easy, since for example the use of x87 instructions can result in the wrong precision), the only portable solution is still to use a software implementation such as MPFR. Which is a shame because you're letting all that fast silicon go to waste.

Even if all the basic operations were fully specified and implemented identically on all architectures, you would still have a noticeable performance vs. reproducibility tradeoff across architectures for "real computation". For example, different architectures have different numbers of registers and different cache hierarchies, which favors different access patterns in a fast matrix multiplication; thus linear algebra routines that attempt to be as fast as possible will necessarily group the arithmetic operations differently, which leads to different results even if individual arithmetic operations are guaranteed to be identical.

It's not insurmountable, but it's a thornier problem than it looks at first.

I am familiar with Kahan from his excellent low-error floating point summation algorithm:

http://en.wikipedia.org/wiki/Kahan_summation_algorithm

Compensated summation (aka Kahan summation) is a great algorithm. Interestingly, pairwise summation [1] is almost as accurate as the compensated algorithm, while being nearly as fast as naïve linear scanning summation. This makes it a great default summation algorithm since it does so well in terms of both accuracy and performance.

[1] https://en.wikipedia.org/wiki/Pairwise_summation

I had missed this in my numerics class many years ago.

Could have saved me a lot of pain.

Almost every year, I reread the famous Goldberg paper, What Every Computer Scientist Should Know About Floating Point Arithmetic[1]. And with every time I'm more astonished at how the hell can anyone do anything nontrivial with floats and trust it.

[1] http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.ht...

This is the reason that support for changing rounding modes is so incredibly important for numerical reliability. One of the best ways to assess if your algorithm is numerically stable is to try it in all the rounding modes. If all of the answers are similar, you're probably doing ok. If they're wildly different, you may be computing complete nonsense.

Remember the bad old days of IBM big iron base sixteen floating-point, which could gratuitously cost you up to three significant bits? Thank goodness those are gone.

And to this day, most FPUs I've worked with don't support gradual underflow in hardware. They can only get 754 compliance with OS assistance.

What do you mean? Intel/AMD FPUs suport it in microcode, but it is somewhat slow. But the OS is not involved.

Heck, your cellphone processor supports gradual underflow (though it's likely turned off in the default fp state).

Good thing I hardly ever work with x86 (Power and ARM primarily)

ARM VFP supports gradual underflow. NEON also supports it on arm64. Most Power implementations that aren't Cell BE do as well.

Reply on news.ycombinator.com