John Gustafson’s crusade to replace floating point with something better

Also, IEEE 754 floating point standard guarantees the results of addition, subtraction, multiplication, division, and square root to be the exact correctly rounded value, ie a deterministic result, contrary to what he says.

What about this then, which somebody comments below:

"On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time, even though the machine did things 80-bits at a time)."

And below:

"I spent a long time once debugging an issue arising from this. Code was basically:

  double x = y;
  [...]
  if (x>y) fail;

This behavior might not be IEEE defined, but even as IEEE undefined (and implementation dependent), it still qualifies those IEEE compliant implementations as non-deterministic.

Or how about:

"parallel systems, most likely GPUs, where it is common to have slightly different results even on the same operation, on the same system. This is due to the fact that several pipelines will do calculations in parallel and depending on which pipeline ends first (which can depend on many factors like current temperature) the sums can happen in a different order, leading to different rounding results."

due to fp associativity?

How can that be? The IEEE standard does not place many requirements on the precision of the transcendental functions, and AFAICT a transcendental doesn't even need to return the exact same answer twice. That is, it is correct for a compiler to, if it can, evaluate a transcendental at compile-time with infinite precision, and if it cannot, call a run-time function that computes it. Math libraries can be dynamically linked, and all can produce different results here, so it is not that a compiler can go out of its way to make things deterministic either.

For instance, the claim that "posits can be a direct replacement of floating point" is not true. There are a few tricks, such the Fast Invertion Square Root that abuses the floating point format to do fast arithmetic operation. This would break replacing with posit.

Also he claims that early FPGA posit implementation uses less area and is has high performance than a Floating Point implementation. This cannot be true^. The posit unit IS A floating point unit with a encoder/decoder unit. The posit floating point unit will actually have a larger word size when decoded than ieee floating point because it has more precision than ieee format. Also the features such different rounding supported by ieee but not by posit uses a negligible extra area.

^ Unless, ofc, he is comparing apples to oranges.

Everyone who reads that who is cognitively typical will understand "floating-point" (which, by the way, needs a dash, if we are going to nitpick) as a nickname for "IEEE 754 floating-point" or "traditional floating-point", rather than an insinuation that posits are, in contrast, fixed-point or something other than floating-point.

Posits are Type III Unums IIRC, the original proposal was with Type I Unums.

Previous proposals were variable length formats, which is pretty much a nonstarter for a variety of reasons. This proposal (posits) is a fixed length format with variable length fields.

Note that fixed radix point does not solve the common issues with representing rational base 10 fractions. A base10 fixed radix solution would, but so would IEEE754's decimal64 spec, which would eliminate representation error when working exclusively in the context of base10 e.g finance, but these are not found in common hardware and do not help reduce propagation of error due to compounding with limited precision in any base.

A 64-bit integer is big enough to express the US National Debt in Argentine Pesos.

If you don't care about performance, then the actual solution has no dependency on hardware:

1. Replace the default format for numbers with a decimal point in suitably high level languages with a infinite precision format.

2. Teach people using other languages about floating point and how they may want to use integers instead.

The end. No multi-generation hardware transition required.

IMO, IEEE 754 is an exceptionally good format. It has real problems, but they aren't widely known to people unfamiliar with floats (e.g. 1.0 + 2.0 != 3.0 isn't one of them).

The numerator is just an integer and integers are just integers and the base doesn't matter. But if the exponent is base 2, then you can have 1/2, 1/4, 1/8 on the base but not 1/5 or 1/10.

They have the survey online at [1] in case you want to see how much you know about fp behavior.

[1] http://presciencelab.org/float

However, as soon as you start doing anything interesting, you have limited precision as a matter of course.

Sadly most languages don't support something like that out of the box.

Where would one go to better understand how floating points are represented?

> I shudder to think how many e-commerce sites use `float` for financial transactions!

The float IEEE-754 represent up to 9 decimal digits (or 23 binary digits) with precision. The double, represent 17 decimal digits. The error upper bound is (0.00000000000000001)/2 per operation. Likely irrelevant for most e-commerce.

Also, the database stores in currency values using fixed point.

> Intel chips have had BCD support in machine code

BCD is floating point encoding not fixed point. AFAIK, only Intel supports it and very precariously.

> I'm surprised nobody has made a decent fixed point lib that is widely used already.

Nonsense. If you do any scientific computation you have likely have Boost, GMP, MPFR installed in your system. They support arbitrary precision arithmetic with integer (aka fixed point), rational and floating point.

It's a bit sad that many of the creator’s claims are exaggerated and his examples cherry-picked though.

The quire stuff has been thrown in there, too, because while we're boiling the ocean, we might as well encourage people to "do the right thing". think of the quire as a really large accumulator cache for things like matrix vector multiplication or tensor outer products, which I hear, AI uses a lot.

Your boss, on the other hand, may try to armchair architect you, which is a real (and underreported) problem,

Yeah, but posits appear to have fewer weird edge-cases. Plus the people trying out posits are constantly trying to find methods to work around the limitations too.

During this years conference on posit maths Florent de Dinechin had a really nice talk bringing up all current issues with posits and ways that floating point maths had found workarounds over the year, as a kind of challenge to make posits catch up[0][1]. The community took it really well, as far as I can see, and Gustafson in particular seemed delighted because he genuinely wants everyone to start using better numerical methods.

[0] https://www.youtube.com/channel/UCOstJ2IVC4Y8mbgN0IsowKw

[1] https://www.youtube.com/watch?v=tcX2nRCdZvs

Long story short: es=log2(nbits) -3

A rounding error is when you have a number, lets say 0.75, and due to rounding it is recorded as 1.00. The "rounding error" is 0.25.

An alternative to rounding to 1.00 would be to have a mechanism which says "the value is between 0.50 and 1.50". This way, there is no actual rounding, as it doesn't commit to a rounded value, so there is technically no "rounding error".

A neat advantage of recording an interval rather than rounding is that the "error" is preserved in the data through arithmetic, so if there is some following code that runs x * 100, a rounding mechanism would say "the value is 100" whereas an interval mechanism would say "the value is between 50 and 150". Then, if the user only looks at the output, it will be clear that there is a wide error range and something needs to be fixed, rather than the output indicating a precise answer when really it suffers from significant rounding errors.

> Gustafson said that for training, 32-bit floating point is overkill and in some cases doesn’t even perform as well as the smaller 16-bit posits

If posits have a bit number, then they are not of variable accuracy. This is simply sloppy explaining.

At some point, rounding has to happen (call it an interval if you want, but you're not being helpful). As a programmer, I WANT rounding to happen. I depend on it. You don't break out the floats unless you are prepared for some information to be lost. If you try to never lose any information, then you wind up with a monstrosity like Maple and Mathematica in which all steps need to be curated by hand to occasionally reduce the giant glob of fractions and implicit solves into something concrete, but imperfect.

I read the article and took a glance at https://www.johndcook.com/blog/2018/04/11/anatomy-of-a-posit..., but I'm still completely unable to answer this question of how accuracy is shed.

Sloppy journalism.

This can represent the exact sum of any number of floating point values (or products of floating point values) with the intermediate operations preserved at infinite precision, up to a single rounding at the end.

e.g., for computing an inner product of vectors \sum_i x_i y_i with x_i and y_i in floating point, a fused multiply-add operation (as on today's computers) would perform something like:

r(x_n y_n + r(... + r(x_2 y_2 + r(x_1 y_1 + 0))...))

where what is performed within the rounding function r() is done to infinite precision and a single rounding at the end. This is also only true if you are accumulating into the same value, and not splitting this operation across multiple values, otherwise there would be additional rounding steps performed.

Using a Kulisch accumulator the result would be:

r(\sum_i x_i y_i)

similarly also only preserving precision if everything is done in a Kulisch accumulator.

You can add a Kulisch accumulator to IEEE floating point or any other floating point as well. In fact the second? ever computer with floating point (the Zuse Z3) performed FP addition using a similar accumulator, but sums-of-products probably didn't preserve the multiplied value though.

I haven't had time to digest it all, but he is certainly critical of Gustafson's proposals. Not sure though the articles linked above cover the latest and greatest Unum III. At any rate, I'd pay close attention to Kahan's critique.

1. Yes -- On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time, even though the machine did things 80-bits at a time).

2. No -- Modern x86 systems use SSE registers, which are 64-bits. There are a whole slew of configuration options, but if Windows / Linux does their job, you should have the same rounding-errors across your program (unless you manually change the rounding options, but that's your own fault if you go that route).

3. Kinda yes -- Floating point operations are NON-associative. (A+B)+C does NOT equal A+(B+C). The best example is Python:

>>> (1+1)+2.053 9007199254740994.0 >>> 1+(1+2.053) 9007199254740992.0

253 is the first value which starts to "round off" 1.0 in double-precision. So (1+2.053) == 2.053, because the value was rounded off.

Due to #3, even if you did everything correctly (ie: set the rounding flags so they were consistent), if your arithmetic happened in slightly different orders (very common in network-games, where different players may have their updates in slightly different orders), you end up with a 1-bit error between clients... which completely borks your simulation.

Because #3 is nonintuitive, many people think that you get different results when using floats. But its simply due to non-associativity.

Wouldn't the equivalent pattern just be to start at +/- typemax(Posit) or whatever? As you would for integers, which lack infinity.

For example, the posit16 with nbits=16 and es=1 encodes the exponent like:

01.0 = 0 01.1 = 1 001.0 = 2 001.1 = 3 0001.0 = 4

The format has a normal exponent field with es bits that encodes exponent in binary. When it overflows, it encodes the carry as unnary format. In the unnary format the run-lenght of the same number encodes a number. For example: 0001 would be 3, 001 would be 2, etc. Of course, the posit format is a bit bore complicate than that (it supports negative exponent, for example). But the ideia is pretty much this.

Because of this encoding, if you are working with numbers that have small magnitude posit will have a LOT of more precision than your floating point format.

But the claim that posit16 can have as much precision as binary64 from ieee-754 is misleading. The posit16 can have up to 16-1-2-1=12 bits of precision. While binnary64 always has 53 bits of precision.

They likely compared the binnary64 with posit16 using the accumulator (aka quire). I'm not sure how the quire would map to real world FPU, it uses a lot os space.

I would say that this is certainly my favorite feature of IEEE floats :)

This being said, I object to your premise: transmission and memory storage are getting comparatively more costly vs computation. From "Computer Architecture, A Quantitative Approach", 6th edition in figure 1.13 there are some values for the TSMC 45nm process, so hardly the leading edge and it only gets worse with finer nodes: - 32 bits integer multiplication: 3.2 pJ - 32 bits float multiplication: 3.7 pJ - 32 bits read in a small 8 kB SRAM: 5 pJ - external DRAM 32 bits read: 640 pJ

We have a lot of transistor nowadays, and as long as the computation can be pipelined halving the memory accesses could be a win (TBC).

Also on from the same Inria team as [1], and also covering hardware cost but more than this: "Posits: the good, the bad and the ugly": https://hal.inria.fr/hal-01959581v3/document

[1] "Hardware cost evaluation of the posit number system", https://hal.inria.fr/hal-02131982/document

Memory bandwidth relative to instruction to instruction latency has gone way down. Memory is slower now relative to the amount of math one can do.

We are also swimming in extra silicon, so even if what you are saying is true (2x the number of gates, all adding to the delay), posits would still be a win.

You are arguing from an arbitrary what-if position, it doesn't look good.

edit: One of several examples found in the "Posits: the good, the bad and the ugly" paper linked in the thread : 10.0 * 2.0 = 16.0 in posit8

You can solve it by switching to decimal floating points, they are defined by IEEE as well:

https://en.wikipedia.org/wiki/Decimal_floating_point

Yes, having redundant bitpatterns is not great when designing a number format, however, even for Float16 (half-precision), making use of the 3% NaNs is wise, but not going to be a gamechanger. Some others discussed pro/con for neg zero and also neg infinity: In my view you want to have a bit pattern that tells you that the answer you get is not real, but whether it's +/- Inf or some NaN is pretty much irrelevant. Using these bit patterns for something else sounds like a very reasonable approach to me. Furthermore, I've never come across a good reason for -0 in our applications.

When it comes to weather and climate models in HPC, I see the following potential for posits: Similar as BFloat16 is supported on TPUs, I could see Posit16 to be supported by some specialised hardware like GPUs, FPGAs etc. I'm saying that because for us it's not important to have a whole operating system running in posits (although I probably wouldn't mind) but to have them for some performance critical algorithms. Unfortunately, weather and climate models are far more complex than some dot products and we usually have to deal with a whole zoo of algorithms causing weather and climate models to cover easily several million lines of code. Now let's say we know our model spends 20% of the time in algorithm A which only requires a certain (low) precision to be stable and to yield reasonable results, then it would be indeed a big game changer if we could run this algorithms in, say, 16bit. In exchange of precision for speed we would probably want to push things to the edge, i.e. if we can just about do it in 16bit, then we should. Now there are several 16bit formats: Float16, BFloat16, Posit16, Posit16_2 (with 2 exp bits), and technically also Int16. Let's forget about the technical details of these formats and let's focus on where they actually considerably differ: What is the dynamic range and where on the real axis do I get how much precision to represent numbers. Yes, from a computer science perspective also the technical details matter, but from our perspective most of it is pretty irrelevant and what actual matters are these two things: dynamic range and where is the precision. Because these two really determine whether your algorithm is gonna crash or whether you can use it operationally on your desktop computer or in a big fat $$$ supercomputer.

For PDE-type problems (that includes CFD and also weather and climate models) I came within the last year of my research to the following preliminary conclusions regarding dynamic range and precision with respect to the above mentioned formats:

Int16: Let's forget about it. Float16: The precision is okay, but rarely needed towards the edges of the dynamic range. Floatmin might work, however, floatmax with 65504.0 is easily a killer. Might work with a no-overflow rounding mode and smart rewriting of algorithms to avoid large numbers. BFloat16: For our applications having only 7 significant bits is not enough, I didn't come across a single sophisticated algorithm that works with BFloat16. Posit16 (with 1 exp bit): Great, puts a lot of precision where it's needed but also allows for a reasonable dynamic range. Posit16 (with 2 exp bits): Probably even better, the sacrifice of a bit precision in the middle is fine and the wide dynamic range gives it the potential to also work with algorithms that are hard to squeeze into a smaller dynamic range.

In short, posits actually fit much better the numbers our algorithms produce. And this can indeed be the game changer: If a GPU supports posit arithmetic and we can run algorithm A on it in 16bit: Wonderful, contract sold! But if we couldn't with BFloat16 or Float16 than there is no future for 16bit in our field.

I explain more about this in this paper: dx.doi.org/10.1145/3316279.3316281

And there are two talks which tell a similar story: https://www.youtube.com/watch?v=XazIx0cMVyg https://www.youtube.com/watch?v=wp7AYMWlPLw

or simply drop me an email if you have questions (unlikely respond here) that you find on my website: milank.de