Floating point visually explained (2017)
fabiensanglard.net
fabiensanglard.net
One thing I dislike in discussions about floats is this incessant focus on the binary representation. The representation is NOT the essence of the number. Sure it matters if you are a hardware guy, or need to work with serialized floats, or some NaN-boxing trickery, but you can get a perfectly good understanding of binary floats by playing around with the key parameters of a floating point number system:
- precision = how many bits you have available
- exponent range = lower/upper bounds for exponent
- radix = 2 for binary floats
Consider listing out all possible floats given precision=3, exponents from -1 to 1, radix=2. See what happens when you have a real number that needs more than 3 bits of precision. What is the maximum rounding error using different rounding strategies? Then move on to subnormals and see how that adds a can of worms to underflow scenarios that you don't see in digital integer arithmetic. For anyone interested in a short book covering all this, I would recommend "Numerical Computing with IEEE Floating Point Arithmetic" by Overton [1].
The binary form handily demonstrates the limitations. Understanding the floating point instructions is kinda optional but still valuable.
Otherwise everyone should just use varint-encoded arbitrary precision numbers.
[a]: Sure, if you `console.log(0.1)`, you'll get 0.1, but it's not possible to express it in binary exactly; only after rounding. 0.5, however, is exactly representable.
Python 3.9.5
>>> 0.1.hex()
'0x1.999999999999ap-4'
>>> 0.2.hex()
'0x1.999999999999ap-3'
>>> (0.1 + 0.2).hex()
'0x1.3333333333334p-2'
>>> 0.3.hex()
'0x1.3333333333333p-2'Because you don't have all the digits available just for the mantissa? That seems quite intuitive to me, even if you don't know about the corner cases of FP. This isn't one of them.
I guess -- the one thing that explicitly looking at the bits does bring to the table, is the understanding that (of course) the number of bits or digits in the mantissa must be less than the number of bits or digits in an equivalent length integer. Of course, this is pretty obvious if you think about the fact that they are the same size in memory, but if we're talking about sizes in memory, then I guess we're talking about bits implicitly, so may as well make it explicit.
* actually, much more than just getting a general flavor, most of the time in numerical linear algebra stuff the actual bitwise representation is usually irrelevant, so you can get pretty far without thinking about bits.
And so far the explanation in the article is the best I found, not least because subnormal numbers appear naturally.
There is a mathematical foundation behind it of course, but it is not easy for a programmer like me. I think it is better to think in term of bits and the integers they make, because that's what the computer sees. And going this way, you get NaN-boxing and serialization as a bonus.
Now, I tend to be most comfortable with a "machine first", bottom-up, low level approach to problems. Mathematical and architectural concepts are fine and all, but unless I have some idea about how it looks like in memory and the kind of instructions being run, I tend to feel lost. Some people may be more comfortable with high level reasoning, we don't all have the same approach, that's what I call real diversity and it is a good thing.
Might you or someone else elaborate on what "NaN-boxing trickery" is and why such "trickery" is needed?
For example, if you are designing a scripting language interpreter, you can make every value a floating point number and use some of the NaNs to represent pointers, booleans, etc.
See also http://wingolog.org/archives/2011/05/18/value-representation...
For integers you can write 0x0B, 11, 0b1011 and have a very precise representation
For floats you write 1e-1 or 0.1 and you get an ugly truncation. If it were possible to write something like 1eb-1 (for 0.5) and 1eb-2 (for 0.25)... people would be incentivate to use nice negative power of 2 floats, which are much less error prone than ugly base conversions.
This way you can overcame the fears around floats being nonexact and start writing more accurate tests (bit per bit) in many cases
There are floating point hex literals. These can be written as 0x1p-1 == 0.5 and 0x1p-2 == 0.25.
You can use them in C/C++, Java, Julia, Swift, ..., but they are not supported everywhere.
the mantissa is written in base 16
the exponent is written in base 10
the exponent itself is a power of 2 (not of 16 or 2), so that's base 2
One can only wonder how that came to be. I think they chose base 10 for the exponent to allow using the 'f' suffix to denote float (as opposed to double)
> ISO C99 and ISO C++17 support floating-point numbers written not only in the usual decimal notation, such as 1.55e1, but also numbers such as 0x1.fp3 written in hexadecimal format. [...] The exponent is a decimal number that indicates the power of 2 by which the significant part is multiplied. Thus ‘0x1.f’ is 1 15/16, ‘p3’ multiplies it by 8, and the value of 0x1.fp3 is the same as 1.55e1.
The initial precision doesn’t really matter, because if you plan to use this value in a computation, it will quickly accumulate an error, which you have to deal with anyway. There are three ways to deal with it: 1) ignore it, 2) account for it, 3) use numeric methods which retain it in a decent range. You may accidentally (1)==(3), but the problem doesn’t go away in general.
(-1)^S * 1.M * 2^(E - 127)
(-1)^S * (1 + M) * 2^(E - 127)
(-1)^S * 2^log_2(1 + M) * 2^(E - 127)
let F := log_2(1 + M)
0 < F < 1
Note: [0 < M < 1] implies [1 < M + 1 < 2] implies [0 < log_2(1+M) < 1]
(-1)^S * 2^(E - 127 + F)
It's certainly much clearer than the windowing and bucketing in this article.
The windowing explanation shows me a path from the intent of the designer through to the implementation (the algorithm). Now I can retain that knowledge.
I still fail to understand what is arbitrary in scientific notation. The designers almost certainly didn't think in terms of windows when creating the data types, they probably thought about the mantissa and exponent the way it's usually explained, and maybe about information density (entropy per bit) when comparing with other possibilities.
Anyway, if it helps, great. Different explanations are always welcome. But this one is post-fact.
I used the word arbitrary, but did not say that scientific notation was arbitrary.
I will try to explain another way. The equation is an assertion. For me, it does not plainly follow from the underlying problem. Once the author has introduced the windowing approach, the equation follows easily for me.
> almost certainly [..] they probably [..] Different explanations are always welcome. But this one is post-fact.
By your own words, it is not clear that it is. It is clear that you doubt that the author of the system was thinking in terms of windows and buckets.
At least part of this is a fear that if I don't properly understand its dynamics I will misapply it
The explanation that you like can’t save from it either. It’s not something you think through and just write the correct code. Quick exercise with the “new” knowledge you’ve got: you have an array of a million floats, add them up correctly:
Floating Point Visually Explained (2017) - https://news.ycombinator.com/item?id=23081924 - May 2020 (96 comments)
Floating Point Visually Explained (2017) - https://news.ycombinator.com/item?id=19084773 - Feb 2019 (17 comments)
Floating Point Visually Explained - https://news.ycombinator.com/item?id=15359574 - Sept 2017 (106 comments)
I'm sure the author intended to keep the article short, but I think it would benefit from more examples, including a number like 5e-5 or something. It isn't clear to me how the window/offset explains that.
Edit: to clarify, I do not understand how the floating point representation of 0.00005 is interpreted with windows/offsets.
The number after e is a power of 10:
5e2 = 5 * 10^2 = 5 * 100 = 500
5e-5 = 5 * 10^-5 = 5 / 100000 = 0.00005
While I always new about
0.1 + 0.2 -> 0.30000000000000004
You can move from one float to the next, which is a meaningfull operation on discreet numbers like integers, but not continuous numbers like rational and irrational numbers.
I also feel like that is a much more important take away from a user of floating points knowing what the mantissa is.
Games are a great way to explain floats because they're so visual. Specifically, check out 3:31, where the game has been hacked so that the possible float values in one of the movement directions is quite coarse.
(If you're curious what a PU is, and would like some completely useless knowledge, this video is also an absolute classic and very entertaining: https://www.youtube.com/watch?v=kpk2tdsPh0A)
What do you mean by continuous? Obviously if you take a number line and remove either the rational or irrational numbers, you will end up with infinitely many holes.
The thing that makes floating point numbers unique is that, for any given representation, there are actually only finitely many. There’s a largest possible value and a smallest possible value and each number will have gaps on either side of it. I think you meant that the rationals and irrationals are dense (for any two distinct numbers, you can find another number between them), which is also false for floating-point numbers.
I meant continuous in the “mathematical sense”. But maybe continuous is actually statistical term, and dense is the correct mathematical term.
Or is that something you can do with integers and SIMD today?
Say 4x4 matrix multiplication?
https://dennisforbes.ca/articles/understanding-floating-poin...
At the time it was to educate a group I was working with about why their concern that "every number in JavaScript is a double" doesn't mean that 1 is actually 1.000000001 (e.g. everyone knows that floating point numbers are potentially approximations, so there is a widespread belief that every integer so represented is the same).