As a general rule, if you find yourself comparing floating point numbers, that's probably not what you want.
I'm wondering what they are going to do with other operators, >=, <= etc
I'm wondering what they are going to do with other operators, >=, <= etc
However, there are cases where comparing floating point numbers for equality works just fine. For example, if you have a variable 'v' and want to update it to a new value 'f', you can do 'if (v == f)' to check if the variable would change. Or if you have a sentinel value -1.0, it is perfectly ok to do 'if (a == -1.0)'
In general, if you look for a number 'f' in variable 'v', you can safely use equality as along as you expect 'v' to be set to 'f', i.e. the value of 'v' is not the result of a computation. This may sound trivial, but this is often omitted when discussing floating point comparison. If someone knows a scenario where the examples above would break, I would be curious to hear! (My main field is audio programming and code like this is omnipresent.)
almost invariably floating-point calculations end up in some kind of comparison; if they didn't we'd probably do the calculations in a galois field or something instead
even equality comparison of floats is useful; there are an abundance of seminumerical algorithms which use it correctly on the results of computations. a trivial example is computing an attractive fixpoint by iterating a function and comparing each hare result to the previous iteration and to a tortoise. you have to be especially careful about x87 extended precision here
ieee-754 requires bit-exact results for the five fundamental operations
here is an example where your 'safe' generalization is not safe:
#include <stdio.h>
int main()
{
float f = 0.7;
printf(f == 0.7 ? "ok\n" : "wtf\n");
return 0;
}
I'm pretty sure OP meant comparison for equality, not comparison in general (which would be absurd, indeed).
> even equality comparison of floats is useful
That's exactly what I tried to point out. However, these are really special cases and you need to be careful and know what you are doing.
In Rust if we insist we want to compare an f32 (what C would call float) to an f64 (double), the language says it doesn't have a way to do that, highlighting our mistake. If we just try to compare an f32 with value 0.7 against 0.7, the type inference concludes 0.7 is also an f32 so it's equal, and if we do the same with an f64, this time by inference 0.7 is an f64 and so again it's equal.
I am a quickly crustifying developer who was an enthusiatic value packer in their younger years but I've changed my tune to be less afraid of using more variables if they are clearer - even if that comes at a cost to data efficiency (and like, an extremely negligible cost in most situations). Though I don't have any objections to value packing for serialization - that's a fundamentally different domain.
Isn't it strictly faster to simply assign? No branching, no branch predictions, only one instruction in all cases.
float freq = getParameter(FILTER_FREQUENCY);
float coeff;
if (freq != mFreq) {
coeff = calculateCoeffFromFrequency(freq); // expensive
mFreq = freq;
mCoeff = coeff;
} else {
coeff = mCoeff; // use cached value
}
// use coeffOf course, that observation may be more or less useful, depending on circumstances.
-4 always removes 4, whether or not there is a number to the left.
The only acceptable grammar for math is s-expressions.
I think that this is an oversimplification, and misses a key point.
Floating point numbers can be interpreted in (at least) two distinct ways.
In one use model, floating point numbers allow a sparse representation of selected points on the (extended) real number line. In this model, the numbers themselves are perfectly precise, and the basic operations do exactly what you'd expect -- but because the result of the expected real-number operation may not be one of the selected representable points, each operation is potentially paired with a second, implicit, rounding operation, and the consequences of the cumulative effects of this rounding operation take some thought.
This first model is often comfortable for those who have done low-level numerical program with sized integers, which are likewise a selected sparse representation of selected points on the integer line. While rounding comes up in fewer operations here, and is simpler, wrap-around becomes an issue.
However, a second model of floating point numbers is that each possible bit sequence represents an /interval/ on the (extended) real number line. In versions of this model, while the set of representable intervals is still very restricted, every point on the real number line falls into a representable interval, which is a property that sometimes eases (and sometimes seems to ease) analysis. In exchange, the operations are much more complex, and their limitations are much more obvious -- in particular, not only can no disjoint intervals be represented, but for any given point on the real line only one interval that contains it is representable -- there's no way to represent intervals of different sizes containing (some of) the same points.
This second model is what people are mostly thinking of when they say that floating point arithmetic is "not precise." In particular, both the inputs and outputs of operations have imprecise interpretations -- that is, the input interval contains infinitely many indistinguishable points, and the output interval contains infinitely many indistinguishable points, even if a real number equivalent of the problem would have a single number in the desired range.
When extended to true interval arithmetic, the second model can be very useful for understanding error propagation and uncertainty, but sadly this is rarely done.
as you point out, the second model can be implemented
Without operators, all you have is a set of objects which you can't tell apart.
[1] https://numpy.org/doc/stable/reference/generated/numpy.isclo...
[2] https://docs.python.org/3/library/math.html#math.isclose
[1] https://www.erlang.org/doc/reference_manual/expressions.html...
This is a sad state of affairs, but nobody was fixing this in the last 30 years, so there's that.
But they provide far more than I need. I'm not worried about liquidity when I eventually sell those stocks, and if I could opt out of buying extra liquidity I definitely would.
If I have a medium size order, then even though they lower the spread they also front-run and limit how much I can buy at that price. So I'd rather have them not be there.
If I have a tiny order, then I don't care what the spread is within reasonable bounds, and I still don't want them to be there.
if you're an investor, otoh, the lower spread and greater liquidity means timely execution costs you less, not more. you aren't paying them for liquidity; they're paying you, or rather you're paying them, but much less than the spread you'd've paid an old-style open outcry market maker
(do you even remember markets before decimalization? minimum spread 12.5 cents)
of course you do need to execute intelligently; you can't just plop a million-dollar order in a hundred-million-a-day market and expect the market not to move against you
It's fine for the market to move against big orders, I just want the movement to, like, take one second. I don't want anyone to change their position in response to an order that hasn't even resolved.
> (do you even remember markets before decimalization? minimum spread 12.5 cents)
Well I'm not suggesting we undo that.
> if you're an investor, otoh, the lower spread and greater liquidity means timely execution costs you less, not more. you aren't paying them for liquidity; they're paying you, or rather you're paying them, but much less than the spread you'd've paid an old-style open outcry market maker
In this scenario I'm a long-term investor so the cost means nothing to me. So it's a matter of whether I want HFTs to profit, and I don't, because their actions are often not win-win. If HFT worked somewhat differently I wouldn't mind them the same way.
As matter of fact, business apps are by far the longest portion of the apps. Just considering all spreadsheets, almost all RDBMS, etc alone.
Binary Floating-point is what is ACTUALLY niche. And it not look like it just because is the default, similar how in certain niches "0" is "true".
Physics simulations too.
nobody needs 170 trillion additions for their nightly invoices
but is there a better algebraic structure to use when thinking of floats? like, in terms of limits, or something that tracks significant digits and uncertainty? how do formal methods handle these?
Can you motivate this with an example for me? For instance, I think of games or location data encoded as FP; then, clearly, comparing them is a critical task to know questions like "what is closer" and so on.
What am I missing?
First point: FPs are unique. Clearly true
Second point: Combing FPs isn't lossless, across most/all arithmetic. Clearly true, the root of our discussion
Third point: decimal arithmetic is lossy too. Strongly disagree. The system of representation is what is lossy - floating point. Arithmetic (between two decimal numbers) is clearly not lossy in and of itself, only as implemented with computers.
Decimal numbers with any limit on digits, even a hundred, are lossy.
Decimal numbers as used by humans outside of computers have limits on digits, so they are lossy.
Combining two FP numbers with arithmetic leads to possible errors that you can't represent with FP on every operation.
Combining two decimal numbers with non-division arithmetic never leads to that case. With division, sure, things are bad, but that's more of an exception than a rule to me.
This is because decimal numbers don't really come with any inherent limit on digits, and it's a bit strange to add a clause to the other's claims before making a counterargument.
But even then, multiplication causes an explosion in number length if you do it repeatedly.
When you specifically talk about numbers not being in computers, I think it's fair to talk about digit limits. Most real-world use of decimals is done with less precision than the 16 digits we default to in computers. Let alone growing to 50, 100, 200, etc as you keep multiplying numbers together to perform some kind of analysis. Nobody uses decimal like that. Real-world decimal is lossy for a large swath of multiplication.
I agree that if you're doing something like just adding numbers repeatedly in decimal, and those numbers have no repeating digits, then you have a nice purity of never losing precision. That's worth something. But on the other hand if you started with the same kind of numbers in floating point, let's say about 9 digits long, you could still add a million of them without losing precision.
And nobody has said anything about irrational numbers as you dismissed in your other comment.
So in summary: decimal division, usually lossy; decimal multiplication, usually lossy; decimal addition and subtraction, lossless but with the same kind of source numbers FP is usually lossless too
Most types are subject to overflow that has similar effects. Most of the FP error people encounter is actually division error. For example, the constant 0.1 contains an implicit division. The "tenths" place is defined by division by 10. I think that almost all perceptions of floating point lossiness come from this fact.
I think people are speaking shockingly imprecisely in this thread. here's my claim:
> Arithmetic (between two decimal numbers) is clearly not lossy in and of itself, only as implemented with computers
Counterarguments are basically on the level of "you cant represent Pi as a finite decimal number", which I find uninteresting.
'decimal' means 'base 10'
numbers, the abstract entities we do arithmetic on, are not decimal or in any other base; it is correct that arithmetic on them is not in any sense lossy
decimal is a system of representation that represents these abstract numbers as finite strings of digits; these are called 'numerals'. it can represent any integer, but only some fractions, those whose denominator has no prime factors other than 2 or 5. such fractions arise non-lossily as the result of the arithmetic operator of division when, for example, the dividend is 3. representing these in decimal requires rounding them, so decimal is lossy
binary floating point is lossy in the same way, with the additional limitation to only being able to represent fractions whose denominator is a power of 2, whose numerator does not have too many significant bits (53 most often), and which are not too positive, too negative, or too small
there are other systems of represention for numbers that are not lossy in this way. for example, improper fractions, mixed numbers, finite continued fractions, and decimal augmented with overbar to indicate repeated digits
Floating point numbers lose precision because binary arithmetic doesn't represent decimal in all cases ( 1/3 is an easy example). It's not hard to get into a situation where you're asking if a number is 0.0 but due to a precision error the number you have is 0.00000000000001 or whatever and it should have been zero.
If you're dealing with anything where precision is paramount (i.e. money or high precision machining) you should consider using something else that matches the real work precision you are trying to model.
You can, of course, use floats and test differences rather than strict equality (i.e. distance is < 0.01) and that's fine too... if you remember to do it, and account for the potential for small precision differences to propagate throughout your calculations and accumulate into larger errors.
That’s not the reason. A counterexample is decimal arithmetic. That represents decimal in all cases, yet loses precision when doing calculations. Once you start doing division that applies even to arbitrarily sized decimal numbers.
The correct reason is because the subset of the reals that is representable as IEEE float isn’t closed under the operations people want to perform on them, including the basic ones of addition, multiplication and division.
You never use the word “comparison” again in the comment!
Strict inequality (> and <) comparison are generally fine as long as you avoid NaNs.
Well, followup, if some error Epsilon can be introduced during any manipulations, when you check for strict inequalities do you check x < y + Epsilon? Does the language implicitly do it for you?
If you know numerics, then you can come up with the correct value for epsilon. But that is hard work. There are various things that the language could do for you, like use a dynamic amount of precision, or interval arithmetic where the error bounds are saved—but! Most of the time people just want the answer faster and with less memory used, which is what you get with bare floats. The people who make it their job to care about numerical accuracy can do it better than the language runtime would anyway.
Most of the problems are more easily solvable at a higher level anyway. Like, imagine Clippy saying to you, “It looks like you’re inverting a matrix. Are you sure that’s a good idea?”
Playing with Python3:
>>> +0.0==-0.0 True
The C in GCC below also returns 1:
#include <stdio.h>
int main()
{
float a = +0.0;
float b = -0.0;
printf("%d\n", a == b);
}You'll get the same in python with
>>> -0. is 0.
False
Python 3.11 gives me this:
>>> -0 is 0 <stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="? True
Because (a) -0 is an integer, and there is only one integer zero, (b) small numbers have only one instance in memory as an optimization.
I would even be happy with a default set of compiler warnings or errors, which I don’t believe I have ever seen.
There can be a separate function `float_equals()` with explicit args that does what people want
The only reason to use the same syntax == is for POLYMORPHIC code that is actually correct.
But it's not going to be correct with floats, because they don't obey the same algebraic laws ... So the syntax should be different!
---
I believe the Erlang compiler optimization presented as justification for this change is a good example
The compiler wants to reason about code, independent of types
But that reasoning about the =:= operator is wrong for the float case.
they're already familiar with rounding errors
from that perspective you're suggesting taking a step backwards from fortran i toward assembly language
the erlang compiler bug presented is not an example of what you're talking about because you're talking about arithmetic, and the buggy comparison operator it's using is not an arithmetic comparison operator
But I would still say that floating point equality is a vanishingly rare operation
I can't think of any real use case for it, other than maybe some (bad, incomplete) unit tests that assert 1.0 == x and assert 1.0 == y
And that use case is perfectly served by float_equals(), and arguably served better if it has some options. Although probably abs(x - y) < eps is just as good, and you don't even need float_equals()
---
Can you show some real use cases for floating point equality in good, production code? (honest question)
It's similar to hashing floats, which Go and Python do allow. I made an honest request for examples of float hashing:
https://lobste.rs/s/9e8qsh/go_1_21_may_have_clear_x_builtin#...
https://old.reddit.com/r/ProgrammingLanguages/comments/10bm2...
I didn't get any answers that appeared realistic. Some people said you might want to create a histogram of floats -- but that's obviously better served by bucketing floats, so you're hashing integers.
Another person said the same thing for quantizing to mesh.
One person suggested fraud detection for made-up values, but that was clearly addressed by converting the float to a bit pattern, and hashing the bit pattern.
----
This isn't a theoretical question since we're working on this for https://www.oilshell.org.
For that case it seems pretty clearly OK to omit === on floats, since I use awk and R for simple stats and percentages, and that's likely what people would do with a shell.
Though as always I'm open to concrete counterexamples.
even hashing is useful for memoization
i think probably you're asking in the wrong places
the lapack source might be a better place to look; we can probably suppose that if something is in lapack it's realistic and also not a bug, and numerical code doesn't get more 'good, production' than lapack
in dgelsx for example i find
ELSE IF( ANRM.EQ.ZERO ) THEN
*
* Matrix all zero. Return zero solution.
*
CALL DLASET( 'F', MAX( M, N ), NRHS, ZERO, ZERO, B, LDB )
* Determine RANK using incremental condition estimation
*
WORK( ISMIN ) = ONE
WORK( ISMAX ) = ONE
SMAX = ABS( A( 1, 1 ) )
SMIN = SMAX
IF( ABS( A( 1, 1 ) ).EQ.ZERO ) THEN
RANK = 0
CALL DLASET( 'F', MAX( M, N ), NRHS, ZERO, ZERO, B, LDB )
GO TO 100
ELSE
RANK = 1
END IF
and in dlar1v:
WORK( INDS+I ) = S*WORK( INDLPL+I )*L( I )
IF( WORK( INDLPL+I ).EQ.ZERO )
$ WORK( INDS+I ) = LLD( I )
TMP = D( I ) / DMINUS
IF(DMINUS.LT.ZERO) NEG2 = NEG2 + 1
WORK( INDUMN+I ) = L( I )*TMP
WORK( INDP+I-1 ) = WORK( INDP+I )*TMP - LAMBDA
IF( TMP.EQ.ZERO )
$ WORK( INDP+I-1 ) = D( I ) - LAMBDA
IF( ABS(MINGMA).EQ.ZERO )
$ MINGMA = EPS*WORK( INDS+R1-1 )
but if you ask an audience of type theorists, sysadmins, and web developers, they won't know that; they aren't numerical analysts and probably haven't inverted a submatrix since sophomore math class, if ever
(there are 6550 files in lapack 3.9.0 of which i looked through 15 to find these examples, suggesting that there are on the order of 1000 realistic examples in there; eventually i'd probably find one that wasn't even comparing to zero)
plausibly instead of giving them floating point by default you should give them fixed point with, say, nine digits after the decimal point; that was the approach knuth took in tex, for example, to guarantee reproducibility, but it also gives human-predictable rounding. and in most cases it should be perfectly adequate for simple stats and percentages. as a bonus you get 64 bits of precision instead of 53
It wouldn't be too surprising to me if 90% of float equalities in real code are comparisons to 0.0 or 1.0 or -1.0 (that's what I meant in my original post, there was a typo)
The point about x >= 0 and x <= 0 is interesting. I suppose to be consistent, you would also remove those operators on floats, although for something awk-like, I don't think it matters either way.
---
The other point is that it looks like the Fortran code isn't actually using == or x.eq.zero polymorphically!
So the first point is that I would want to understand conceptually where float equality would be equal. You mentioned fixed points, although there I also suspect that abs(x - y) < eps is a better test in most situations, though certainly there could be counterexamples.
The second point is -- would any of those situations be polymorphic with int and float, or string and float? It depends on the language, but it seems vanishingly unlikely in most languages, and in particular in say in C++
I would probably go look at the equivalent Eigen code if it mattered with respect to Oil, but it probably doesn't
i agree that prefix notation is not as bad for these as i had thought before looking
i think there are plausible cases where your float might be polymorphic with double, complex, a vector, a matrix, or an element of a galois field, but i think it would only be polymorphic with int in cases like stopping change propagation when an observable value gets recomputed to the same value from new inputs, and in those cases what you need is exact bitwise equality, not arithmetic equality
an awk-like thing might be better with fixed-point, like tex and metafont; you could call the data type 'btc` and represent it as a 64-bit integer that's multiplied by 10*-8 for output
The solution is to either use decimal floats if they suit your need (and eat the performance penalty), or to use a linter that flags float comparisons by equality.
Another one is that you would lose transitivity on equality. With a 1% epsilon, you would have
1.000 == 1.005
1.005 == 1.010
1.010 == 1.015
1.000 != 1.015
You also lose that, if a == b and c == d it follows that a + c == b + d.
The behavior around zero also will go against intuition. If you consider 0.99 and 1.0 to be equal, do you really want 1.0E-100 and -1.0E-100 be different?
Then it wouldn't really be an equality comparison any more. For instance according to your rule 1.0 and 1.01 are the same number, but they clearly are not. But if you want to do this regularly you could create a special type and overload the equality operator (if your language supports it, otherwise you'll have to call the function directly) that points to 'boolean about_equal(float f1, float f2, float epsilon)' or something equivalent where epsilon is the amount that you would allow f1 and f2 to deviate from each other to still call them equal. And you could define epsilon as ((abs(f1) + abs(f2)) / 100) .
The bitwise comparison isn't 'blatantly wrong', it's the expected behavior in just about every programming language. I'm not aware of any language that has a native float operator that does this but as you can see you can usually simply add one yourself if you really need it. But this need rarely comes up and usually indicates that you are doing something wrong.