How to compare floats in Python
davidamos.dev
davidamos.dev
Decimal is also not a solution to floating point approximation. Decimal has all of the same rounding properties that binary floating point has, and actually it's a bit less well behaved. The differences are that Decimal round trips between textual decimal representations, and that Decimal supports an arbitrary choice of bit length.
There are cases where one does need to check for convergence to arbitrary values, but these are probably not what you are doing, and if they are you should probably know more about the precise numerical guarantees that you have.
This is not true.
> Unlike hardware based binary floating point, the decimal module has a user alterable precision (defaulting to 28 places) which can be as large as needed for a given problem
> Both binary and decimal floating point are implemented in terms of published standards. While the built-in float type exposes only a modest portion of its capabilities, the decimal module exposes all required parts of the standard. When needed, the programmer has full control over rounding and signal handling. This includes an option to enforce exact arithmetic by using exceptions to block any inexact operations.
Plenty of people find that 0.2 not being exactly representable in binary floating point is not intuitive.
[0]: https://en.wikipedia.org/wiki/Decimal32_floating-point_forma...
[1]: https://en.wikipedia.org/wiki/Decimal64_floating-point_forma...
[2]: https://en.wikipedia.org/wiki/Decimal128_floating-point_form...
Depending on your domain, they usually are. Perhaps your floating point number represents a reading from a sensor which has much less precision than your floating point type.
It would be more correct to model all of your approximate variables with error bars (and perhaps even error distributions) but in practice, treating the closest floating point number as a point estimate and doing all your comparisons to within some reasonable tolerance may be an acceptable approach.
I must admit putting a mystery epsilon in my code here and there for that reason. Admittedly the wrong thing to do.
I mean, yes, but there are plenty of cases where a result would be equal from an analytic perspective, but the implementation never realizes this due to floating point arithmetic.
>You also shouldn't treat exact floating point values as if they are approximate—floating point is not approximate, some floating point operations are (conditionally) approximate. A value of exactly 1.0, say from the result of min(1.0, f), is not going to arbitrarily change to a different value.
You're not wrong, but I think you also need to be careful here in how you communicate this. I've encountered plenty of situations where low-level details of floating point arithmetic mattered because it was only (in a sense) approximate, and I've even gotten non-deterministic behavior out of what should have been a deterministic program due to floating point issues.
Generally, though, if it isn't fairly obvious that equality should be preserved, and if you don't have exactness as a requirement, then you probably shouldn't be building your solution around equality checks at all.
Other than that I'm not sure what you're adding. The point is people are unclear when they write code because they do not understand floats, and because few people really understand round-off error, they should as a rule compare with np.isclose unless, as you say, they now for a fact they expect a specific floating point number (although the times that happens is by and far the minority in actual code).
Alas, floating point arithmetic is not necessarily deterministic in the sense that running the same Python or C code twice is guaranteed to give you the some results.
First, your compiler is often allowed to do weird things. (Like compile the same code in different ways.) And there's some weirdness with the state of flags that might be different from iteration to iteration.
See https://randomascii.wordpress.com/2013/07/16/floating-point-... for some more information.
I was taught that you can only get into trouble when you e.g. subtract two approximately equal values from each other.
Eg when you want to add a list of (non-negative) numbers, it makes a difference what order you add them in. The usual recommendation is to add them in ascending order.
(i.e. in the parent's case of "checking that, if you add a bunch of disjoint probabilities, they're at most 1", doing that with approximate equality seems fine to me?)
> There are cases where one does need to check for convergence to arbitrary values [...]
the advice in the article doesn't strike me as wrong per se. If you're implementing a numerical algorithm for some reason and you want to check that it's well behaved, adding a couple of tests where you check for approximate equality of some results as detailed in the article seems fine to me. (Of course, ideally you would additionally also formally prove the numerical stability of that algorithm.)
I guess the article is maybe missing one crucial piece of advice, namely that floating point computations are not guaranteed to be well-behaved. If you don't know much about floating point computations and just read this article, you might be tempted to think that every computation involving floats should lead to a result that is "close" to the "real" result. But that's not true, numerical algorithms can be poorly behaved and error terms can multiply, leading to catastrophically wrong results.
You are misunderstanding me, this is not what I was saying at all. I think much of the time when people feel like they should use floats they can actually use floats, yet only in very rare, niche scenarios should math.isclose be used.
Using exact equality comparisons with floating point numbers, while fairly uncommon, seems a lot more frequently the right thing to do than math.isclose: for example, sorting algorithms, hashmaps and caches, checking for equality between structs, sentinel values.
Even in most cases where approximate equality is wanted, like maybe you are snapping a position to another position, or maybe you are checking that your new audio processing algorithm is producing noise below a noise floor, even then I would hesitate to use math.isclose, because it still probably isn't a good fit for the domain, particularly with it having an awkward default relative error term.
I mean, math.isclose() doesn't even form an equivalence relation, so of course it doesn't make sense to base struct or hash value equality off of it.
I don't think anyone was suggesting to use approximate equality as a form of actual equality. The only reasonable equality for floats is exact equality, yes. However, it's also an equality which is useless for calculation.
> I think much of the time when people feel like they should use floats they can actually use floats
I disagree, many LOB applications deal almost exclusively with discrete settings, and in such cases, floats are generally not necessarily appropriate. Money, for example, should generally not be represented as a float, unless you're doing e.g. financial mathematics on it, estimating return rates and so on (and then with the understanding that these calculations won't be exact).
Then what is the solution?
As someone whose worked with finance where floating point approximation lead to one being attacked by auditor's chasing 1 cent error that somehow crept in over millions of operations, what other choices are there?
Floating point introduces the worst sort of error. If you are expecting exact results floating point gets it right with almost no effort 99.999% of the time. The remaining 0.001% it gets it wrong, and it happens at seemingly random. A heisenbug in other words. But if you force programmers to use integers, all those missing fractions of cents suddenly become glaringly obvious.
Decimals / fixed point / integers don't "solve" the approximation problem. They just make it plain, forcing programmers to think about it, code for it, and test for it. If you care about what happens to every last bit, then there is no other solution.
Or to put it another way, if you don't particularly care what happens to the lsb, then fine use floating point. Problems happen because people really do care, but are seduced by floating points 99.999% success rate and use it anyway. Hint: if you are comparing for the equality of two numbers computed in two different ways, you absolutely care.
One solution to address some of the precision issues (not necessarily z-fighting, as it operates in a different coordinate system) is to dynamically re-adjust the player's world origin as they move throughout the world. This way, they are always getting the highest float precision for things near to them.
A 64 bit integer and a 64 bit float both chop up your coordinate system into the same number of points, but with the integer, those points are equally spaced which is the behaviour you'd expect from a Cartesian coordinate system (based on the symmetry group of translational invariance).
And even a 32-bit integer is still fine enough resolution to support four kilometres at one-micrometer resolution. With 64 bits per axis you can represent the entire solar system with 15 nm resolution, while maintaining equal resolution at any location, and exact distance calculations between any points no matter how close or how far.
So for the graphics you just subtract all world coordinates from the camera coordinate, and cast the result to float; for the game physics and AI, you work directly in fixed point.
The fundamental difficulty of comparing floats is that the format ensures a near constant number of digit of precision regardless of the scale. This is very useful for most calculations because it means you can calculate without worrying too much about the amplitude of your numbers. But it means that the smallest representable difference between two numbers gets bigger as numbers get bigger: that's why they are called floating point.
That's why using tolerance, etc. is not so reliable: because the tolerance will depend on the magnitude of the numbers, even by doing those simple tricks. In particular, it is important to understand epsilon is only "correct" around 1, i.e. a + eps != a is only true of a is close to 1. More precisely, epsilon is the smallest number such as 1 + eps != 1.
Yes, these are denormalised numbers. They aren't as well behaved.
def d(in: Any) -> Decimal
"Round to nearest even" for binary floating point is weird to anyone who doesn't have a numerics background. "Round 0.5 up" is normal to humans because that's what most have been taught.
Future programming languages should probably default to Decimal Floating Point and allow people to opt-in to binary floating point on request.
Not really. If you do computations like "convert Fahrenheit to Celsius" or "pay 6 days of interest at this APR" or a million other things, you run into the same basic faulty assumptions as ever
That's what they do in accounting, I think? At least Wikipedia tells me it's also called banker's rounding.
Incidentally, it's also what Python's Decimal does by default. See https://docs.python.org/3/library/decimal.html#rounding-mode...
from decimal import Decimal as d
def dec(value, prec=4):
"""Return the given value as a decimal rounded to the given precision."""
if value is None:
return Decimal(0)
value = Decimal(value)
ret = Decimal(str(round(value, prec)))
if ret.is_zero():
# this avoids stuff like Decimal('0E-8')
return Decimal(0)
return retUnfortunately C++ STL does not offer anything close to isclose (boost does however). Neither does Java. Float.compare is just == in disguise.
But while going through the docs I stumbled upon this Gem
> If f1 and f2 both represent Float.NaN, then the equals method returns true, even though Float.NaN==Float.NaN has the value false.
https://docs.oracle.com/javase/7/docs/api/java/lang/Float.ht...
You make it sound like that's crazy or something but the goal of .compare is to be used as a "comparator" to provide an ordering of all possible values in the implementation of various data structures or sorting algorithms, while the goal of the various operators is to support programmatic logic.
new String("Hi")==new String("Hi"); // False
new String("Hi").equals(new String("Hi")); // True
1==1;// True
Float.NaN==Float.NaN; // False
new Float(0f/0f).equals(new Float(0f/0f)); // True!!!Not quite: APL does comparisons with a configurable relative tolerance, which is nonzero by default. My article [0] about comparing one number to many quickly in this system starts with a discussion of the reasoning and mathematics there. It brings a lot of implementation difficulty, particularly with hash tables. It's possible to build a hash on tolerant doubles by making two hashes per lookup, and even complex numbers with four per lookup, but no one's figured out how to deal with entries that contain multiple numbers yet.
Nonetheless I would say it does make it easier to write working programs overall. But not by much: I've been working on an APL derivative and initially left out comparison tolerance as there were some specifics I was unsure of. A year or so later I noticed I hardly ever needed it even for numerical work and dropped any plans to add it to the language.
[0] https://www.dyalog.com/blog/2018/11/tolerated-comparison-par...
I'm curious what obstacle you have in mind for "multiple numbers".
The problem is equivalent to a distance query on a data structure, i.e. enumerate everything within a distance tol of a query point x in an appropriate distance metric. There are many data structures for this problem (it's a basic building block in computational geometry). But I assume the hashing solution you have in mind is what we usually call a hash grid in gamedev and graphics, which works well when tol is fixed for the lifetime of the data structure. [1] Namely, you define a grid with cell size tol and map a point x to the index cell(x) of its containing cell, which is just a rounding operation. Then you can use a hash table keyed by cell(x) to store the contents of an unbounded, sparse grid. To perform a distance query from x you need to check not just x's cell but some of the immediately neighboring cells and then filter all of those candidates based on the pairwise distance. [2] [3]
This approach works with any distance metric (including the usual suspects L^2, L^1 and L^inf) and in any dimension d although the worst-case number of cells to check in the hash is 2^d, so the curse of dimensionality is present and in high-dimensional spaces another data structure not based on rectilinear partitioning would be preferable. But hashing does work with "multiple numbers" if by "multiple numbers" you mean a vector (with not too many components) where tolerance is defined by a distance metric.
[1] Actually, hash grids work well any time the query radius is on roughly the same scale as the cell size. But if the query radius is a lot smaller than the cell size then the grid is too coarse to perform adequate pre-filtering. And if the query radius is larger than the cell size you have to check a bigger neighborhood of cells (i.e. more hash lookups) to enumerate all the candidates.
[2] Depending on the read vs write ratio, you can actually flip this around by storing each point in multiple cells so that queries only need one hash table lookup at the expense of slightly less precise pre-filtering.
[3] Instead of doing multiple hash table lookups, you can also have each occupied cell store pointers to neighboring cells (null for unoccupied neighbors), which replaces hash table lookups for the neighbors with plain memory loads. A variation on this trick is to instead store bit flags to avoid doing hash table lookups for neighboring cells which are known to be empty; this takes up far less memory.
maybe because you can trivially write it yourself with whatever precision you want:
float a = 0.1;
float b = 0.2;
// now test for equality to within half of the smallest
// value float can represent
bool equal = fabs (0.3 - (a + b)) < (FLT_EPSILON/2);
bool isclose (float v1, float v2, float prec = FLT_EPSILON/2) {
return fabs (v1 - v2) < prec;
}
There is also (for rolling your own fn)
import sys
sys.float_info.epsilon
Also, I wouldn't use epsilon, because that is a very small number and is related to representation of numbers. I would instead use a larger "tolerance" value, in `if |a-b| < tol ...` because that indicates the expected accuracy of your algorithm (unless you're really working with 1e-80 tolerance?). But TFA explains that.
But back to point (a) is that not reasonable? (If not, I've got some code to change... :)
I imagine the downside with a chosen "epsilon" (which I used as an alias for any tolerance) is that it can be truncated on some machines, plus this equation on its own doesn't do relative accuracy.
On an LEGO EV3, Python's float is 1e38 max with 7 significant digits.
I've never run into problems, which is why I keep using it, too.
https://bugs.python.org/issue1580
I thought there was a PEP for it, but evidently not.
Although this is generally an improvement, it can contribute to confusion.
https://github.com/Planimeter/grid-sdk/blob/master/engine/sh...
https://en.wikipedia.org/wiki/Machine_epsilon
https://pubs.opengroup.org/onlinepubs/009696899/basedefs/flo...
This is because == is an overloaded concept. Python has chosen to define == in the way that makes the most mathematical sense in the case of NaN.
Maybe there's a metaprogramming way to override how == in interpreted for floats?
with float_equality_test( math.isclose):
assert (0.1 + 0.2) == 0.3
# This still allows (+0.0 == -0.0):
def my_float_eq( a :float, b :float) ->bool:
return (math.isnan( a) and math.isnan( b)) or (a == b)
with float_equality_test( my_float_eq):
assert (1, 2.0, math.nan) == (1, 2.0, math.nan)Making NaNs compare unequal to themselves was a hack that dates back from the time modern floats were first introduced.
https://github.com/numpy/numpy/issues/10161#issuecomment-350...
plus a ton of flogging/benchmarking common libraries (numpy, scipy, scikit-learn) and discussion about np.isclose()
... and also this gem:
import numpy as np
a = 1
b = 10
rtol = 1
atol = 0
np.isclose(a, b, rtol=rtol, atol=atol) # True
np.isclose(b, a, rtol=rtol, atol=atol) # False
import math
math.isclose(a, b, rel_tol=rtol, abs_tol=atol) # True
math.isclose(b, a, rel_tol=rtol, abs_tol=atol) # TrueIf you are doing something that requires you to reason about their precision beyond deciding how many bits is enough, you should probably not be using floats, because then you've added some more reasoning the next guy has to understand.
There is a good reason to compare float equality though: when there's no calculation involved and you're basically just comparing object identity.
Checking if a float timestamp has changed by comparing it to an old copy of exactly the same variable is fine.
They're like analog values that can be exactly stored and copied.
Now of course you pay for that by losing absolute precision, but chances are that if you’re working with numbers like 1e20 you don’t much care about anything after the decimal point.
If your language is going to plug into the numerical computing stack (BLAS+LAPACK then all the fun stuff built up on that) then it'll need to talk binary floats.
All the annoying stuff that numericists understand but would like to not mess around with, like rounding directions and denormals, are handled nicely with 754 floats.
These are all sort of backward compatibility/legacy issues in the sense that they are based on decisions made in the past, but the hardware and libraries aren't going anywhere I bet!
Also, note that IEEE 754 does define a decimal interchange format. I bet they aren't handled as nicely in hardware, though.
Logarithmic numbers: more range, less precision https://github.com/cjdoris/LogarithmicNumbers.jl
Double arithmetic: stitch 2 floats together https://github.com/JuliaMath/DoubleFloats.jl
Of course, if you are calling BLAS/LAPACK, you are constrained to use floats, but the recommendation on DoubleFloats is clear: if you know you algorithms, use the increased precision only in the parts that matter
This isn’t really a (helpful) explanation, but it’s ample setup for: oh I see your problem, you’re converting from base 10 to base 10.
You don't. At least, not exact numbers. You almost always have to round the numbers up, or just aproximate the value.
So, while
1.2 + 2.3 != 3.5
1.2 + 2.3 < 3.6Eg if you compare (x + 1.2) + (x + 2.3) < (2*x + 3.6) that goes wrong for big enough x.
You know that at big enough values, floating point numbers are farther apart than integers?
What is this: "math.isclose()"???
In computing floating point numbers cannot be compared for equality. It is obvious why.
So "math.isclose()" is for people who cannot use "<" or ">" operators?
Golly. Do any programmers need it explained to them why floating point numbers are not precise and equality does not apply?
Decimal types are an abomination. The Fraction type blows my mind. Why? It is the sort of nonsense C++ programmers loved to do in the twentieth century, and we all (?) learnt from that and do not do such opaque things any more.
This is all very elementary mathematics, and a language is wasting its time making easy things easy. Programmers must learn the easy things
No, it's a shortcut for `abs(a-b)<tolerance`. Is `math.sin` for people who cannot implement Taylor expansions themselves?
"math.isClose()" is making easy things easy. Not easier, a little bit harder, and it is hiding what is actually happening. Like the C++ nonsense I mentioned that did all that (and I was a sinner in those days)
"math.sin(..)" makes a hard thing easy, which is a good thing.
Do you see what I mean? math.colse()" increases the cognitive load for no benefit.
That seems to me to be a matter of perspective. It doesn't increase cognitive load for me (appropriate name, meaning is clear, checking the implementation, that's how I would've done it anyway, etc.) so I guess opinions differ.
from your earlier comment:
> Golly. Do any programmers need it explained to them why floating point numbers are not precise and equality does not apply?
I think the answer is _yes_. Actually, all programmers need that explained to them. Especially python programmers, as python is used by a lot of beginning programmers. So I don't see the problem here.