Ask HN: Is float divergence/drift a real issue in many industries? | Hacker News Reader

Ask HN: Is float divergence/drift a real issue in many industries?

8 pointsSwapnopam6 comments

Was wondering if float divergence/drift a very real issue which affects industries or is just a minor inconvenience at best.

View on HN

For physics simulations, or even in RF work, the tolerances you need are quite a bit more than say, AI work. For example, for one of our RF algorithms, we "have" to use doubles (I actually can think of an optimization that can let us use floats, but it's quite involved to fix it) to get the required precision. That precision requirement is 10^-15. For physics simulations, I've heard of accuracies of up to 10^-20.

I will say, generally, it depends on what you mean. There's like, numerical drift, and then there's converting to the discrete part. Generally, I'd say that "getting into the discrete part" of an algorithm, unless you're trying to match behavior of another piece of software, is fine. Like, if you're playing a game, people are not going to care if the shot just grazed them and missed, but the "Actual True Value" is like 10^-9 off or something. They will just chalk it up to "close, unlucky". If you have one canonical place for your floating point errors from which your "discrete" parts derive, it's no biggie.

But at work, I recently debugged an algorithm in which the author, effectively wanted to compute the integer and fractional parts of a double type. Their reasoning was that, for the fractional part, there can be imprecision, so you should do some fancy math to recover the fractional part.

This is giga wrong for the application we were building. You never want to try computing the "same double" in two different ways and compose the results like this. Intuitively, if one path "thinks" your double is 15.00001, and the other path "thinks" its 14.99999, you're now saying that the double is 15 (integer) + .99999 (fractional) = 15.99999: this is way off now!

You should just use one double, and extract out the integer and fractional parts. It doesn't matter if the double is imprecise - it's authoritatively imprecise. Much easier to reason about.

This is the first time I've debugged an edge case like this since I wouldn't implement something like this, but it's an example of how subtle floating point errors can be.

Agreed with you on that. One of the reasons for my post was, I was told by some engineers over at aeronautical companies that such float drift is trivial — anything beyond 10^-5 or 10^-8 becomes meaningless. Nearly the same was said by some engineers at HFT and quant firms. Which led me to wonder if cross-hardware drift is really a big deal. From what I've tested, even differences between an Intel i5 and Intel i7 pile up, let alone between Intel and AMD. Curious if the "it's within tolerance" answer holds in your RF work or if you've hit cases where the hardware boundary actually mattered.

Ah, I see. We don't care at that level because we aren't trying to reproduce deterministic replicated simulations as the HFT industry often wants. SDRs have their own noise and the physical world is uncertain to where, as long as your algorithm isn't above, yeah say the 10^-5 to 10^-8 range in output, it's probably fine.

It's probably the determinism and reproducibility that's the key constraint in HFT settings. In other industries it's just generic textbook numerical analysis (analyze batched eps error), and then in engineering land it's "ehh good enough".

> even differences between an Intel i5 and Intel i7 pile up

i5 and i7 are terms which lack precision. Are you seeing different results for the same operations on the same arguments in the same order across an i5 and an i7 with the same core architecture?

Regardless, it would be great if you have examples of operations that have different results on different processors; my expectation was that I could expect same width, same operation, same arguments to provide the same result across most x86 processors (early pentiums not dividing properly and maybe more variance in the socket 7 days where there were so many vendors). Most of the issues I've seen documented were not running the same sequence of operations especially around 80-bit floats on x87 if something spills to a 64-bit double in memory. All that said, I prefer to deal in integers whenever possible, I know the edges of floating point are sharp, and I'd rather not be cut.

In my own experience with data science work:

On multiple occasions I’ve seen noticeable accumulated rounding error crop up by the end of a cumprod over very long timeseries of floats.

But in my cases it has just been a minor nuisance that I have to double check whether a mismatch came from float-related quirkiness or a code mistake.

I got into the habit of just using integers where applicable for cases that need to match down to the last bit, and numpy.isclose() for inherently floaty cases where it’s anticipated and reasonable.

Only speaking from my own experience, which doesn’t involve chaotic systems or anything like that. I’m curious to hear from people in other domains who may have more exciting stories.

I've had to wrestle with float precision issues writing physics code for games. Single precision floats do not behave well when you are doing calculations far from the origin (especially when doing subtractions). I ended up having to adopt double precision floats to track the camera and entity positions in my game with kilometer-scale environments. I've also had issues with floats when doing repeat transformations on a rotation basis without periodic orthonormalization.

Reply on news.ycombinator.com