Catastrophic Cancellation (2020)

This is a strange tweet. It assumes people are familiar with classical cancellation error, but not familiar with error analysis. Which in my experience... people either understand both, or are ignorant of both concepts.

The general point is that "cancellation error" happens more than just in floating-point operations, but also in "classic scientific sig-fig error analysis".

---------

The tweet should either be dumbed down to discuss cancellation error in floating-point arithmetic, or elevated up and assume people know about sig-fig analysis. It sits at a weird point in the "assumed knowledge" curve.

---------

For people unfamiliar with cancellation error, try the two following statements in Python3 (which defaults to double-precision... aka 53-bits of mantissa).

    poor_ordering = 9007199254740992.0 + 1.0 + 1.0 + 1.0 + 1.0 - 9007199254740992.0
    good_ordering = 9007199254740992.0 - 9007199254740992.0 + 1.0 + 1.0 + 1.0 + 1.0

What are the values of "poor_ordering" vs "good_ordering" ?? What does this tell us about double-precision?

9007199254740992.0 == 2^53. So it is impossible for a double-precision number to accurately represent +/- 1.0 at 2^53. (Note that +/- 2.0 will work out just fine).

Play around with 9007199254740992.0 +/- 1.0, or 2.0, and other values for about 15 minutes, and you'll probably learn everything you need to know about cancellation error from that playtime alone.

Double-precision numbers are composed of 52-explicit bits + 1 implicit bit + 1 sign bit + 11-bit exponent bits (yes, 65-bits total. The implicit bit "doesn't count", but makes 0.0 and subnormal numbers harder to deal with)

It wouldn’t and cannot, because there’s only one operation in that calculation, so there’s nothing to rearrange.

That example only shows that relative errors can explode even when doing simple calculations.

Another thing is that, in computers, calculations often are imprecise.

Because of catastrophic cancellation and similar issues, that means that the computer result of a calculation can be quite different from the mathematical result.

To make matters worse, in real life, we often don’t know the exact values of things we measure, so even if our calculations are mathematically perfect, the outcome of a calculation by computer can be quite different from the real result.

So, if you do a computer calculation, say to compute how strong a bridge has to be, you really, really need to know now close the computed value, at worst, is to the mathematically exact result.

That’s what numerical analysis is about. For a given calculation, it might say such things as

“if the input is between 100 and 200, to get a result with n decimal digits of precision, you’ll have to compute all intermediate results with 4 × n digits.”

or

”but if you rearrange the computation like this, you only need to use 2 × n digits for n digits of precision in your result”

Doesn't help for the "older question." That's just used as an intuitive example for how imprecision can arise from subtraction.

Rearranging helps when you can only store intermediate results with finite precision, but you can compute them to arbitrary precision.

I think it's a bad example, but illustrates an important point he doesn't make explicit, that sometimes the variable you need to estimate is the DX itself, not just X1 and X2 to produce X2 - X1 = DX. With a sufficiently high amount of variance in your approximations of X1 and X2 their difference will tell you little to nothing about DX.

maybe he wants to implictly display correlations. if you substract directly you make the implicit assumptions the two even dates are uncorrelated.

If you add a function which increments a dummy value 4000 times, your test coverage will increase!

These tweets are already threaded on twitter, why is this app even needed?

It tends to pollute the replies more often than not, with more people invoking it than actually replying.

The Celsius temperature scale is an interval scale [0] which means that it is possible to calculate differences but not ratios. The Kelvin temperature scale is a ratio scale [0] (it has an "absolute zero") that allows to do that.

Beside that if there are uncertainties involved, one should do proper propagation of uncertainty anyways. [1]

[0] https://en.wikipedia.org/wiki/Level_of_measurement

[1] https://en.wikipedia.org/wiki/Propagation_of_uncertainty

This is talking about the error in the difference between two values with the same units though. For temperature, it wouldn’t matter if you’re using C, K, or F for your starting values, the % error of the difference would be the same (I think).

Yes, because there is an arbitrary choice of origin which renders the relative error dependant on units. If you're measuring a length, for instance, or an interval of time, the relative error is independent of which units you choose. If you're measuring e.g. a distance to some point, then again you have an arbitrary choice of origin.

I am sometimes joking with friends by suggesting: Let's meet at 12 o'clock ± 5%.

Yes it does and should be used when precision is important.