Math Keeps Changing
macwright.com
macwright.com
Comparing floats for equality is not a good idea, not just because of the 0.1+0.2!=0.3 behaviour (which, yes, is at least consistent), but because floats - or numerical methods in general - are simply not meant to be used for exact results. If that's what you want - e.g. big integer or fixed point arithmetic - there are other types you should be using. So the takeaway is not that JS has a "glitch" or is messing up - but a misunderstanding of the guarantees of numerical methods.
Also, the suggested solution to compare actual vs. expected value by having their absolute distance be less than some epsilon is half correct, but what you really should be looking at is the relative error, i.e. the absolute error divided by the magnitude of the number.
This actually depends on the specific problem you're talking about, and there's also the issue of forward vs. backward error. Unless you're doing something that requires additional care, unit-testing against abs(expected - actual) < 0.001 is relatively safe (and if you are, then God help you).
100% agree on the rest of the comment, requiring numerical methods to converge to a specific IEEE 754 double precision floating point is just dumb.
This is very true, but the mention of data science is really significant here.
If you're trying to productionize a neural network or something similar, you ideally want to be able to pin down every source of noise to make sure your results are reproducible, so you can evaluate whether a change in your results reflects a change in the data, a change in the code, or just luck from your initial weights - with a big enough network, for all you know it decided that the 10th decimal place of some feature was predictive. I wouldn't be surprised if the author brings up tanh as an example specifically because it's a common activation function.
If you're pinning all your Javascript dependencies, but the native code it dynamically links to gets ripped out from under you, it could completely mislead you about what produces good results. Similarly, if you want to be able to ship a model to run client-side, it would be nice if you could just test it on Node with your Javascript dependencies pinned and be reasonably confident it'll run the same on the browser.
Of course, if you can't do that it's not the end of the world, since you can compensate by doing enough runs, and tests on the target platform, to convince yourself that it's consistent, but it's a lot nicer if things are just as deterministic as possible.
> for all you know it decided that the 10th decimal place of some feature was predictive
isn't this kind of a misfeature of the model? If it really depends that highly on a function output that's not guaranteed to be precise even in the best of cases and certainly not when you have to assume noisy input data... that doesn't seem to be particularly robust.
But one should expect a single function does not vary in the last 5 digits as one example did. That's over 16 bits. Why should anyone use double precision if that's the kind of slop an implementation can have?
Why not? The whole selling point of floats is that they're a fast approximation to real arithmetic in a bunch of useful cases. I'd personally be much more miffed by an excessively slow implementation of a transcendental than one which was off even by a large number of bits. If there's a fast solution with better precision then that's great, but low precision by itself doesn't strike me as particularly problematic.
> Why should anyone use double precision if that's the kind of slop an implementation can have?
Because by using float64 instead of float32 you can cheaply get a ton more precision even with sloppy algorithms. If there's only 16ish bits of slop you could probably get a full-precision float32 operation just by using an intermediate cast to float64 (proof needed for the algorithm in question of course).
0.9999999999999999999999999999999
1.0000000000000000000000000000000
This is such a disheartening conclusion. It really was a bug in your code, describing it as a "glitch in the language" is disingenuous. Your code was assuming higher level of precision (or accuracy? I get those always mixed) from the system was providing, that is not a glitch in the language, but bug in your code making that false assumption.
This also leads to the question where did the original "13098426.039156161" reference value come from? Has he actually verified by hand that it is correct to the last decimal? WolframAlpha gives lot more decimals for the value, how did he decide that exact number of digits?
Of course hindsight 20/20, but numerical code just is generally much more difficult and complex than many people give credit to, at least before they get bitten by some issue like the author.
It doesn't really work when you're dealing with transcendental functions such as the trigonometric ones. There is simply no way to get "accurate" results for such calculations in general, unless of course, you deal with everything symbolically.
I've recently seen a version of this error where someone's code was calculating how many bits a field needed to be to hold the maximum value they cared about.
bits = floor(log(max_value) / log(2)) + 1
And if you want just the floor of the base-2 log, you almost don't even need to calculate it at all; it's right there in the exponent bits.
And one should not expect that the CPUs will ever standardize the results of the functions there. We are de facto left only with IEEE 754, and there are even regarding that continuous attempts to implement less.
So what's left is the wiggle room, but not only in JavaScript as the specification or specific interpreters, but in other languages too.
"stdlib" sounds good, but do they guarantee to never change any implementation of some math function?
I don't agree what the author saw was really "a glitch in the language itself." It's directly allowed by the language to happen.
It is a great rabbit hole, however I have to note that the raised issue seems to be with float32, while JavaScript uses float64 ?
However, the point was not so much about precision, it was about consistency.
Specific examples include differential equation solvers (vastly different orders of magnitude to deal with), plotting libraries (only have to be correct to within a pixel), and a ton of other graphics-related operations for that matter (only have to be correct to within a pixel, and most classes of small errors won't be important anyway).
Some amount of error can even be desirable (see a lot of the work the last half-decade in reduced precision neural networks).
Floats aren't bad -- they're a tool in your toolbox, and they won't hammer every nail.
Even worse the language of math was never adequately explained. You were expected to learn the meaning of some new squiggle or greek letter by... its context? Symbols and terms you'd never seen before were just thrown at you without explanation. Definitions when given sounded circular or were given in terms of other things that had never been defined clearly.
I had one professor who just turned his back to the class and wrote things on the board. Students were meant to just magically understand.
I'll never forget in year one of college being absolutely stuck in calculus. I called up my dad and he tried to help me a bit, then he stopped and said: "you know what a derivative is, right?" I said no, not really. They tried to explain in class but it didn't click. He said "a derivative of a function is the rate of change of that function." I thought for a second and then said "thanks, now I understand calculus." I was un-stuck instantly. My professors never explained it that clearly.
Later on I took classes in things like population genetics and evolutionary dynamics. The professors of those classes explained the relevant math better than my math professors did to the point that if I'd taken those classes first I would have done better in the math classes that were their prerequisites.
> He said "a derivative of a function is the rate of change of that function."
I'm confused by this comment. That's the intuition that you see in just about every introductory calculus textbook: that the derivative is the instantaneous rate of change.
This was back in the late 1990s and early 2000s. I hope things have improved.
My conclusion is different though. I think there are different types of thinkers and different teaching styles and when the subject becomes very complex the disparity between the two is exacerbated. And luck plays the role in matching up with the right teacher/professor for you.
I was taught calculus in high school, well before university, in Australia in the late 1970s.
Visual analogies such derivatives being tangential rates of change and 2nd derivatives being local curvatures were taught and drawn on the board in the first week of two years of Calc I and Calc II classes that came before leaving high school to attend university.
Nearly all math textbooks and lectures mention practical applications. The problem rather seems to be that you have/had a different understanding of "practical" than your math professor.
I always had the benefit of being the tutor, though. I have to imagine that teaching a calculus one course is near impossible with 30 students on 30 different levels of experience and aptitude.
That said, I think that first year of calculus is often taught valuing mechanics over concepts.
I like to imagine what 12 years of math would look like if people weren't chained to a timeline. Here's khanacademy, go!
https://betterexplained.com/articles/colorized-math-equation...
Chances are your teachers said almost, if not exactly, the same, but it didn’t stick yet.
In my first analysis class, the professor said we shouldn’t worry if we couldn’t complete any of the exercises, adding something along the lines of “If, during the course, you look back at the exercises from four weeks ago, you will find you can easily do them, and won’t even understand why anybody could have trouble doing them”. That turned out to be o so true.