Why does integer representation of a floating point approximate its logarithm? | Hacker News Reader

Why does integer representation of a floating point approximate its logarithm?

stackoverflow.com

116 pointssegfaultbuserr19 comments

In a recent talk[1], Bill Dally of Nvidia suggests using actual logarithms, as they're more consistent in relative error than floats, which is especially relevant at low precision as increasingly used in AI. Multiplications then become simple additions, but then addition becomes more complicated, as in general it needs lookup tables. This is the same mechanism as a slide rule, and also reminds me of the way the Yamaha DX7 works. The lookup tables can be expensive, and the talk contains an idea how to batch them when adding up the results of many many multiplications (as is also often the case for AI workloads).

[1]: https://www.youtube.com/watch?v=kLiwvnr4L80

I would guess that additions are pretty important in deep learning workloads. For backpropagation, accumulator on the big delta matrix is very important.

What a wonderfully written question... with a patently obvious answer: IEEE-754 floating point formats store the exponent in the most significant bits (other than the sign bit, which is why the question is limited to positive values).

Indeed! Given the effort and knowledge put into the question, the obvious answer seemed too... obvious for this poster to not already know it. I thought I must be missing something!

A striking reminder that fluency is not the same thing as competence...

Perhaps because the poster seems to be historian and not a CS practioner.

From the question

"Why I am asking

I am writing a history of the Fast Inverse Square Root, information about which is collected at 0x5f37642f.com. Part of investigating this as a historian means asking questions like this from the community. I do not have the expertise to distill, by myself, much of this because I am not a mathematician or a computer scientist."

The key that TFA probably missed is that the bit pattern following the exponent (which is by definition an approximation of the logarithm) is the "linear" bit of the "piecewise linear approximation", while the exponent is the "piecewise" bit. That's really all the answer needed to explain in the end.

"Why is an IEEE 754 positive [normal] number interpreted as an integer a piecewise linear approximation of its logarithm?" -> "Because the bit pattern for it will have an exponent followed by a mantissa, where the exponent by definition is a very rough approximation of the logarithm, and where the mantissa can be interpreted as a linear interpolation between the exponent and the next exponent, thus the two pieces as the "piecewise" (exponent) and "linear" (mantissa) approximation of the number's logarithm."

The only remaining bits of interest is the bias of the exponent and the assumed-1 leading digit of the mantissa being left out. The latter is a compression technique (why wastefully include a bit that's always set or always reset?) that happens to help here (avoiding having to shift the mantissa bits by one). The former is easy enough to deal with.

Fun fact, related:

Julia has a function to give you the next greater floating point number for any floating point number, nextfloat(x), and it is basically implemented by casting x to an integer, adding 1, and casting it back.

https://github.com/JuliaLang/julia/blob/3120989f39bb7ef7863c...

Positive floating numbers can indeed be compared by their binary representation. The fact that incrementing the number by one gives you the next floating point number is a trivial corollary.

This can be useful: https://en.wikipedia.org/wiki/Fast_inverse_square_root https://www.youtube.com/watch?v=p8u_k2LIZyo

Roughly, the answer should be intuitable from what a float is,

    A float is ...-exp-int
    A log is exp-int -> int

In applications like graphics rendering or scientific computations, how does the choice of precision in floating-point representation (single vs double precision) affect the accuracy and performance of logarithmic calculations, especially when relying on this approximation method? Are there benchmarks or scenarios where the difference between these representations is particularly notable?

If njuffa and edp have weighed in on a floating-point question, it is canonically answered (if you're really lucky you might sometimes get one of the French to join in, but any combination of 2 suffices).

Let me just guess: floating point exponent is the same base as the logarithm base so you can just operate on the mantissa only and consider the exponent separately.

This may be the longest Stack Overflow question I've ever seen, with a code block, three graphs, and a list of links to related questions, comprising four and a half screenfuls on my display. (The accepted answer is five and a half screenfuls.)

People who complain about their experience of posting questions on StackOverflow often seem to have started with the idea that it was a website for getting help, rather than what it actually is: a way to compete for the attention of a certain community of potential answerers. Like most games, this too has become harder over the years, in this case as the player needs to work harder to distinguish themselves from the mass of low-effort posters.

(Not every question needs to be this long--this one's length seems to be because the asker is more a historian than a mathematician--but length and such sections--graphs, links etc--are one way it gets signaled that you really have put effort into the question.)

There may be several ways to look at StackOverflow. For example, as someone who answered thousands of questions, I don't care about author's effort--I optimize for future readers from google when considering what question is worth the time to answer it.

Yeah I agree; there are several strategies for the game as a question asker. For me when answering (only a few hundred answers across all Stack Exchange sites, so I'm already in the "tail" and probably not representative of the typical answerer), I mainly care about whether the question seems interesting and challenging to me (I'm especially likely to answer if I think no one else would answer). But the point remains that with several thousand questions being asked a day on Stack Overflow, most potential answerers are indeed likely considering "what question is worth the time to answer it" as you said, so a question needs something (effort, interest, broad utility… or just a stroke of good luck) to get "successful".

So when asking (~80 questions asked so far across SE sites, only 8 on Stack Overflow), I try to put effort into it (and show it), but ultimately still just treat each question as a gamble and if it doesn't "work out" (question got few/zero/negative votes, or no useful answers) it's no big deal (just lost a turn, try again next time… at least I learned something about what sorts of questions work well on this particular site, and got an opportunity to state my question as clearly as I could, which is often the most useful part of the process). But I can see how people who approach asking questions on StackOverflow from a mindset of "I need help and hope to get it here" can easily get frustrated.

Yeah. Admittedly I haven't been burnt in a while but stackoverflow (stackexchange, really, i haven't truly learnt from stackoverflow in a while) is the highest signal to noise I know of online

Reply on news.ycombinator.com