2,038 karma · joined June 11, 2018
Specialized in artificial intelligence, high-performance computing, and floating point arithmetic.
Nestor Demeure (https://nestordemeure.github.io/about/)
[0]: https://www.poetryfoundation.org/poems/43521/beowulf-old-eng...
You will indeed need higher precision if you need a very large number of digits (more than what double precision gives you).
However, even with high precision, the algorithms you use to compute your moments matter: you might not have as many digits as you hope for. Variance, for example, is famously numerically unstable (sometimes giving a negative variance) when computed naively. A stable summation algorithm plugged into a variance computation (even a carefully written one) can sometimes give you six additional digits of precision in the variance that you might not get when doubling the precision (it depends on how big of a cancellation you had in your operations; too big, and doubling the precision will not be enough to recover, while a compensated summation would do fine).
I have seen this play out once in a clustering algorithm that would only work on some data when using a compensated summation to compute the variance.
2. It varies wildly depending on the algorithm you are using, the implementation of said algorithm, and the operations you are performing outside of the summation (if you sum the results of a loop performing non-trivial operations, that can easily overlap with the summation overhead making it essentially free). The only rule of thumb I can give is that runtime cost increases with desired accuracy. My recommendation would be to switch from a naive sum to using an exact accumulator and observe the difference in output value (letting you know how impactful numerical errors in that sum were) and runtime (letting you know how expensive the switch is for you), then dial it down if needed.
In general, using a well-placed compensated (or exact) sum/dot product algorithm will solve most of your problems at a low cost. Nowadays, most people are familiar with Kahan summation (and related tricks like sorting numbers), but they do not know that it does not help much (often less than 128-bit floating points) and that the state of the art for those algorithms has moved quite a bit since. Nowadays, you can ensure that your sum/BLAS operation produces a result within one ulp of the exact result!
I highly recommend perusing the Handbook of Floating-Point Arithmetic (the Accurate Algorithm online page also seems fine [0]) to get an idea of what can be done. Implementations of these ideas can be found in most languages used for numerical computation (a quick Google search gave me options for C++/Python [1], Rust [2], etc.).
[0]: https://accurate-algorithms.readthedocs.io/en/latest/index.h...
A great thing in the Posit proposal is that they also consider a distinct accumulator type for things like sums and dot product. In my experience (I did a PhD on measuring floating point error in large numerical simulations) those operations are the ones that are most likely to sneak on you and need fixing. But most people don't realize that those accumulator type are actually easy to use with IEEE floats, they are entirely orthogonal to using Posits.
Note that the editor's presentation linked in HN only shows the belief plot (showing that psi researchers tend to be believe in psi), but the actual article draw most conclusions from their figure 2 (which studies correlation between belief in psi and what they call open-minded thinking)
The one exeption to this, for my taste, might be Earthsea, parts of it definitely fell short of its reputation as far as I am concerned.
[0]: https://en.wikipedia.org/wiki/Large_language_model#:~:text=t...
The downside is that you might lose information (the improved prompt might not be an exact semantic match with the original prompt), plus it becomes less and less useful as model's ability to understand commands improve.
Basically, if you express yourself clearly, betting on model improvement is enough, there is no need to add a faillible extra-step.
It is easier to get started with some instruments to give you feedback but it is not particularly hard.
Will they be deployed? They would make the OpenAI image model significantly more useful than the competition.