Hacker News Reader

Comment by krapht | Hacker News Reader

Physics simulations. There's a rule of thumb that to get an n-bit accurate result after a long chain of calculations, intermediate results should be stored with 2n bits. Often using the full dynamic range of a float is necessary because the magnitude of different physical phenomena varies so wildly.

I guess people do store intermediate results in floats in order to take advantage of GPU acceleration. However, once you do that, you have to be careful in your programming and pay a lot more attention to underflow, overflow, and numerical accuracy. People who write scientific software usually also aren't experts in numerical analysis. Even if the code you write reliably works with floats, the library you use might not be. It's just a huge pain to make sure everything is accurate.

View on HN

After moving from theoretical high energy physics to data science I m really happy I don't have to care about numerical precision on my computations.

The problem is that numerical errors when solving partial differential equations not only propagate but increase in magnitude during the propagation. If you are not careful you will end up with a 100% wrong answer at the end of a big computation.

I've always argued that if you are getting close to having to worry about underflow, overflow etc. then you have an ill-conditioned problem and just increasing the size of your intermediate results won't help you a huge amount because you need more precision from your inputs. There are very few fields where you need more than the 7 decimal digits afforded by floats. Maybe the only exceptions are in astrophysics.

When solving linear systems of equations (which arise pretty much everywhere) Krylov subspace methods are usually quite effective because the Krylov subspace is orthonormal.

If your floating-point precision isn't high enough, you'll end up instead with a "subspace" that isn't spanned by orthogonal vectors, and the consequences of this are pretty drastic (requiring n-times the number of iterations to solve the system, you'll never find a solution, etc.). These all happen even if your system of equations has a good condition number, you just need to make the precision low enough.

This is why most people use double precision, and some people use quad precision. For many systems, e.g., if you are using CG as your solver, quad precision can cut the number of iterations by a large factor (2x-4x). These problems are still bandwidth bound, and using quad precision duplicates your memory bandwidth requirements, but if it reduces the number of iterations by 4x, you just halved your time to solution.

Lattice QCD, especially near the physical point, has poorly-conditioned matrices that one wishes to solve Mx=b for. The state-of-the-art is that the [sparse] matrices can be as large as 4×3×(128^3 × 192) ~ 5e9 on a side. It's not so rare to find legitimately difficult problems in hard sciences.

Double precision has been the mainstay of scientific computing for decades, and no, it's not because all those scientists are dumb.

Pretty much any ODE solver you want double. All of CFD (fluid sims) and FEM (mechanics) use double. It can also be a big help in graphics. Basically, if you're doing fine meshing, doubles will help you. There's a balance between your meshing size as well. Your life is a whole lot easier and you'll get better answers if you just use double. Ask literally any computational {physicist,scientist,engineer}. We've all tried without double. Trust me, I'd love to have lower memory constraints.

That argument would be wrong. Double can be a big help, for example when performing solid modelling operations on triangle meshes. Best would be actually exact arithmetic, but it being often too slow, Double is often good enough, while Float isn't.

I thought for this the standard practice is fixed-point. Requires more planning and mental gymnastics but usually much faster and gives you full control of the precision.

Maybe this has changed from my DSP days.

I am confused. If you used to work in DSP then you know the standard practice involves MATLAB, where the default is double precision. This is also the default datatype in NumPy. Most engineers don't like working in fixed-point unless they have to for other reasons, like moving it onto an FPGA or something.

The person you're responding to probably worked with DSP chips, which are generally not floating-point. e.g. Motorola 56000, TigerSHARC, Blackfin.

Why not? Because of interop with other libs/tools? It's not _that_ hard, but I can see the problem if whole workflow isn't like that.

You know a cool thing you can do to help fixed point interop between operations in a complex system where you absolutely don't want to accidentally overflow anywhere? You can tack on some bits to the number to control an overall scale of the fixed point number. Let's call it an exponent ;)

Agreed. Let's standardize it :))

most of it was FPGA work but in some cases DSP processors. Even when you have floating point support it is much slower than if you use fixed point. As for MATLAB, modeled plenty of filters in it for fixed-point math.

Reply on news.ycombinator.com