Hacker News Reader

Comment by TorKlingberg | Hacker News Reader

I feel like floating point is a completely separate branch of programming that a lot of coders never use. When they do it's often a mistake, like for currency. Yet floating point is very useful for scientific calculations and simulations, which is what computers were all about for the first couple of decades.

I have a suspicion that on a fundamental level, floating point isn't actually good for games or machine learning. They're just used because existing computers are so good at floating point number crunching, especially GPUs.

View on HN

> When they do it's often a mistake, like for currency.

The most beautiful FP bug I remember was a denial of service in some webservers where by setting the header "Accepted Language: en-gb;q=0.3 en;q=0.8 ..." to a specific value you could send the official Java floating-point parser in an infinite loop (and this affected several Java webservers).

So at each webpage request, you were sending one CPU core of the webserver basically busy-looping. Hence crashing the webserver in a few requests at most.

Now it'd be heresy if I were to say that maybe, just maybe, had the standards mandated to use 0 to 100 for weighting instead of 0 to 1.0 we could have dodged a whole source of potential issues!?

No, no, I realize this is heresy: let's all keep using numbers that cannot even be represented correctly (except as strings), parse those strings into something approximately representing what's written in the string, then make more approximation errors while doing computation with these numbers, propagation errors, epsilon estimation errors, and keep insisting we all could have authored "What every computer scientist should known about floating-point numbers" (which is 80 pages long and, well, approaching a treaty) ; )

/rant off

Meh, it's not like there can't be a bug in whatever other parsing code a webserver uses.

Maybe using a float is overkill in this case, since you're never going to hit more than a hundred languages. But it's at least not setting any limitations.

The issue is that the representation of floating point numbers imposes additional cognitive burden that the code must handle. How does the web server header parsing code handle positive/negative infinity? How about NaN?

NaNs/infs are particularly vicious, as they propagate through computations.

This. Floating point numbers, as they are lossy attempts to discretize continuous data, are more complex and usually non-deterministic. They should never be chosen when a non-lossy representation is available for the problem domain.

laughs in Ada

    -- IEEE754/64-bits: Interfaces.IEEE_Float_64
    -- And here we strip out non-numeric values.
    Use Interfaces;
    Subtype Numeric_Real is IEEE_Float_64 range IEEE_Float_64'Range;

Done.

Based on the OP, this was the official parser for floats. That's a bit more extravagant of a bug.

I also would invite you to find a similar instance where a parsing bug for integers crashes the server. Not throws an exception, but falls over in an uncatchable way. I'm not sure I have ever seen one.

No, it is setting limitations, they're just harder to understand and predict. Here's what the RFC says:

  qvalue         = ( "0" [ "." 0*3DIGIT ] )
                 | ( "1" [ "." 0*3("0") ] )

(https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3....)

There are 1001 valid values with 1117 representations. Only in practice, Firefox clamps to 2 decimal places, and it used to clamp to 1 decimal place (https://bugzilla.mozilla.org/show_bug.cgi?id=672448), and who knows what other software does.

And it gets worse: the grandparent suggests there are servers that parse it as floating point. Do they accept q=1e2? If they accept it, are there clients that send it? Do you need to be compatible with those clients?

Don't blame the protocol, blame the language. Nothing prevents the use of fixed point for such numbers. It's just the Java ecosystem and the weakness of languages that use pervasive exceptions to the point that nobody can keep track of them all that is the cause of such issues.

> Don't blame the protocol, blame the language. Nothing prevents the use of fixed point for such numbers.

I absolutely blame the protocol. Fixed point isn't widely available and used, but integers are. Someone below says protocol spec goes to three digits past the decimal. Using a constrained integer would have allowed the same range of values, but without the complexity of decimal numbers; that's a protocol mistake.

In addition to the ease of using floating point numbers to incorrectly represent decimals, there's also fun with formatting, comma vs period, number of digits to report, etc.

There is nothing wrong with the protocol. The protocol has nothing to do with what representation is used by software that speaks the protocol. Software could just as well parse those values as integers. Just drop the decimal point and pad with zeroes. Or use a decimal type (but there is really no need to).

The protocol has an obvious implementation choice and some non-obvious alternatives.

If the protocol had nothing to do with it, I don't think you'd see the same bug come up in as many implementations

Assigning blame is a bit academic.

I think a protocol that used integers instead of decimal point would have led to more correct implementations. So it would be a better protocol.

It’s on implementors to be correct. No one said the Java implementation needed to use float or double. Java isn’t the only web server implementation language.

Probably https://www.exploringbinary.com/java-hangs-when-converting-2...: “Java — both its runtime and compiler — go into an infinite loop when converting the decimal number 2.2250738585072012e-308 to double-precision binary floating-point. This number is supposed to convert to 0x1p-1022, which is DBL_MIN; instead, Java gets stuck, oscillating between 0x1p-1022 and 0x0.fffffffffffffp-1022, the largest subnormal double-precision floating-point number.”

As someone who has started out making games on Playstation 1 that didn't have floating point hardware, I can with authority say that developing games with only integers suck. Floating point number are far easier and robust to use.

Only integers? Wow. Were int pairs used to represent numbers with decimals or what?

To use only integers, you generally try to express all numbers in units of the smallest quantity. So any money is in integer cents, any distance is in integer (spatial resolution size). It gets annoying with derived units: Time can be stored in integer frames, but then you'll need a rational type of some kind to properly handle velocity. Or you just round to the nearest (spatial resolution) per frame and hope the errors don't look too weird on the other side.

In signal processing on constrained domains (constrained including the lack of an FPU), the usual alternative to floating point is "fixed point" arithmethic.

Basically you use the platform's native types (e.g. uint32) and decide which of the bits are for the integer part and which are for the fractional. uint32 can be interpreted as 20/12 for example. Your register went from representing "units" to representing "2^-12 increments of an unit". The ALU can do sum/comparison/subtractions transparently, but multiplications and (god forbid) divisions require shifting to return to the original representation.

The choice of how to split the bits is a tradeoff between range and precision. It can vary from one routine to another. It's a pain to compose code with it.

Short story: native float types are a blessing for programmer productivity and library interoperability.

The Wikipedia page explains the operations: https://en.wikipedia.org/wiki/Q_(number_format)

Let's say you have two 4 bits integer to form an 8 bit decimal number:

1111.1111 is (1 + 2 + 4 + 8) + (1/2 + 1/4 + 1/8 + 1/16)

The GPU made use of fixed point numbers. So you have a set number of decimal places for a number.

Isn't that what caused the PS1 graphics "wobble"? I remember you could just stare at a wall, rotate the camera, and the textures would slightly jitter and move. It's really obvious in some the Metal Gear cutscenes, especially this conversation with Donald Anderson where the camera pans up from the floor (at 15m) or the sweep(15m31s):

https://www.youtube.com/watch?v=VLTqIXZ1jQQ#t=15m00s

I never had a PS1 but it always bugged me when playing on someone elses.

> I never had a PS1 but it always bugged me when playing on someone elses.

Conversely I like the fact a lot that it does. And some others that probably have fond memories of PS1 too recreated this on modern computers using shaders.

https://github.com/dsoft20/psx_retroshader

This is because of two things: the affine texture mapping was incorrect, and there was no sub-pixel precision. The sub-pixel part is kinda float-related (though you could make use of fixed point too).

yes, and the PS1 had a vector unit built in to the CPU that was used for world transform and it was only 24 bits.

Early PS1 games fit the entire world in to a 24 bit space, and you can tell that there is steping if you drive very slow in a game like Ridge Racer. Later games moved the world around in the coordinate system to manage precision.

Another reason why the graphics looks extra "popy" on PS1 (And early DirectX games) is that vertices where computed at a pixel accuracy. OpenGL, Doom, Quake and more modern hardware have sub pixel accuracy.

No. The DS uses only fixed point numbers and does not wobble.

Did you use macros/operator overloading for all that bit shifting or just hardcoded it everywhere?

We made a simple 3d demo for uni using fixed math (and we were clueless then and weren't using macros because sometimes you could refactor some operations out that way and we wanted all the speed). There were lots of bugs caused by this.

Can't imagine how painful writing a whole game this way would be.

Floats are essential for machine learning, scientific computing, and any other application where the goal is to model mathematical processes involving real numbers. The idea that there is something out there that could do the job better than floats is almost certainly wrong.

I'm fascinated by how common it is for programmers to hate floats. Yes, if you write business software for a living you may not have much use for them. But there's a lot more to computing than business software.

> Floats are essential for

Not necessarily true. Theoretically you can replace floats by fixed-point representations given enough bits, and in practice this also works often.

It's a pity languages/hardware don't have good built-in support for fixed-point numbers though.

Well, as mentioned elsewhere, because of historical purposes, GPUs tend to optimize floating point operations really heavily. Whereas bigint style things likely don't benefit, in part because there's no "standard" for them at that low a level. So it seems to me to be a bit of chicken and egg; low level languages don't have them as part of the standard, so graphic cards can't universally optimize for them, and since graphic cards don't support optimizations for them, there isn't anyone clamoring for it.

Okay, so now, relative to floating point arithmetic, you've sacrificed an enormous amount of dynamic range and have to worry about overflow and underflow much more than before.

What did you gain from this, exactly?

I'm sure there are some special situations where fixed point is better, but for general purpose scientific and technical computing, floating point is obviously the right choice.

On some low power architectures without FPUs it is significantly more power/compute efficient to use fixed point arithmetic. But that's really it.

Fixed point is obsolete on the vast majority of compute devices today. It is only a reasonable alternative in ultra low power scenarios or on tech stacks that haven't migrated to modern architectures, like some DSPs (which are also really only applicable in ultra low power devices).

On modern CPUs, fixed point is slower and error prone (both in usage and the computations themselves) relative to floating point.

I don't think it's a pity that fixed point support is out the door. It sucks. Floats minus subnormals are the least surprising numerical representation out there, and the most versatile.

Do you really want to worry about whether your numbers are saturating or wrapping around? Keeping track of what your max/min are? Tracking decimal places? Implementing division by hand? Screw all that. Fixed point is awful and is essentially an academic exercise today.

For some reason many programmers love to hate technologies that are old.

Hate floats, love big decimals

Hate OOP, love functional

Hate RDBMS, love nosql

Hate html web pages, love SPA

In my opinion it's a confluence of influences.

1) the same social media attitudes poisoning society generally. "I've got an opinion, and it's worth just as much as yours". News flash - opinions are like arseholes, everyone has one.

2) Inexperience - devs who have only ever built chat apps and don't understand why relationships hence RDBMS are common and useful in line of business applications. Devs who have not tried to capture the water level in a tank or the voltage from a solar panel and don't get why floats are useful.

3) resume padding. nuff said

Not disputing your other points, but functional programming is by most measures older than OOP. The FP hype now is definitely more a matter of rediscovering the wisdom of the past, and I think that's a good thing.

That's an excellent point.

Although to go down that road, schemaless databases preceded RDBMS by decades, and then were replaced for good reasons.

One example where floating point does not match the problem field is GIS. Coordinates are bound by the earth's circumference. Fixed point would make much more sense, but neither hardware nor software support is there.

I mean most GIS software is pretty aware of precision models. Take GEOS for example, it's got a variety of ways to specify the precision in number of decimal places for a geometry. It still uses doubles underneath, but that's to avoid unneeded code complexity.

Lots of the math in games and ML assumes you’re operating in the reals. For example, the optimization in neural networks. Integers and fixed point won’t work.

Floats have a totally different algebra than the reals, though.

Only in the strictly mathematical sense. To a first approximation, floating-point numbers behave like real numbers, and that's good enough for many use-cases (not all, though).

Similarly, fixed-width integers have a totally different algebra than true integers, and yet they're immensely useful.

fixed-width integers have exactly the same semantics as true integers bar overflow, which is a very easy to understand concept.

Floating-point numbers behave like a physicist would do calculations rounding at every step, but doing everything in binary. Or hexadecimal if you prefer, it's equivalent, but still difficult to get your head around to. Then there additionally is + and - 0 and + and - infinity, and several settings for how rounding is done exactly.

"To a first approximation, floating-point numbers behave like real numbers"

This is incorrect. To count as an approximation, there have to be some accuracy bounds. This is impossible to define, as the reals don't have the same cardinality as any floating point number system.

Now, for many interesting and useful real-valued functions, a float-valued function can be defined that is an approximation. But there is no general way to map a function defined on the reals to a function defined on the floats with a known accuracy.

Yeah, but closer to the reals than fixed point or integers. That’s kind of the point of the standard.

Doesn't ML work fine with fixed point numbers? I don't think there is a particular use case for very large or very small numbers.

Referring to ML is so vague that it’s not particularly meaningful. The answer in general is no. There’s been some work showing neural networks can function with low precision floating point, but that’s still pretty different than fixed point.

ML optimization algorithms generally do require proper handling of very,very small numbers for e.g. calculation of gradients.

There's a bunch of work on discretization of models after you've done optimizing them - that works, you can get high-performance inference with low-bit fixed point numbers; but that's after the learning has been done using proper floating point (however, you don't necessarily need high accuracy floating point, e.g. single precision may work betted than double precision simply because it's twice less bytes to copy over).

It's the exact opposite. It really loves large and small numbers, the precision of a number doesn't matter that much

That's the reason why they introduced https://en.wikipedia.org/wiki/Bfloat16_floating-point_format

It's similar to normal half precision float but the exponent is the same as with 32bit float and mantissa has been sacrificed to get room for it.

In javascript, every number is technically floating point. :)

What better alternative is there for handling currencies? Multiplying the original amount by 100 and casting to an int? ($3.50 -> 350)?

Use ints. No multiplication necessary. Everything is done in cents. You add/remove the decimal point when doing string conversions.

Use decimal type for that

It is the opposite.

From Microsoft BASIC to Javascript, many programmers have worked in languages that only have floats. Financial calculations involve exponential/log math much like scientific problems. There are issues with rounding and representation there (e.g. there is no such thing as $0.01 in floating point, only $0.50, $0.25, $0.125 and some sum of 1/2 fractions that comes very close to $0.01 and even reads in and writes out as $0.01 even though it isn't.)

> very useful for scientific calculations and simulations

Don't forget control systems.

In some game engines, Fixed Point Arithmetic is used instead of Floating Point, because it's faster, and the approximation is generally good enough for a game.

I dont think this is true. Integer math is used in some game engines because they need to be deterministic, between networked computers (and different CPUs rounds floating point numbers differently). I have written an engine like that, and i know that Starcraft 2 is all integer for the same reason. No one does it because its faster or easier. Its a pain.

I never said it was easier and I never said ALL game engines.

And computing integers IS faster than computing floats, at least on a CPU.

>And computing integers IS faster than computing floats, at least on a CPU.

In the abstract yes. In reality not so much. A lot of what floats are used for in games is vector math (you are often in a 2D/3D world), and vector math in fixed point requires a lot more work in integer math since you constantly need to shift down things in order to avoid overflow. Overflow bugs are a constant problem when doing dot products in fixed point integer math.

Another common operation in vector math is square root (to normalize vectors) Modern CPU's do that very fast, and you can also use clever approximations that use the way floating point's are represented in hardware.

Most of whats done in 32 bit floats in games, needs to be done with 64bit integers to manage these problems if you decide to use integers, and that means that your data grows, more cache misses, and things slow down.

On top of this everything you do in a game needs to be drawn on the GPU so you need to convert everything to floats anyway to show them on screen, and converting between integers and floats is also slow.

I never use the square root in games, I always compare squared distances (since the properties are the same).

If you try to do with Fixed Point Arithmetic what was intended to be done with Floating Point, you're the problem.

A 32 bits integer can hold values up to 4 billions, if I have that kind of value in a simple game, then yes i will switch to Floating Point Arithmetic, but when does that use case happen if you're not writing a physic simulation game ?

It appears you have a very niche definition of "game", one which excludes basically all 3D games.

Yes, that's exactly why I wrote:

> some game engines

and

> simple games

A 3D game is not a simple game.

Ugh, do you really have to be this nitpicky? Fine, you wrote:

> the approximation is generally good enough for a game

No, it's not generally good enough for a game – only for some games, and not good enough for any game with 3D graphics.

Speaking of nitpicky, I thought the gamecube's GPU was integer-driven? And it has plenty of 3D games.

Technically, 32-bit fixed-point numbers have 8-bit more precision compared to 32-bit floating-point numbers, but the real problem is it's quite cumbersome to fully utilize the precision offered by fixed-point numbers as you'd have to keep in mind their numerical range all the time.

> A 32 bits integer can hold values up to 4 billions, if I have that kind of value in a simple game, then yes i will switch to Floating Point Arithmetic, but when does that use case happen if you're not writing a physic simulation game ?

All the time. Let me give you an example from a game I made. During early R&D I used 32 bit numbers and a fixed point of 11 bits. That means that a normalized vector is between 1024 and -1023. You can do 2 dot products between vectors before it breaks it breaks. (10 + 10 + 10 bits plus one sign bit). That means that you have to do a lot of shifting down. the world can only be 20 bits large, because you need to be able to multiply world coordinates with vectors without getting overflow.

20 bis is very low resolution for a real-time world. You get problems with things not being able to move slow enough at high frame rates. (this was in 2D). Switching to 64 bit was the right move.

The biggest problem with fixed point is reciprocals behave very poorly; ie. x·(1/x)=1 and 1/(1/x)=x fail very badly. In order for these to work you obviously need the same amount of numbers above 1 as below 1, like in floating point.

Using fixed point with 11 bits after the decimal, with numbers as low as 100 (18 bits) we already have errors as large as 100·(1/100)=0.977.

Modern CPUs, like Zen2, have vastly more f64 multiplier throughput than 64 bit integer multiplier throughput.

This goes so far, that Intel added an AVX512 instruction to expose this 52/53 bit multiplier for integer operations, to accelerate bigint cryptography calculations like RSA and ECC.

It depends on which 0 you are talking about. The 1-1=0 or the 1/infinity zero. Somewhere deep in a physics department there is an equation where the distinction either confirms or denies the multiverse.

Which modern general-purpose CPUs do not follow IEEE 754? I don't think what you're saying is true.

They all follow IEEE 754. But that defines the representation of the bits in a floating point number, not what happens when you operate on them. Different implementations do rounding differently, and there are arguments about how to do this correctly. There can be multiple ways to represent the same number even (this is a reason not to ever do a == compare on floating point number that have been operated on). Not only have there been differences between CPU makers, but also between different CPU of the same vendor.

A separate, complicating factor is that floating point math often don't yield the same results with different levels of optimization turned on. The reason to use integer math for games is because you want it to be deterministic so can be an issue.

IEEE 754 does also specify operations and their behavior. E.g. section 5.1 (from 2008 edition):

> All conforming implementations of this standard shall provide the operations listed in this clause for all supported arithmetic formats, except as stated below. Each of the computational operations that return a numeric result specified by this standard shall be performed as if it first produced an intermediate result correct to infinite precision and with unbounded range, and then rounded that intermediate result, if necessary, to fit in the destination’s format (see 4 and 7).

Well the fact is that different hardware rounds it differently, so its the world we live in. My guess is that actually implementing something in hardware " to infinite precision and with unbounded range" might not be realistic.

Starcraft II uses floats, but for the game simulation portion it uses an internal software floating-point library to ensure consistency.

You are right, all the graphics are full of floats. Writing "all" was a bit of an overreach.

Actually, the rounding for basic add/multiply are strictly defined and configurable (some algorithms need other rounding rules than the default "round to nearest even").

If you only use those (maybe including division, I'm not sure), and don't rely on hardware support for sqrt, sin/cos/tan, approximate-inverse, etc., you can totally use them deterministically between different architectures.

It might be a little faster for addition and subtraction in synthetic benchmarks -- and with the use of pipelining might end up being slower overall -- but multiplication and division are considerably faster with floating point.

division, yes, but multiplication is faster, but it's not huge. The limiting factor for multiplication is that it's O(N^2) in multiplied digits; so a 32-bit fixed point has 32x32 and IEEE fp32 multiplication has 23x23.

That's assuming that you're implementing multiplication in software though, right? Since CPUs have dedicated multipliers, aren't both around a single cycle?

They are not in general (iirc 64b mult is down to 3 cycles now), but usually in an x86 cpu it can run microinstructions out of order enough so that the multiplication looks like it's finished in one cycle.

Reply on news.ycombinator.com