Hacker News Reader

Comment by H8crilA | Hacker News Reader

Why? Google has certainly researched their floats before committing an entire line of silicon chips. It's easy to just enumerate all possible float16 configurations in a simulator to see which one performs best on a wide range of neural network applications. Then pick the best one. Big data driven organizations do this all the time (brute force through an entire line of solutions, pick best results).

View on HN

Also, a format that's just a slight tweak on IEEE-754 is going to be way easier to implement on existing hardware than a wholly new format.

bfloat16 is "best" primarily because it has the same exponent range as float32. That makes it easy to port models that have been developed using float32 to bfloat16. (As opposed to using int8 or float16, both of which have a smaller exponent range.)

It's possible that some other custom format is better in absolute terms for models trained specifically for the custom format. But for the current ecosystem, where models are trained primarily using float32, bfloat16 is a very good choice.

I know nothing about ASIC or CPU simulators but I suspect that it's not as easy as you make it sound: for machine-learning related tasks, performance doesn't only come from raw compute numbers: you'll also want to model the actual data movement costs across the caches hierarchy and registers. Because a lot of time training is not necessarily compute-bound: the relative cost of data transfer (VS compute) can be quite high, or even dominate.

fp16 and bfloat16 are the same size, so there's no difference in data transfer rates.

You can also implement them both on a FPGA and take performance numbers that way.

Reply on news.ycombinator.com