Hacker News Reader

bfloat16 is stable enough for ML training, which is what TPUs exist for

Be careful: 16 bits was quite a lot of colors around 20 years ago. Now we would laugh at it. Ditto for 640Kb of RAM (who would need more?) etc.

Not trying to be dismissive just saying that... computational limits are limits on what can be done, in the end.

30 years ago. The time flies...

well, you are right and I grow old, I grow old, I shall wear the bottoms of my trousers rolled…

I stand corrected.

Wrong way to think about the problem. Lowering bits does not mean lowering model capacity. Its the opposite in fact - it allows you to you to fit more parameters.

Well, yes within your constraints. In the end, you are choosing between two aspects. Same as with screens: you could have resolution (1024x768!!) or color (16bits!).

Edit: the term I could not remember is tradeoff.

kind of. if we get computers that are 1000x faster, it just becomes a tradeoff between higher precision or 1000x more parameters. the reason resolution has stopped being pushed is that our eyes have severe diminishing returns. it's not yet known whether brains do.

Not always though!

Reply on news.ycombinator.com