Hacker News Reader

> until hardware FP8(FP4 anyone?) becomes more widely & cheaply available

Nvidia 40xx Tensor Cores support FP8 (but not the fancy async stuff on H100s). For some reason it's not used for models (AFAIK).

Some people report throttling issues in some cases, though.

I think PyTorch recently added FP8 & even Intel's Neural-Compressor has experimental support. But yeah most HF LLM models are currently just INT8 or FP4/NF4 loaded thanks to BitsandBytes by Tim Dettmers.

What are the trade-offs with INT8? I thought even the INT4 loss of accuracy was small and the INT8 loss almost unmeasurable.

Hey Akash would you mind elaborating about what you mean by contract mining?

Basically augmenting users parsing PDFs & looking to prefill values into Excel instead of typing it all out. Ex. Liabilities, time-period/frequencies mentioned, owners of clauses etc.