Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by Havoc | Hacker News Reader

Full thread

Havoc·

I'm guessing by lossless they mean something other than what the word usually means in compression context?

>achieving near information-optimal compression without any loss of precision

So perhaps more lossless as in didn't lose perplexity/benchmarks?

In my mind lossless is precisely zero bits lost along the way.

View on HN

The first sentence of the introduction ends with "we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM size by 30% while preserving outputs that are bit-for-bit identical to the original model" so yes it's lossless.

information-optimal compression is "the theoretical minimum number of bits needed to represent data without losing any information, based on the data's entropy", so I think they mean the same thing you do

Yeah, they’re saying that this compression is almost as good as is theoretically possible without losing any information.

A good example that information, i.e. bits, are only meaningful with respect to an end. If you don't know what the bits in a float will be used to, you can't throw them away, but if the floats are in a function, and you know that what some bits are can't affect the output of the function regardless of input, then you can throw those bits away and still have a lossless compression of the function.

Think Morse code, where frequently used letters have shorter codes than less frequent ones. This ensures zero loss of information.

The part you quote is a few sentences past the sentence that says "preserving outputs that are bit-for-bit identical to the original model".

Reply on news.ycombinator.com