Hacker News Reader

Comment by Zagreus2142 | Hacker News Reader

``` We evaluated several precision pairings across our internal retrieval benchmark suite. Scores are NDCG@10 averaged across the suite, scaled to 0–100. NDCG@10 (Normalized Discounted Cumulative Gain at rank 10) measures how well the top 10 results are ordered against the ideal ranking, rewarding relevant documents more when they appear higher, with 100 being a perfect ranking. The full-precision baseline averages 90.26. Int8 query against binary documents averages 89.65, a 0.61 point drop, while reducing document-vector storage by 32x ```

Saying "Near lossless" to mean 90% accurate retrieval of saved vectors is simply a lie. Lossy-ness is binary, not something you can paper over with getting close enough. And 90% is not close. Sure, LLMs are all about gradient descent on noisy data sets so I guess this is acceptable in this field but that terminology usage still bothered me

View on HN

I don't believe that's what they were saying at all though. The claim appears to be that it's near lossless relative to their own baseline that uses float. Which I'd grant, since a 32x storage reduction for 0.61% loss in quality is a reasonable trade off when you've already decided to accept that ~90% is "good enough".

near lossless refers to being 89.65/90.26 = 99.32% of baseline, i'm pretty sure.

yes exactly.

Yeah, what bugs me about stuff like that is like they spend all this time and then they output several or minimal real testing to prove the theory It's like you're building your model to And just because it takes a long time to compute and do the testing, you'd rather publish your article and then try to get credit on something that hasn't really been proven. Look, prove your results. Study it. Ruggedize it. Make sure it works. Then, show us.

>Saying "Near lossless" to mean 90% accurate retrieval of saved vectors is simply a lie. Lossy-ness is binary, not something you can paper over with getting close enough.

Lossy-ness is a binary, "near lossness" however is still valid (and is not the same as saying lossless).

How else, if not by comparing to "lossness" (whether with a more abstract qualitative term like "near" or with some distance or error measurement) do you report the level of fidelity to non-lossy results?

Depends on the context, but even in abstract for whatever domain, 90% sounds pretty close if we're talking about a linear level of degradation corresponding to each X% level.

In this case if this is calculated on a lossless baseline that's itself close to 90% - it's distance from it, it doesn't represent distance from some pure 100% perfect retrieval. So ~90% vs ~89% is very close to lossless capability.

Ask a SOTA LLM when Newton was born without any access to internet : the answer is Lossless for our shared culture understanding of this question. Not Near-lossless, lossless. Ask the same LLM when YOU were born, the answer is just wrong for almost anyone in the world, not lossy. Between the two there is a whole new field of Lossyness to study.

90% depends entirely on what the measure means here, do you understand what "Normalized Discounted Cumulative Gain at rank 10" means to the set of data that we are comparing ?

Sometimes coming up with new codecs (compressors decompressors) means coming up with new ways to interpret artifacts of the real world. And this is exactly why LLM are so powerful and they are like a giant Lossy (but Near-Lossless for various use cases) ZIP file / Database of the whole knowledge of the training data.

Nobody is trying to manipulate you here, humanity just has to find new explanations for complex topics.

    Lossy-ness is binary

Lossless is binary in pure information theory. to quote my other comment :

Lossless is objective for information theory. To get from the real world to digital world you need an analog to digital converter, this process is by definition lossy. We are interested in the real world, and information is pure but never represents exactly reality. Lossyness is baked into our problem statement here.

Using terms like near lossless means we think we are very close to reality for what we’re trying to do

Reply on news.ycombinator.com