Honey, I shrunk the embeddings: Matryoshka vs. PCA | Hacker News Reader

Honey, I shrunk the embeddings: Matryoshka vs. PCA

dylancastillo.co

61 pointsdcastm18 comments

Good benchmarks!

> You can push this further by combining quantization with truncation or PCA. The resulting vectors can be dramatically smaller while still preserving a surprising amount of retrieval quality.

Counterintuitively - quantisation can also be combined with a random rotation step before the quantisation. A random rotation spreads information across more dimensions, allowing more aggressive quantisation without losing accuracy. Ironically - almost the opposite of a PCA.

I do wonder if relevant here though. It relies on the embeddings having "structure", i.e. that principal components point along basis vectors, which may not be the case with text embeddings.

Source: https://research.google/blog/turboquant-redefining-ai-effici...

in my experience it's very similar to what you typically need to do in order to get the best results in other fields (like audio), where being really particular/deliberate in how you manicure and format the input can pay orders-of-magnitude dividends

sometimes adding noise over a frequency range is better than removing it entirely, the opposite tends to be true for text and especially code, where you'll want lists of language and framework keywords, and then a pass on top of that to scan the codebase itself for its own slang. You can then 'double dip' and use these lists to weight the results after the fact

Nice! I’ve been working on something similar and found similar results.

In my experiments, I used lots of embedding models and the results were not nearly as uniform as this curve, just FYI. I didn’t use any of the API-based models though

I also wrote about this exact comparison when using PCA and MRL to quantize static models, see: https://stephantul.github.io/blog/mrl-pca/

Thank you! Will take a look at your results.

I couldn't find much when I first looked into this, which is why I ended up writing the article.

Ah I meant more to say that I was working on this as well. I haven’t published the results for this comparison specifically yet.

Did you find much difference in inference latency or throughput between baseline and PCA?

PCA is applied after the model, so there should be no difference in embedding throughput. Lookups in the index should be faster, but that speedup also applies equally to MRL.

So I guess the answer is: no

Thanks for doing this benchmarking Dylan. I wanted to teach people PCA in my original article, but had no idea it would stack up this well against Matroyshka!

Feels like a “just use logistic regression” moment :)

It does, tbh I was also surprised by the results!

Nice! Thank you for doing this. I am not surprised PCA holds up. It comes with some pretty strong theoretical guarantees. The problem with it has always been scalability. The O(n^3) complexity makes it infeasible to use for massive datasets.

That is where something like Matryoshka embeddings has appeal. You trade a little bit of performance for a guarantee of training + validation set coverage.

The original MRL paper (https://arxiv.org/pdf/2205.13147) reported an SVD baseline, which showed comparable performance (Table 1, top-1 accuracy) at d>=256, but much degraded performance at lower dims (d \in {8,32,64}). (Though Table 2, nearest-neighbor accuracy, doesn't show degradation until d <= 16.)

In your conclusion, you report that PCA won on most dimensions. Did you investigate why you found that PCA outperforms MRL when the original paper found that their SVD baseline did not?

I also found that interesting but our evaluation methodology is quite different, so I didn’t go too deep into it

Yeah...so I'm having difficulty conceptualizing your result. If you were publishing this I feel like it would be an important distinction to make.

Are the goals different, or should the original paper have done something more similar to your benchmark? Or something else?

PCA is good, but you could also try playing around with Sparse (robust) PCA. The sparsification loses orthogonality, but does not necessarily lose information, it can yield a different rotation and cleaner vectors. Now whether that matters in the context of LLMs/Embeddings - I cannot tell.

>a new problem emerged: how do you store and search all that information efficiently?

This is no way a new problem. The idea that embeddings or even search didn't exist until LLMs is simply false.

That's not a fair read of what I said

Well to me it's written in a way that suggests that users had no need for searching through their documents to find information. User's have had this need before vector search was used for ranking, so starting the story of how to search user's documents with vector databases is ignoring history.

That sentence is written as an introduction to an article about embeddings compression. So yeah, it does take a bit of a shortcut from "search" to "vector storage", but that's irrelevant to the rest of the article.

Reply on news.ycombinator.com