Hacker News Reader

I guess the naive approach, if we wanted to do a quick lossy matrix multipy, would be to take the truncated SVD and use that. How does this library compare to the boring strategy, I wonder?

View on HN

We found sparse, truncated PCA to be the most competitive baseline. We beat it by a lot (see the paper [1]), but the other big drawback is that trading off the rank vs sparsity was an ugly hyperparameter tuning problem. By ugly, I mean that the results were really sensitive to getting this right, it wasn't easy to set a priori, and took a while to iterate on because the sparse PCA trained much more slowly than any other practical alternative.

There are situations where PCA/SVD is the right approach though. Namely, if you need really little error, our method often can't do that, whereas throwing away dims that explain almost no variance can. Also it's just easier to implement.

[1] https://arxiv.org/pdf/2106.10860.pdf

SVD performs poorly for compression in terms of accuracy / compression ratio. As the Bolt paper said, for that, product quantization is the way to go. However, the accuracy trade-off is still pretty big, that's why it is mostly used for coarse similarity recalls (such as faiss or scann).

Reply on news.ycombinator.com