Hacker News Reader

Probably something similar to https://arxiv.org/abs/2310.07707

Just PCA the thing on a couple billion embeddings, store those coefficients (only ~3000 floats) and in the API, report the n components with the highest variance/eigenvalue?

Probably as simple as training the smaller model to approximate the larger model. Well studied and done via tinylm and minilm.

Honestly the answers to these questions are usually the most obvious thing. It could just be some basic dimensionality reduction technique precomputed for each input/output size combination.

That’s still just a few thousand matrices. I’m sure they can handle the training and distribution of that set.