Can something like this be used in training as a way of temporarily 'compressing' parameter count?
[1] https://arxiv.org/abs/1807.04629
I'd be specifically curious about applying PQ to transformers. It's quite depressing to me that the ultra-large-scale model training is inaccessible to the average poor person like me. The dream would be to figure out some method of compressing the parameter count significantly (or rather make models more efficient) such that it is possible to train and/or run 100 billion/1/10/100trillion+ parameter models on Colab say, as 'crazy' as that sounds to most probably.
In speech recognition a sub-problem is to figure out how to encode the conditional distribution P(features|phone), where features is an n dimensional vector, say n in the hundreds, and there are about 50 possible values of phone. The simplest way to encode P(features|phone) would be as a lookup table -- but that is impractical for n > 2. The authors describe two ways to proceed:
Option A: discretise the feature space -- apply vector quantisation, to partition the feature space to e.g. 256 regions. Replace each feature vector with a region label 1...256. Then the distribution P(features|phone) can be approximated as P(label|phone) and stored as a lookup table.
Option B: Don't discretise the feature space. Instead, approximate the distribution P(features|phone) by a mixture of k n-dimensional Gaussians, at the cost of storing k(n + n^2) parameters for the mean and covariance matrix of each Gaussian in the mixture.
Russel and Norvig state "vector quantization is no longer popular in large-scale systems" (this is from my ~2003 second edition of the book, in the context of speech recognition).
Can anyone who has more familiarity with the two applications explain why vector quantisation fell out of favour in speech recognition vs mixtures of Gaussians, and why vector quantisation is used in document/image similarity search? Are there approaches to document/image similarity search using continuous distributions that are competitive with vector quantisation approaches?
Or are these two problems fundamentally quite different as the speech subproblem is trying to infer an state out of a very small state space (50 phones) while document search may be trying to infer a state out of a larger state space (millions of documents).