Ziplm: Gzip-Backed Language Model

It should be obvious that this is junk quality as a language model. But it's a cool example of the equivalence between compression codes and probability distributions, so I hope people find it interesting for that reason. You can also get a bit of an intuitive sense for the patterns that gzip and bzip2 pick up on in text (they like repetitive strings).

Of course, it doesn't really matter, as the whole thing is obviously just a toy, but I still think that this approach should be able to produce much better output than the garbled Moby Dick example.

"Google Replaces BERT Self-Attention with Fourier Transform: 92% Accuracy, 7 Times Faster on GPUs" (2021) https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-ba...

The next step of Conway's Game can be calculated with FFT and also 2D Convolution.

Convolution > Visual explanation: https://en.wikipedia.org/wiki/Convolution

[0] http://bactra.org/notebooks/nn-attention-and-transformers.ht...

Compression can also be used as "machine learning method": but all data vectors with the same class label into a separate bucket; any new, unseen and unlabeled data item can be added to each bucket. The most likely class is the one the bucket of which grows the least when compressed after adding it. The University of Waikato group (Ian Witten and co-workers) did a fair amount of that kind of work, perhaps first, e.g. Frank, Eibe, Chang Chui and Ian H. Witten (2000) "Text categorization using compression models", https://www.cs.waikato.ac.nz/~eibe/pubs/Frank_categorization... - yes, published 23 years ago!).

1. Schmidhuber's classical work applying a time-based Kolmogorov complexity to neural nets

https://pubmed.ncbi.nlm.nih.gov/12662875/

2. Applying a form of Kolmogorov complexity to learn formal languages using RNNs

https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00489...

3. Hinton and Van Camp - Keeping the neural networks simple by minimizing the description length of the weights

https://dl.acm.org/doi/10.1145/168304.168306

My colleague recently sent me this: https://arxiv.org/abs/2212.09410

"Less is More: Parameter-Free Text Classification with Gzip

"...We propose a non-parametric alternative to DNNs that's easy, light-weight and universal in text classification: a combination of a simple compressor like gzip with a k-nearest-neighbor classifier. Without any training, pre-training or fine-tuning, our method achieves results that are competitive with non-pretrained deep learning methods on six in-distributed datasets. It even outperforms BERT on all five OOD datasets, including four low-resource languages. Our method also performs particularly well in few-shot settings where labeled data are too scarce for DNNs to achieve a satisfying accuracy."

scipy.special.log_softmax(-code_lengthsself.conversion(1/temperature))

Your codes are K-ary but this doesn't look like its taken into account ala the README. What is the log(256) conversion factor? What is 1/temperature for?

This is a good example of how old methods can be pushed quite far if similar resources were devoted to them. Who knows, they might even posses advantages hitherto unmet due to a lack of exploring at larger scales.

That said, Transformers have a number of practical advantages. The learned projection matrices in attention lend Transformers a dynamic adaptability with respect to learned patterns that help make them programmable by their context, able to work out patterns present in context zero shot and on the fly. gzip based language models will be limited to their dictionary of patterns. The underlying vector space of neural language models also makes semantics more readily learnable (driving novel synthesis such as neologisms and more) while feed forward layers can learn a large range of computations.

http://bactra.org/notebooks/nn-attention-and-transformers.ht...