ChatLZMA – text generation from data compression

First, I came upon this reddit post [1] which describes being able to convert text into some ridiculous symbol soup that makes sense to ChatGPT.

Then, I considered the structure of my Typescript type files, ex [2], which are pretty straightforward and uniform, all things considered.

Playing around with the reddit compression prompt, I realized it performed poorly just passing in my type structures. So I made a simple script which essentially turned my types into a story.

Given a type definition:

    type IUserProfile {
        name: string;
        age: number;
    }

"IUserProfile contains: name which is a string; age which is a number; .... IUserProfiles contains: users which is an array of IUserProfile" and so on.

Passing this into the compression prompt was much more effective, and I ended up with a compressed version of my type system [3].

Regardless of the variability of the exercise, I can definitely say the prompt was able to generate some sensible components which more or less correctly implemented my type system when asked to, with some massaging. Not scalable, but interesting.

[1] https://www.reddit.com/r/ChatGPT/comments/12cvx9l/compressio...

[2] https://github.com/jcmccormick/wc/blob/c222aa577038fb55156b4...

[3] https://github.com/keybittech/wizapp/blob/f75e12dc3cc2da3a41...

https://gamefaqs.gamespot.com/pc/183289-disgaea-pc/faqs/2623...

and thought about a question I'd thought about for a while which is extracting facts from that sort of thing and one notable thing is that certain named entities appear over and over throughout the document (say "Cave of Ordeal") and how both attention and compression-based approaches can draw a line between those occurrences.

i post also this short video because correct few numbers who where miss calculated in the paper and show little less optimize and simpler implementations who still works well https://youtu.be/jkdWzvMOPuo?si=K1VRtJ5BtqREa2mz source code implementations of the video https://github.com/Sentdex/Simple-kNN-Gzip

This got me thinking about the possibility of building a series of simple context/token probability tables in SQLite and running the show that way. Assuming we don't require massive context windows, what would prevent this from working?

It's not like we need to touch every row in the database all at the same time or load everything into RAM. Prediction is just an iterative query over a basic table - You could have a simple key-value pair of context & the next most likely token for the given context. All manner of normalization and database trickery available for abuse here. Clearly a shitload of rows, but I've seen some 10TB+ databases still satisfy queries in seconds. You could even store additional statistics per token/context for online learning scenarios (aka query-time calculation of token probabilities). You could keep multiple tokenization schemes online at the same time and combine them with various weightings.

What would be more efficient/cheaper than this if we could make it fit? Wouldn't it be easier to iterate basic tables of data and some SQL queries than to trip over python ML toolchains and GPU drivers all day?

I'd be interested in how well you could finetune 3.5 to use different compression.

[0] - https://platform.openai.com/playground/p/hfLUBCTE8RrRYPIRxEe...

Instead of sampling strings character-by-character, this one adds random bytes to the compressed text and then decodes.

Turns out you can get a lot better text out of these simple models of you add some basic LLM features.

ChatGPT is already built with data compression, the training loss is cross entropy which means the explicit goal of the training is to compress the training dataset to the fewest bits.