spidy__

yep!!

Ummm I don't have a lot of idea about crypto but in compression the idea was that you just share the model along with the metadata file to the other user and they de compress it, not sure how things will differ in crypto

If I were only using a transformer that would have been true, but we use arithmetic coding alongside our transformer to fix those mistakes (layman terms). You can read about arithmetic coding, its a pretty cool topic.

Ohh that "what I tried that didn't work" was a section that I specifically wrote myself (and then polished with AI) because I wanted to document what are the different approaches I tried to compress more but failed.

Also thanks for the reference looks like a interesting read.

I mean neural compressors provide great compression, BUTT the issue is they are really slow like in my project it takes around 45 minutes for de compression of 100 mb so I doubt if it would be useful, also using a transformer in user's browser sounds like a heavy task.

I tried with zstd 19

Ofcourse, am doing this just because I enjoy it.

While we are on the discussion I have mentioned a question at the end of the discussion around an assumption am trying, can you please check it out and see if you have any suggestions?

Would be awesome if someone can validate or help.

Ohh yeah , I took it from Project Nayuki as mentioned in the file as well, i tried to pip install it but there were some issues so just took the file and kept the copy right as it is.

Its not an issue is it? I am not sure.

Oh, sounds interesting. I hadn't considered using a diffusion model for this. My current approach generates the document byte by byte with an autoregressive transformer, so I'm curious how a diffusion model would improve memorization or reconstruction quality.

Can you point me to something that i can read? I really wanna try this approach , diffusion model does sounds interesting for compression.

1. I wrote the content as what i want to mention in the documentation and just used AI to polish it so that its easy to understand, is it hard to understand the documentation right now?

2. Have added the link for downloading both the enwik9 slice and the nyc dataset. Apologies I forgot to add it.

You can get it from here - https://github.com/samyak112/pym-particles/blob/main/README....

3. Other than zip i tested it with zstd19, and now that you mentioned LZMA2 and BZIP2

I got results on enwik9 100mb slice as

zstd - 28mb bzip2 - 30mb lzma2 - 26mb

I will mention these and results from ZPAQ in the readme for both files, thanks for pointing them out!!!

But the thing is this neural compression approach cant be used right now, as it takes hours to compress and de compress a 100mb file so not really usable and more of a fun project.

I might be confused by the question, but I overfit the model on a single file and then transport the model along with the arithmetic coding file. There have been ideas where you generalize a model (constant weights) and then pass the arithmetic coding file along with it. So that way you only pass the arithmetic coding file.

BUT my model size is just 900KB (for 100mb file atleast) so it is negligible

Yeah yeah, I just found the idea kinda interesting so wanted to implement it

Sounds interesting man, soo am a bit confused maybe but can you run this on enwik9?

For the model overhead to become significant enough to eat into the gains, the file size would need to be fairly small, right? I assumed nobody would use this for compressing anything below 100 MB.

I tested with 100 MB files because anything larger takes a long time to evaluate. The actual target was at least 1 GB, and in that case I would use a 100 MB model (Shannon entropy rules).

I also tried it on a 100 MB Photoshop file and was able to compress it down to 45 MB, whereas ZIP could only get it down to 60 MB. So yeah still not losing gains.

Reallly?? So have you published something so far? Can i read something? Sounds like you got some interesting ideas.

I tried it on a enwik9 100 mb slice and was able to compress it to 20 mb + 900kb transformer so 21mb.

I know the top submission was able to get it to 13 mb.

Still trying some ideas to get better compression.