wildstrawberry

Appreciate the followup!!!

There were a few tells of AI based on my use of AI for personal projects, especially the end section where it says "what I tried that didn't work", I've seen Claude put sections like that in the documentation.

Big thanks for linking the datasets. Here's my results from ZPAQ (max effort):

./zpaq.exe add archive.zpaq nyc_taxi_dataset_100mb_slice.txt -m5

Time: 218 seconds

Final size: 9.57MB

./zpaq.exe add archive.zpaq enwik_9_slice_100mb.txt -m5

Time: 199 seconds

Final size: 20.46MB

So, your approach is comparable to ZPAQ for the wiki dataset and achieves a better compression ratio than ZPAQ with the taxi dataset. Cool!

Bit of a tangent but if you're curious there's an interesting writeup from a few years back that compares lossless text compression algorithms at various effort levels (speed vs compression ratio). I read it recently which prompted all of my questions

https://giannirosato.com/blog/post/lossless-data-comp/

Three questions:

1. How much was AI used to generate documentation for this project?

2. The 100MB CSV data sources are not provided in the repo so it doesn't seem possible to reproduce your results. The enwik9 dataset says it is a "slice" of the larger data set, and there are many NYC taxi trip record datasets that exist. Can you provide the datasets used to generate your results?

3. I am surprised to see performance comparisons only between your transformer and WinZIP. What were your results when comparing your transformer to more modern approaches like LZMA2 (level 9), BZIP2 and ZPAQ (max effort)?

I wrote a paper late last year that demonstrated a practical attack against ballot secrecy for a Canadian online voting vendor. Thought you all might find it interesting.

A third-party observer can deduce (with a high probability) the candidate a voter voted for, despite the voter's connection to the voting serve being encrypted. This is because the length of the encrypted data leaks information.

The creator of the grid collapse video also made a video about the risk of EMPs, and argued that while the impacts could be severe, the 'hollywood-style' devastation that many predict on the internet (and in this thread) is not likely.

https://practical.engineering/blog/2022/11/7/how-would-a-nuc...